8357826: Avoid running some jtreg tests when asan is configured

[MBaesken]

There are a couple of jtreg tests, especially in the HS area, with very special assumptions about memory layout/sizes .
Those fail when the address sanitizer is configured ( --enable-asan ).
The change adds a way to tag those tests with 'requires' so that they can be avoided easily when running jtreg tests with ASAN enabled.
Adjusting the tests for "pleasing" the sanitizer is not always desired (if possible for some tests it can be done later) .
While at it, also same is also added for ubsan .

---

Progress

• Change must be properly reviewed (1 review required, with at least 1 Reviewer)
• Change must not contain extraneous whitespace
• Commit message must refer to an issue

Issue

• JDK-8357826: Avoid running some jtreg tests when asan is configured (Bug - P4)

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.org/jdk.git pull/25575/head:pull/25575
$ git checkout pull/25575

Update a local copy of the PR:
$ git checkout pull/25575
$ git pull https://git.openjdk.org/jdk.git pull/25575/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 25575

View PR using the GUI difftool:
$ git pr show -t 25575

Using diff file

Download this PR as a diff file:
https://git.openjdk.org/jdk/pull/25575.diff

Using Webrev

Link to Webrev Comment

[bridgekeeper]

👋 Welcome back mbaesken! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

[MBaesken]

The change to src/hotspot/cpu/x86/gc/z/zAddress_x86.cpp was added because of zgc issues with ASAN but we will address this in another change so I remove it from here.

[openjdk]

❗ This change is not yet ready to be integrated.
See the Progress checklist in the description for automated requirements.

[openjdk]

@MBaesken The following labels will be automatically applied to this pull request:

• hotspot
• serviceability

When this pull request is ready to be reviewed, an "RFR" email will be sent to the corresponding mailing lists. If you would like to change these labels, use the /label pull request command.

[mlbridge]

Webrevs

• 01: Full - Incremental (aa796c8a)
• 00: Full (3ad0d93a)

[MBaesken]

TestBreakSignalThreadDump shows this, so it does not work well with asan too

 stdout: [];
 stderr: [==12484==ASan runtime does not come first in initial library list; you should either link runtime to your application or manually preload it with LD_PRELOAD.

[plummercj]

Can you document why each tests fails so we have it on record? Can be done in the PR or the CR.

[dholmes-ora]

Changes look fine but I agree with Chris that we need to document why these tests don't work with ASAN, though I think I'd prefer to see an @comment before the @requires !vm.asan in the actual test files - assuming the reason can be stated clearly and succinctly.

[afshin-zafari]

In ASAN built JDK, some gtests and some other JTREG tests in runtime/ErrorHandling also fail.
Do we exclude these in another PR? or should they also be handled/excluded here?
The 'some' word in the PR's title is not strict, IMO.

[lmesnik]

Thank you for implementing exclusion this way. I'll approve PR once you address feedback about commenting.

[MBaesken]

gc/arguments/TestUseCompressedOopsFlagsWithUlimit.java
---------------------------------------------------------------
 stderr: [==46460==ERROR: AddressSanitizer failed to allocate 0xdfff0001000 (15392894357504) bytes at address 2008fff7000 (errno: 12)
==46460==ReserveShadowMemoryRange failed while trying to map 0xdfff0001000 bytes. Perhaps you're using ulimit -v

ulimit clashes with the memory requirements of ASAN

runtime/Thread/TestBreakSignalThreadDump.java
---------------------------------------------------------------
stderr: [==18432==ASan runtime does not come first in initial library list; you should either link runtime to your application or manually preload it with LD_PRELOAD.

loading of the jsig lib does currently not work well with ASAN lib

runtime/XCheckJniJsig/XCheckJSig.java
---------------------------------------------------------------
 stderr: [==71228==ASan runtime does not come first in initial library list; you should either link runtime to your application or manually preload it with LD_PRELOAD.

loading of the jsig lib does currently not work well with ASAN lib

runtime/cds/appcds/aotCode/AOTCodeCompressedOopsTest.java
---------------------------------------------------------------
reports ==35621==ERROR: AddressSanitizer: heap-buffer-overflow on address ...

this will be fixed hopefully so we could maybe remove the !asan tagging

serviceability/dcmd/vm/SystemDumpMapTest.java
---------------------------------------------------------------
Missing patterns in dump:
0x\\p{XDigit}+-0x\\p{XDigit}+ +\\d+ +[rwsxp-]+ +\\d+ +\\d+ +(4K|8K|16K|64K|2M|16M|64M) +com.*\\[heap\\]
test SystemDumpMapTest.jmx(): failure [410ms]

ASAN changes the memory map dump slightly, but the test has rather strict requirements

serviceability/dcmd/vm/SystemMapTest.java
---------------------------------------------------------------
test SystemMapTest.jmx(): failure [381ms]
java.lang.RuntimeException: '0x\\p{XDigit}+-0x\\p{XDigit}+ +\\d+ +[rwsxp-]+ +\\d+ +\\d+ +(4K|8K|16K|64K|2M|16M|64M) +com.*\\[heap\\]' missing from stdout/stderr

ASAN changes the memory map dump slightly, but the test has rather strict requirements

serviceability/sa/ClhsdbCDSCore.java
---------------------------------------------------------------
Output and diagnostic info for process 45808 was saved into 'pid-45808-output.log'
crashOutputString = [[0.028s][error][cds] An error has occurred while processing the shared archive file. Run with -Xlog:aot,cds for details.
[0.029s][error][cds] Mismatched values for property jdk.module.addexports: java.base/jdk.internal.misc=ALL-UNNAMED specified during runtime but not during dump time
[0.029s][error][cds] Disabling optimized module handling
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x000014d4d60ef8d2, pid=45808, tid=46654
#
# JRE version: OpenJDK Runtime Environment (25.0.0.1) (build 25.0.0.1-internal-adhoc.myuser.jdk)
# Java VM: OpenJDK 64-Bit Server VM (25.0.0.1-internal-adhoc.myuser.jdk, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, linux-amd64)
# Problematic frame:
# V  [libjvm.so+0x3d6b8d2]  Unsafe_PutInt+0x592
# No core dump will be written. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again

java.lang.RuntimeException: Test ERROR java.lang.RuntimeException: Output doesn't contain the location of core file.: expected true, was false
	at ClhsdbCDSCore.main(ClhsdbCDSCore.java:171)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
	at java.base/java.lang.reflect.Method.invoke(Method.java:565)
	at com.sun.javatest.regtest.agent.MainActionHelper$AgentVMRunnable.run(MainActionHelper.java:335)
	at java.base/java.lang.Thread.run(Thread.java:1474)
Caused by: java.lang.RuntimeException: Output doesn't contain the location of core file.: expected true, was false

Seems no core was written, maybe ASAN is to blame or my test environment ?

serviceability/sa/ClhsdbFindPC.java
---------------------------------------------------------------
java.lang.RuntimeException: Test ERROR java.lang.RuntimeException: Output doesn't contain the location of core file.: expected true, was false
	at ClhsdbFindPC.testFindPC(ClhsdbFindPC.java:317)
	at ClhsdbFindPC.main(ClhsdbFindPC.java:339)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
	at java.base/java.lang.reflect.Method.invoke(Method.java:565)
	at com.sun.javatest.regtest.agent.MainWrapper$MainTask.run(MainWrapper.java:138)
	at java.base/java.lang.Thread.run(Thread.java:1474)
Caused by: java.lang.RuntimeException: Output doesn't contain the location of core file.: expected true, was false

Looks similar to ClhsdbCDSCore issue

[MBaesken]

Turns out cds/appcds/aotCode/AOTCodeCompressedOopsTest.java was a real bug, so I guess we should remove it from this exclusion. Are you fine with the short explanations given, if yes I would add them as comment to the tests .

[MBaesken]

In ASAN built JDK, some gtests and some other JTREG tests in runtime/ErrorHandling also fail. Do we exclude these in another PR? or should they also be handled/excluded here? The 'some' word in the PR's title is not strict, IMO.

Yes it is not strict ; I did mostly tests with ASAN on Linux x86_64 and Linux ppc64le so far .
On x86_64 I saw a few more tests have issues with ASAN, but the intention of this PR was to just include the ones where it was clear to me what happens and where a 'fix' is not likely (and mostly also the ones I saw failing across the 2 OS/CPU platforms I mentioned).
Maybe that's why we should better remove the exclusion of AOTCodeCompressedOopsTest , because this is not some kind of incompatibility of ASAN with special test requirements, but a real memory issue.