RAM Class Persistence: Fix Open liberty server hangs

[lzhou2025]

The Open liberty server hangs on startup when loading classes from RCP cache and initializes the frozen classes. It is caused
by deadlock. Two threads wait for each other on RCP cache mutex and class table mutex in classes loading from RCP cache in loadNonArrayClass(). The changes are to pass the flag J9_FINDCLASS_FLAG_EXISTING_ONLY to class table lookup functions. If it's set for a frozen class, then it won't be loaded from RCP cache. The real problem is that a thread requested a class with J9_FINDCLASS_FLAG_EXISTING_ONLY is attempting to perform a RCP classload (loadWarmClass) which should never happen. We would have similar deadlock issues in a non-RCP scenario if a thread requesting a class with J9_FINDCLASS_FLAG_EXISTING_ONLY attempted to do a full classload. This is why the only threads that are allowed to load classes are VM threads and the JIT threads always call with J9_FINDCLASS_FLAG_EXISTING_ONLY.

Fixes: #22753

Co-authored-by: Tobi Ajila tobi_ajila@ca.ibm.com

[lzhou2025]

@babsingh as discussed in the meeting.

[babsingh]

The description of the deadlock is incomplete. Please describe both threads involved and explain how the two monitors interact to cause the deadlock.

[babsingh]

@lzhou2025 These changes won't work. How did you test them?

[babsingh]

The documentation needs more detail. Currently, the connection between the solution and how it resolves the deadlock is missing in the commit message below.

The Open liberty server hangs on startup when loading classes
from RCP cache and initializes the frozen classes. It is caused
by deadlock. Two threads wait for each other on RCP cache mutex
and class table mutex. The changes are to check the flag
J9_FINDCLASS_FLAG_EXISTING_ONLY if set for a frozen class, then
it won't be loaded from RCP cache. HOW IS THE DEADLOCK RESOLVED?

[babsingh]

The documentation needs to be better.

• What are the two threads?
• What code paths are taken by them?
• Why the monitors aren't released?
• How the fix resolves the deadlock?

[lzhou2025]

The documentation needs to be better.

* What are the two threads?

* What code paths are taken by them?

* Why the monitors aren't released?

* How the fix resolves the deadlock?

There are multiple hangs, this is one of them. Because they are all random, it's difficult to duplicate the case. If I remember correctly, the case is one thread owns RCP cache mutex, waits on class table mutex, other thread owns class table mutex, waits on RCP cache mutex. @TobiAjila suggested fix is to prevent the two threads to run into the situation, I believe.

[tajila]

The real problem is that a thread requested a class with J9_FINDCLASS_FLAG_EXISTING_ONLY is attempting to perform a RCP classload (loadWarmClass) which should never happen. In the RCP restore world if the class is frozen it is not loaded, so nothing should be done in this case.

We would have similar deadlock issues in a non-RCP scenario if a thread requesting a class with J9_FINDCLASS_FLAG_EXISTING_ONLY attempted to do a full classload. This is why the only threads that are allowed to load classes are VM threads and the JIT threads always call with J9_FINDCLASS_FLAG_EXISTING_ONLY.

We can mention in the commit message that it solves the deadlock, but it shouldn't be in the code because its really just part of the J9_FINDCLASS_FLAG_EXISTING_ONLY contract and the goal here is to enforce the existing rules in the RCP scenarios.

You may also want to add some of my comments above to the commit message.

[babsingh]

Thanks for updating the documentation. I have validated the fix: the hashClassTableAt* functions now return NULL when J9_FINDCLASS_FLAG_EXISTING_ONLY is specified. This prevents loadWarmClassFromSnapshot from being invoked and eliminates the conditions that previously led to deadlocks.

jenkins test sanity.functional plinux jdk21

[babsingh]

jenkins test sanity.functional plinux jdk21

[babsingh]

jenkins test sanity alinux jdk21