Troubleshoot deadlocks
The following sections describe some of the most common deadlock scenarios and suggestions on how to avoid them.
Before you begin
Implement exception handling in the application. See Implement exception handling in locking scenarios for more information.The following exception displays as a result:
com.ibm.websphere.objectgrid.plugins.LockDeadlockException: MessageThis message represents the string that is passed as a parameter when the exception is created and thrown.
Procedure
- Problem: LockTimeoutException exception.
Description: When a transaction or client asks for a lock to be granted for a specific map entry, the request often waits for the current client to release the lock before the request is submitted. If the lock request remains idle for an extended period of time, and a lock is never granted, LockTimeoutException exception is created to prevent a deadlock, which is described in more detail in the following section. You are more likely to see this exception when using a pessimistic locking strategy, because the lock never releases until the transaction commits.
Retrieve more details:The LockTimeoutException exception contains the getLockRequestQueueDetails method, which returns a string. Use this method to see a detailed description of the situation that triggers the exception. The following is an example of code that catches the exception, and displays an error message.
try { ... } catch (LockTimeoutException lte) { System.out.println(lte.getLockRequestQueueDetails()); }
The output result is:
lock request queue −>[TX:163C269E−0105−4000−E0D7−5B3B090A571D, state = Granted 5348 milli−seconds ago, mode = U] −>[TX:163C2734−0105−4000−E024−5B3B090A571D, state = Waiting for 5348 milli−seconds, mode = U] −>[TX:163C328C−0105−4000−E114−5B3B090A571D, state = Waiting for 1402 milli−seconds, mode = U]
If you receive the exception in an ObjectGridException exception catch block, the following code determines the exception and displays the queue details. It also uses the findRootCause utility method.
try { ... } catch (ObjectGridException oe) { Throwable Root = findRootCause( oe ); if (Root instanceof LockTimeoutException) { LockTimeoutException lte = (LockTimeoutException)Root; System.out.println(lte.getLockRequestQueueDetails()); } }
Solution: A LockTimeoutException exception prevents possible deadlocks in the application. An exception of this type results when the exception waits a set amount of time. You can set the amount of time that the exception waits by using the setLockTimeout(int) method, which is available for the BackingMap. If a deadlock does not actually exist in the application, adjust the lock timeout to avoid the LockTimeoutException.The following code shows how to create an ObjectGrid object, define a map, and set its LockTimeout value to 30 seconds:
ObjectGrid objGrid = new ObjectGrid(); BackingMap bMap = objGrid.defineMap("MapName"); bMap.setLockTimeout(30);
Use the previous hardcoded example to set ObjectGrid and map properties. If you create ObjectGrid from an XML file, set the LockTimeout attribute within the backingMap element. The following is an example of a backingMap element that sets a map LockTimeout value to 30 seconds.
<backingMap name="MapName" lockStrategy="PESSIMISTIC" lockTimeout="30">
- Problem: Single key deadlocks.
Description: The following scenarios describe how deadlocks can occur when a single key is accessed using a S lock and later updated. When this happens from two transactions simultaneously, it results in a deadlock.
Table 1. Single key deadlocks scenario Thread 1 Thread 2 1 session.begin() session.begin() Each thread establishes an independent transaction. 2 map.get(key1) map.get(key1) S lock granted to both transactions for key1. 3 map.update(Key1,v) No U lock. Update performed in transactional cache. 4 map.update(key1,v) No U lock. Update performed in the transactional cache 5 session.commit() Blocked: The S lock for key1 cannot be upgraded to an X lock because Thread 2 has an S lock. 6 session.commit() Deadlock: The S lock for key1 cannot be upgraded to an X lock because T1 has an S lock.
Table 2. Single key deadlocks, continued Thread 1 Thread 2 1 session.begin() session.begin() Each thread establishes an independent transaction. 2 map.get(key1) S lock granted for key1 3 map.getForUpdate(key1,v) S lock is upgraded to a U lock for key1. 4 map.get(key1) S lock granted for key1. 5 map.getForUpdate(key1,v) Blocked: T1 already has U lock. 6 session.commit() Deadlock: The U lock for key1 cannot be upgraded. 7 session.commit() Deadlock: The S lock for key1 cannot be upgraded.
Table 3. Single key deadlocks, continued Thread 1 Thread 2 1 session.begin() session.begin() Each thread establish an independent transaction 2 map.get(key1) S lock granted for key1. 3 map.getForUpdate(key1,v) S lock is upgraded to a U lock for key1 4 map.get(key1) S lock is granted for key1. 5 map.getForUpdate(key1,v) Blocked: Thread 1 already has a U lock. 6 session.commit() Deadlock: The U lock for key1 cannot be upgraded to an X lock because Thread 2 has an S lock. If the ObjectMap.getForUpdate is used to avoid the S lock, then the deadlock is avoided:
Table 4. Single key deadlocks, continued Thread 1 Thread 2 1 session.begin() session.begin() Each thread establishes an independent transaction. 2 map.getForUpdate(key1) U lock granted to thread 1 for key1. 3 map.getForUpdate(key1) U lock request is blocked. 4 map.update(key1,v) <blocked> 5 session.commit() <blocked> The U lock for key1 can be successfully upgraded to an X lock. 6 <released> The U lock is finally granted to key1 for thread 2. 7 map.update(key2,v) U lock granted to thread 2 for key2. 8 session.commit() The U lock for key1 can successfully be upgraded to an X lock. Solutions:
- Use the getForUpdate method instead of get to acquire a U lock instead of an S lock.
- Use a transaction isolation level of read committed to avoid holding S locks. Reducing the transaction isolation level increases the possibility of non-repeatable reads. However, non-repeatable reads from one client are only possible if the transaction cache is explicitly invalidated by the same client.
- Use the optimistic lock strategy. Using the optimistic lock strategy requires handling optimistic collision exceptions.
- Problem: Ordered multiple key deadlock
Description: This scenario describes what happens if two transactions attempt to update the same entry directly and hold S locks to other entries.
Table 5. Ordered multiple key deadlock scenario Thread 1 Thread 2 1 session.begin() session.begin() Each thread establishes an independent transaction. 2 map.get(key1) map.get(key1) S lock granted to both transactions for key1. 3 map.get(key2) map.get(key2) S lock granted to both transactions for key2. 4 map.update(key1,v) No U lock. Update performed in transactional cache. 5 map.update(key2,v) No U lock. Update performed in transactional cache. 6. session.commit() Blocked: The S lock for key 1 cannot be upgraded to an X lock because thread 2 has an S lock. 7 session.commit() Deadlock: The S lock for key 2 cannot be upgraded because thread 1 has an S lock. Use the ObjectMap.getForUpdate method to avoid the S lock, then you can avoid the deadlock:
Table 6. Ordered multiple key deadlock scenario, continued Thread 1 Thread 2 1 session.begin() session.begin() Each thread establishes an independent transaction. 2 map.getForUpdate(key1) U lock granted to transaction T1 for key1. 3 map.getForUpdate(key1) U lock request is blocked. 4 map.get(key2) <blocked> S lock granted for T1 for key2. 5 map.update(key1,v) <blocked> 6 session.commit() <blocked> The U lock for key1 can be successfully upgraded to an X lock. 7 <released> The U lock is finally granted to key1 for T2 8 map.get(key2) S lock granted to T2 for key2. 9 map.update(key2,v) U lock granted to T2 for key2. 10 session.commit() The U lock for key1 can be successfully upgraded to an X lock. Solutions:
- Use the getForUpdate method instead of the get method to acquire a U lock directly for the first key. This strategy works only if the method order is deterministic.
- Use a transaction isolation level of read committed to avoid holding S locks. This solution is the easiest to implement if the method order is not deterministic. Reducing the transaction isolation level increases the possibility of non-repeatable reads. However, non-repeatable reads are only possible if the transaction cache is explicitly invalidated.
- Use the optimistic lock strategy. Using the optimistic lock strategy requires handling optimistic collision exceptions.
- Problem: Out of order with U lock
Description: If the order in which keys are requested cannot be guaranteed, then a deadlock can still occur.
Table 7. Out of order with U lock scenario Thread 1 Thread 2 1 session.begin() session.begin() Each thread establishes an independent transaction. 2 map.getforUpdate(key1) map.getForUpdate(key2) U locks successfully granted for key1 and key2. 3 map.get(key2) map.get(key1) S lock granted for key1 and key2. 4 map.update(key1,v) map.update(key2,v) 5 session.commit() The U lock cannot be upgraded to an X lock because T2 has an S lock. 6 session.commit() The U lock cannot be upgraded to an X lock because T1 has an S lock.
Solutions:
- Wrap all work with a single global U lock (mutex). This method reduces concurrency, but handles all scenarios when access and order is non-deterministic.
- Use a transaction isolation level of read committed to avoid holding S locks. This solution is the easiest to implement if the method order is not deterministic and provides the greatest amount of concurrency. Reducing the transaction isolation level increases the possibility of non-repeatable reads. However, non-repeatable reads are only possible if the transaction cache is explicitly invalidated.
- Use the optimistic lock strategy. Using the optimistic lock strategy requires handling optimistic collision exceptions.
Parent topic:
Troubleshoot
Related concepts
Related tasks
Troubleshoot client connectivity
Implement exception handling in locking scenarios
Related reference
IBM Support Assistant for WebSphere eXtreme Scale