Troubleshooting tips for the dynamic cache service

Troubleshooting tips for the dynamic cache service

The dynamic cache service works within an application server JVM, intercepting calls to cacheable objects.

Servlets are not cached

Enable servlet caching. On the web container settings page of the administrative console, select the Enable servlet caching check box.

Cache entries are not written to disk

Cache entries are written to disk when the cache is full and new entries are added to the memory cache. Cache entries also are written to disk when Flush to disk is enabled in the administrative console and the server is stopped.
Verify that Disk offload is enabled on the Dynamic cache service settings page of the administrative console. Also verify that cache entries written to disk are serializable and do not have the PersistToDisk configuration set to false.

Some servlets are not replicated or written to disk

Ensure that the attributes and response are serializable. If we do not want to store the attributes, use the following property in the cache policy:
<property name="save-attributes">false</property>

Dynamic cache service does not cache fragments on the Edge

Set the EdgeCacheable property to true in the cache policy for those entries that are to be cached on the Edge.
<property name="EdgeCacheable">true</property>

Dynamic cache invalidations are not sent to the IBM HTTP Server plug-in

The DynaCacheEsi.ear file is required to send invalidations to external caches.
Install the DynaCacheEsi.ear file using the administrative console.

Cache entries are evicted often

The cache is full and new entries are added to the cache.
Cache entries are evicted when the cache is full and new entries are added to the cache. A least recently used (LRU) eviction mechanism removes the least recently used entry to make space for the new entries.
Enable Disk offload on the Dynamic cache service settings page of the administrative console so the entries are written to disk. We can also increase the cache size to accommodate more entries in the cache.

Cache entries in disk with timeout set to 0 expire after one day

The maximum lifetime of an entry in disk cache is 24 hours. A timeout of 0 in the cache policy configures these entries to stay in disk cache for one whole day, unless they are evicted earlier.
Set the timeout for the cache policy to a number less than 0.

Unable to monitor cache entries on the Edge

Use the dynamic cache monitor to monitor the contents of the memory cache, disk cache and external caches, like the Edge cache. For the ESI processor's cache to be visible in the cache monitor, the DynaCacheEsi.ear application must be installed and the esiInvalidationMonitor property must be set to true in the plugin-cfg.xml file.

Set the esiInvalidationMonitor property in the plugin-cfg.xml file to true. Refer to the Displaying cache information article for more information about monitoring dynamic cache service activity.

Fine tune cache for my environment

Use the Tivoli Performance viewer to study the caching behavior for the applications. Also consider performing the following actions:

Increase the priority of cache entries that are expensive to regenerate.
Modify timeout of entries so that they stay in memory as long as they are valid.
Enable disk offload to store LRU evicted entries.
Increase the cache size.

Cleaning the disk cache files after installing the fix pack or a new release

If the server is configured to use the disk cache, we must delete the disk cache files because the disk cache files are not compatible to the previous version.
Failure to remove the old disk cache files results in a ClassCastException error in the systemerr.log file when we access the cache from the disk.
IBM recommends using the High Performance Extensible Logging (HPEL) log and trace infrastructure . We view HPEL log and trace information using the logViewer .
To delete the disk cache...

Note your disk offload location. If we do not know the disk cache offload location...

Click...
Servers > Application servers >server > Container services > Dynamic cache service tree

The location is specified in the Disk offload field. If the location is not specified, the default directory profile_root/temp/your_node/server/_dynacache is used.

Verify the you stop the server and delete all the files under the offload location.
If we use the WebSphere Application Server, Network Deployment product, delete the disk cache files for each server.

Set the flush attribute to true on every <jsp:include> tag in the cacheable JavaServer Pages file

Ensure we set the flush attribute to true on every <jsp:include> tag in the cacheable JavaServer Pages file

When we obtain the JavaServer Pages (JSP) file from the dynamic cache, a part of the page is not displayed.
The flush attribute is set to false on the <jsp: include> tag in the JSP file.
When the cacheable JSP file includes another JSP file and if the flush attribute is set to false on the <jsp: include> tag, any data written to the parent output stream prior to the<jsp: include> tag are not cached.
Set flush=true on every <jsp: include> tag in the cacheable JSP file.

(ZOS) File system I/O activity when the application server is idle

You experience file system I/O activity when the application server is idle. WAS is polling the Hierarchical File System (HFS) for the cachespec.xml file, even when there are no cache policies defined.
Dynamic cache queries the file system every 30 seconds to check for updates to the cache policy file. By checking for cache policy file updates, dynamic cache can automatically update any changed cache policies.

The dynamic cache is enabled by default because several system components rely on it for performance reasons. We can disable the dynamic cache service using a wsadmin script or through the administrative console. Refer to the Using the dynamic cache service topic for more information about enabling the dynamic cache service.

CAUTION:
By disabling dynamic cache we might experience slower performance of security because it uses the distributed map functionality of the dynamic cache service.
Set flush=true on every <jsp: include> tag in the cacheable JSP file.

Dynamic cache limitation when using the JSTL <c:import> tag to include a fragment

When a cacheable fragment is included using the JavaServer Pages Standard Tag Library (JSTL) <c:import> tag, part of the page content disappears and part of the page content displays twice on a cache hit.
Dynamic cache relies on flushing the content prior to and after including a fragment so that the parent content preceding the include is not lost, and also to prevent pulling the child content into the parent fragment.
However, in the case of the JSTL <c:import> tag, the flush=true attribute, which flushes the parent writer prior to the child fragment is actually invoked, is not supported. Also, JSTL buffers the responses, so that the child writer is not flushed following the child fragment is done. Subsequently, the child response is pulled into the parent.

Restriction: Dynamic cache will return multiple include statements when the JSTL <c:import> tag is used in a cached JavaServer Pages (JSP) file.
To avoid this problem, surround the <c:import> statement with out.flush method statements as follows:
<% out.flush(); %>
<c:import url="DNCParent2.jsp" />
<% out.flush(); %> 
Service integration bus messages are repeated

The service integration bus messages are repeated in the logs of the cluster members hosting a production application

In a multi-cell environment, the following messages are repeated in the logs of the cluster members hosting the production application:
[time_stamp] CWSIT0007W: It is not possible to contact
 the bootstrap server at 
 9.9.9.9:7299:BootstrapBasicMessaging because of exception: 
 com.ibm.websphere.sib.exception.SIResourceException: 
 CWSIC1001E: 
A client attempted to connect with a remote messaging  engine (9.9.9.9:7299 - BootstrapBasicMessaging) but
 the connection cannot be completed. Ensure the  messaging engine is started: exception com.ibm.ws.sib
 .jfapchannel.JFapConnectFailedException: CWSIJ0063E: A 
 network connection to host name 9.9.9.9, port 7299
 cannot be established...
[time_stamp] 00000023 SystemOut O RemoteInvalidator
 unable to connect to ...
First, ensure each service integration bus member in the remote cell is started. Next, ensure that the SIB_ENDPOINT_ADDRESS port is specified correctly for each service integration bus member in the remote cell or core group. If the wrong port is specified, delete the outbound configuration "--setup=dynacacheOutSIB --delete..." and reconfigure it using the correct port. When everything is working correctly, a message similar to the following message displays in the logs:
[time_stamp] 0000000a RemoteInvalid I   
DYNA1075I: The "RemoteJMSInvalidator" external 
cache group is connected. The configured JNDI name
of the queue connection factory is 
"DynacacheDestination-cell2" and the JNDI name of 
the targeted destination is 
"jms/DynacacheOutBoundConnectionFactory-cell1".
Platform messaging component cannot authenticate the user ID

In a multi-cell environment, a service integration bus member has the following error:
[time_stamp] 00000022 SibMessage   E  [:] CWSII0050E: 
The Platform Messaging Component can not authenticate 
the user ID.
This message might be misleading because the dynamic cache service does not use secure buses.
This message might indicate one of the following problems:
The service integration bus server is receiving a request for a bus destination that does not exist. Ensure the correct remote cell name was used when executing "--setup=dynacacheOutSIB" and "--setup=dynacacheECA" in the sending cell.

No service integration bus members are available. This problem often occurs because the wrong cluster is specified when doing the inbound setup "--setup=dynacacheInSIB". Ensure the service integration bus cluster is specified for the "--cluster" option and not some other cluster.

The messaging engine's unique ID does not match that found in the data store

In a multi-cell environment, a service integration bus member has the following error:
[DynacacheBus-edgeaphid10Cell01-cg2:edgeaphid10Node01.
 cg2SIBServer-DynacacheBus-edgeaphid10Cell01-cg2] 
CWSIS1535E: The messaging engine's unique id does not
 match that found in the data store. 
 ME_UUID=D520787E8CA7F18A, ME_UUID(DB)=980C0B42B3A904F3
[time_stamp] 0000002f SibMessage    I   
[DynacacheBus-edgeaphid10Cell01-cg2:edgeaphid10Node01.
 cg2SIBServer-DynacacheBus-edgeaphid10Cell01-cg2] 
CWSIS1546I: The messaging engine, 
 ME_UUID=D520787E8CA7F18A, INC_UUID=7228ea45e216f3ef,
 has lost an existing lock or failed to gain an initial
 lock on the data store.
[time_stamp] 0000002f SibMessage    I   
[DynacacheBus-edgeaphid10Cell01-cg2:edgeaphid10Node01.
 cg2SIBServer-DynacacheBus-edgeaphid10Cell01-cg2] 
CWSIS1519E: Messaging engine edgeaphid10Node01.
 cg2SIBServer-DynacacheBus-edgeaphid10Cell01-cg2 
 cannot obtain the lock on its data store, which ensures 
 it has exclusive access to the data.
[time_stamp] 0000002d SibMessage    E   
[DynacacheBus-edgeaphid10Cell01-cg2:edgeaphid10Node01.
 cg2SIBServer-DynacacheBus-edgeaphid10Cell01-cg2] 
CWSIS0002E: The messaging engine encountered an exception
 while starting. Exception: com.ibm.ws.sib.msgstore.
 PersistenceException: 
 CWSIS1501E: The data source has produced an unexpected
 exception: com.ibm.ws.sib.msgstore.persistence.
 DatasourceWrapperStateException: New connections cannot
 be provided because the persistence layer has been
 stopped	at com.ibm.ws.sib.msgstore.persistence.impl.
 PersistentMessageStoreImpl.start
 (PersistentMessageStoreImpl.java:188)
	at com.ibm.ws.sib.msgstore.impl.MessageStoreImpl.
   start(MessageStoreImpl.java:1175)
	at com.ibm.ws.sib.admin.impl.JsMessagingEngineImpl.
	 start(JsMessagingEngineImpl.java:491)
Stop the server, delete the directory in WAS_INSTALL_ROOT\profiles\AppSrv01\filestores\com.ibm.ws.sib that corresponds to the service integration bus member, and restart the server.

dynacacheJMSSIB script can not locate DynacacheMessageHandler.ear file

In a multi-cell environment, the dynacacheJMSSIB script can not locate DynacacheMessageHandler.ear file, and the following message appears in the log:
WASX7017E: Exception received while running file 
"../../../util/dynacacheJMSSIB.py"; exception 
information: 
com.ibm.ws.scripting.ScriptingException: WASX7115E: 
Cannot read input file "/opt/WAS/6.1/cf270928.19
 /profiles/AppSrv01/logs/DynacacheMessageHandler.ear"
Run the dynacacheJMSSIB script from WAS_INSTALL_ROOT/profiles/PROFILE_NAME/bin so that it can locate ../../../util/dynacacheJMSSIB.py and ../../../installableApps/DynacacheMessageHandler.ear at the appropriate relative paths.

Related:

Cache monitor
Troubleshoot the dynamic cache service
Displaying cache information
Task overview: Using the dynamic cache service to improve performance
Use High Performance Extensible Logging to troubleshoot applications
Install enterprise application files with the console
Use the dynamic cache service
Web container settings
Dynamic cache service settings