Troubleshoot tips for the dynamic cache service

Network Deployment (Distributed operating systems), v8.0 > Reference > Troubleshoot tips

Troubleshoot tips for the dynamic cache service

The dynamic cache service works within an application server JVM, intercepting calls to cacheable objects. This article describes some common runtime and configuration problems and remedies.

Servlets are not cached

Recommended response Enable servlet caching. On the web container settings page of the admin console, select the box...

Enable servlet caching

Cache entries are not written to disk

Explanation Cache entries are written to disk when the cache is full and new entries are added to the memory cache. Cache entries also are written to disk when Flush to disk is enabled in the administrative console and the server is stopped.
Recommended response Verify that Disk offload is enabled on the Dynamic cache service settings page of the admin console. Also verify that cache entries written to disk are serializable and do not have the PersistToDisk configuration set to false.

Some servlets are not replicated or written to disk
Recommended response
Ensure that the attributes and response are serializable. If you do not want to store the attributes, use the following property in your cache policy:
<property name="save-attributes">false
</property> 
Dynamic cache service does not cache fragments on the Edge
Recommended response
Set the EdgeCacheable property to true in the cache policy for those entries that are to be cached on the Edge.
<property name="EdgeCacheable">true
</property> 
Dynamic cache invalidations are not sent to the IBM HTTP Server plug-in

Explanation The DynaCacheEsi.ear file is required to send invalidations to external caches.
Recommended response Install the DynaCacheEsi.ear file using the admin console.

Cache entries are evicted often

Problem The cache is full and new entries are added to the cache.
Explanation Cache entries are evicted when the cache is full and new entries are added to the cache. A least recently used (LRU) eviction mechanism removes the least recently used entry to make space for the new entries.
Recommended response Enable Disk offload on the Dynamic cache service settings page of the admin console so the entries are written to disk. We can also increase the cache size to accommodate more entries in the cache.

Cache entries in disk with timeout set to 0 expire after one day

Explanation The maximum lifetime of an entry in disk cache is 24 hours. A timeout of 0 in the cache policy configures these entries to stay in disk cache for one whole day, unless they are evicted earlier.
Recommended response Set the timeout for the cache policy to a number less than 0.

I cannot monitor cache entries on the Edge

Explanation Use the dynamic cache monitor to monitor the contents of the memory cache, disk cache and external caches, like the Edge cache. For the ESI processor's cache to be visible in the cache monitor, the DynaCacheEsi.ear application must be installed and the esiInvalidationMonitor property must be set to true in the plugin-cfg.xml file.
Recommended response Set the esiInvalidationMonitor property in the plugin-cfg.xml file to true. Refer to the Displaying cache information article for more information about monitoring dynamic cache service activity.

I want to tune cache for my environment

Recommended response Use the Tivoli Performance viewer to study the caching behavior for the applications. Also consider performing the following actions:

Increase the priority of cache entries that are expensive to regenerate.
Modify timeout of entries so that they stay in memory as long as they are valid.
Enable disk offload to store LRU evicted entries.
Increase the cache size.

Clean the disk cache files after installing the fix pack or a new release if you use the disk cache function

Symptom If the server is configured to use the disk cache, delete the disk cache files because the disk cache files are not compatible to the previous version.
Problem Failure to remove the old disk cache files results in a ClassCastException error in the systemerr.log file when you access the cache from the disk.
New feature: Beginning in WAS v8.0 you can configure the server to use the HPEL log and trace infrastructure instead of using SystemOut.log , SystemErr.log, trace.log, and activity.log files or native z/OS logging facilities. If you are using HPEL, you can access all of your log and trace information using the LogViewer command-line tool from your server profile bin directory. See the information about using HPEL to troubleshoot applications for more information on using HPEL.New feature:
Recommended response To delete the disk cache, perform the following steps:

Note your disk offload location. If you do not know the disk cache offload location, perform the following steps:

Click Servers > Application servers > server_name
> Container services > Dynamic cache service in the administrative console navigation tree.
The location is specified in the Disk offload field. If the location is not specified, the default directory PROFILE_ROOT/temp/your_node/server_name
_dynacache is used.

Make sure that the you stop the server and delete all the files under the offload location.

If you use the WAS ND, delete the disk cache files for each server.

Set the flush attribute to true on every <jsp:include> tag in the cacheable JSP file

Symptom When you obtain the JSP file from the dynamic cache, a part of the page is not displayed.
Problem The flush attribute is set to false on the <jsp: include> tag in the JSP file.
Description When the cacheable JSP file includes another JSP file and if the flush attribute is set to false on the <jsp: include> tag, any data written to the parent output stream before the <jsp: include> tag are not cached.
Recommended response Set flush=true on every <jsp: include> tag in the cacheable JSP file.

File system I/O activity when the application server is idle

Symptom You experience file system I/O activity when the application server is idle. WAS is polling the Hierarchical File System (HFS) for the cachespec.xml file, even when there are no cache policies defined.
Explanation Dynamic cache queries the file system every 30 seconds to check for updates to the cache policy file. By checking for cache policy file updates, dynamic cache can automatically update any changed cache policies.
Possible response The dynamic cache is enabled by default because several system components rely on it for performance reasons. We can disable the dynamic cache service by using a wsadmin script or through the admin console. Refer to the Using the dynamic cache service topic for more information about enabling the dynamic cache service.
CAUTION:
By disabling dynamic cache you might experience slower performance of security because it uses the distributed map functionality of the dynamic cache service.
Recommended response Set flush=true on every <jsp: include> tag in the cacheable JSP file.

Dynamic cache limitation when using the JSTL <c:import> tag to include a fragment
Problem When a cacheable fragment is included using the JavaServer Pages Standard Tag Library (JSTL) <c:import> tag, part of the page content disappears and part of the page content displays twice on a cache hit.
Description Dynamic cache relies on flushing the content before and after including a fragment so that the parent content before the include is not lost, and also to prevent pulling the child content into the parent fragment.
However, in the case of the JSTL <c:import> tag, the flush=true attribute, which flushes the parent writer before the child fragment is actually invoked, is not supported. Also, JSTL buffers the responses, so that the child writer is not flushed right after the child fragment is done. Subsequently, the child response is pulled into the parent.
Restriction: Dynamic cache will return multiple include statements when the JSTL <c:import> tag is used in a cached JSP file.
Recommended response
To avoid this problem, surround the <c:import> statement with out.flush method statements as follows:
<% out.flush(); %>
<c:import url="DNCParent2.jsp" />
<% out.flush(); %>
Service integration bus messages are repeated in the logs of the cluster members hosting a production application
Problem
In a multi-cell environment, the following messages are repeated in the logs of the cluster members hosting the production application:
[time_stamp] CWSIT0007W: It is not
possible to contact the bootstrap server at
9.9.9.9:7299:BootstrapBasicMessaging because of exception:
com.ibm.websphere.sib.exception.SIResourceException: CWSIC1001E:
A client attempted to connect with a remote messaging engine (9.9.9.9:7299 -
BootstrapBasicMessaging) but the connection cannot be completed. Ensure the messaging engine is started:
exception com.ibm.ws.sib.jfapchannel.JFapConnectFailedException: CWSIJ0063E: A
network connection to host name 9.9.9.9, port 7299 cannot be established...
[time_stamp] 00000023 SystemOut O RemoteInvalidator unable to connect to ...
Recommended response
First, ensure each service integration bus member in the remote cell is started. Next, ensure that the SIB_ENDPOINT_ADDRESS port is specified correctly for each service integration bus member in the remote cell or core group. If the wrong port is specified, delete the outbound configuration "--setup=dynacacheOutSIB --delete..." and reconfigure it using the correct port. When everything is working correctly, a message similar to the following message displays in the logs:
[time_stamp] 0000000a SystemOut O RemoteInvalidator successfully connected
to DynacacheDestination-edgeaphid10Cell01-cg1 using
Platform messaging component cannot authenticate the user ID
Explanation
In a multi-cell environment, a service integration bus member has the following error:
[time_stamp] 00000022 SibMessage    E   [:] CWSII0050E: The Platform Messaging Component can not authenticate the user ID.
This message might be misleading because the dynamic cache service does not use secure buses.
Recommended response This message might indicate one of the following problems:
The service integration bus server is receiving a request for a bus destination that does not exist. Ensure the correct remote cell name was used when executing "--setup=dynacacheOutSIB" and "--setup=dynacacheECA" in the sending cell.
No service integration bus members are available. This problem often occurs because the wrong cluster is specified when doing the inbound setup "--setup=dynacacheInSIB". Ensure the service integration bus cluster is specified for the "--cluster" option and not some other cluster.
The messaging engine's unique ID does not match that found in the data store
Explanation
In a multi-cell environment, a service integration bus member has the following error:
[DynacacheBus-edgeaphid10Cell01-cg2:edgeaphid10Node01.cg2SIBServer-DynacacheBus-
edgeaphid10Cell01-cg2]
CWSIS1535E: The messaging engine's unique id does not
match that found in the data store. ME_UUID=D520787E8CA7F18A,
ME_UUID(DB)=980C0B42B3A904F3
[time_stamp] 0000002f SibMessage    I
[DynacacheBus-edgeaphid10Cell01-cg2:edgeaphid10Node01.cg2SIBServer-DynacacheBus-
edgeaphid10Cell01-cg2]
CWSIS1546I: The messaging engine,
ME_UUID=D520787E8CA7F18A, INC_UUID=7228ea45e216f3ef, has lost an existing lock
or failed to gain an initial lock on the data store.
[time_stamp] 0000002f SibMessage    I
[DynacacheBus-edgeaphid10Cell01-cg2:edgeaphid10Node01.cg2SIBServer-DynacacheBus-
edgeaphid10Cell01-cg2]
CWSIS1519E: Messaging engine
edgeaphid10Node01.cg2SIBServer-DynacacheBus-edgeaphid10Cell01-cg2 cannot obtain
the lock on its data store, which ensures it has exclusive access to the data.
[time_stamp] 0000002d SibMessage    E
[DynacacheBus-edgeaphid10Cell01-cg2:edgeaphid10Node01.cg2SIBServer-DynacacheBus-
edgeaphid10Cell01-cg2]
CWSIS0002E: The messaging engine encountered an exception  while starting. Exception: com.ibm.ws.sib.msgstore.PersistenceException:
 CWSIS1501E: The data source has produced an unexpected exception:
 com.ibm.ws.sib.msgstore.persistence.DatasourceWrapperStateException: New
 connections cannot be provided because the persistence layer has been stopped
 at com.ibm.ws.sib.msgstore.persistence.impl.PersistentMessageStoreImpl.start
 (PersistentMessageStoreImpl.java:188)
 at com.ibm.ws.sib.msgstore.impl.MessageStoreImpl.start(MessageStoreImpl.java
 :1175)
 at com.ibm.ws.sib.admin.impl.JsMessagingEngineImpl.start(
 JsMessagingEngineImpl.java:491)
Recommended response Stop the server and delete the directory in WAS_INSTALL_ROOT\profiles\AppSrv01\filestores\com.ibm.ws.sib, that corresponds to the service integration bus member and restart the server.
dynacacheJMSSIB script can not locate DynacacheMessageHandler.ear file
Explanation
In a multi-cell environment, the dynacacheJMSSIB script can not locate DynacacheMessageHandler.ear file, and the following message appears in the log:
WASX7017E: Exception received while running file
"../../../util/dynacacheJMSSIB.py"; exception information:
com.ibm.ws.scripting.ScriptingException: WASX7115E: Cannot read input file
"/opt/WAS/6.1/cf270928.19/profiles/AppSrv01/logs/DynacacheMessageHandler.ear"
Recommended response Run the dynacacheJMSSIB script from WAS_INSTALL_ROOT/profiles/PROFILE_NAME/bin so that it can locate ../../../util/dynacacheJMSSIB.py and ../../../installableApps/DynacacheMessageHandler.ear at the appropriate relative paths.
Cache monitor
Task overview: Using the dynamic cache service to improve performance
Use HPEL to troubleshoot applications
Install enterprise application files with the console
Use the dynamic cache service
Related

Troubleshoot the dynamic cache service
Display cache information

+
Search Tips | Advanced Search

Explanation	Cache entries are written to disk when the cache is full and new entries are added to the memory cache. Cache entries also are written to disk when Flush to disk is enabled in the administrative console and the server is stopped.
Recommended response	Verify that Disk offload is enabled on the Dynamic cache service settings page of the admin console. Also verify that cache entries written to disk are serializable and do not have the PersistToDisk configuration set to false.

Explanation	The DynaCacheEsi.ear file is required to send invalidations to external caches.
Recommended response	Install the DynaCacheEsi.ear file using the admin console.

Problem	The cache is full and new entries are added to the cache.
Explanation	Cache entries are evicted when the cache is full and new entries are added to the cache. A least recently used (LRU) eviction mechanism removes the least recently used entry to make space for the new entries.
Recommended response	Enable Disk offload on the Dynamic cache service settings page of the admin console so the entries are written to disk. We can also increase the cache size to accommodate more entries in the cache.

Explanation	The maximum lifetime of an entry in disk cache is 24 hours. A timeout of 0 in the cache policy configures these entries to stay in disk cache for one whole day, unless they are evicted earlier.
Recommended response	Set the timeout for the cache policy to a number less than 0.

Explanation	Use the dynamic cache monitor to monitor the contents of the memory cache, disk cache and external caches, like the Edge cache. For the ESI processor's cache to be visible in the cache monitor, the DynaCacheEsi.ear application must be installed and the esiInvalidationMonitor property must be set to true in the plugin-cfg.xml file.
Recommended response	Set the esiInvalidationMonitor property in the plugin-cfg.xml file to true. Refer to the Displaying cache information article for more information about monitoring dynamic cache service activity.

Symptom	When you obtain the JSP file from the dynamic cache, a part of the page is not displayed.
Problem	The flush attribute is set to false on the <jsp: include> tag in the JSP file.
Description	When the cacheable JSP file includes another JSP file and if the flush attribute is set to false on the <jsp: include> tag, any data written to the parent output stream before the <jsp: include> tag are not cached.
Recommended response	Set flush=true on every <jsp: include> tag in the cacheable JSP file.

Symptom	You experience file system I/O activity when the application server is idle. WAS is polling the Hierarchical File System (HFS) for the cachespec.xml file, even when there are no cache policies defined.
Explanation	Dynamic cache queries the file system every 30 seconds to check for updates to the cache policy file. By checking for cache policy file updates, dynamic cache can automatically update any changed cache policies.
Possible response	The dynamic cache is enabled by default because several system components rely on it for performance reasons. We can disable the dynamic cache service by using a wsadmin script or through the admin console. Refer to the Using the dynamic cache service topic for more information about enabling the dynamic cache service. CAUTION: By disabling dynamic cache you might experience slower performance of security because it uses the distributed map functionality of the dynamic cache service.
Recommended response	Set flush=true on every <jsp: include> tag in the cacheable JSP file.