Import large files - HCL WebSphere Portal

Steps to improve performance

1. Increase the web containers transaction timeout setting. Go to...
   Servers | Server Types | WebSphere appservers | portal_server | Container Services | Transaction service

   ...and increase the value of the Total transaction lifetime timeout setting from 120 to 0 (unlimited) or 600 (10 minuites).

2. Increase the maximum number of threads allowed in the thread pool used by the web container. Click...
   Servers | Server Types | WebSphere appservers | portal_server | Thread pools | WebContainer

   ...and set the value of the Maximum Size field to 100 threads.

3. Log on as DB User and increase number of transaction logs.

4. When importing files, a temporary directory is used to store the files during the upload process. If the size of the uploaded files exceeds the available disk space for the temporary directory, the import operation fails. When uploading large files, ensure that there is sufficient disk space to accommodate the import. The location of the temporary directory is specified by the property...
   jcr.binaryValueFileDir

   ...in...

   WP_PROFILE/PortalServer/jcr/lib/com/ibm/icm/icm.properties

5. IBM recommends disabling JCR text search indexing on the HCL WebSphere Portal server before exporting or importing large libraries to reduce the load on the database.

   Log in to the WAS console and click...

   Resources > Resource Environment > Resource Environment Providers > WCM WCMConfigService > Custom properties

   Set jcr.textsearch.enabled to false.

   Restart the server after changing these settings. After completing exporting or importing the library enable JCR text search again. It can take some time to rebuild the indexes.

   You can also set property in icm.properties.

6. Update the maximum Java heap size used by the portal application server on the subscriber server

7. Increase file upload size. Go to...
   Resources | Resource Environment | Resource Environment Providers | WCM WCMConfigService | Custom properties

   ...and for the property...

   resource.maxUploadSize

   ...specify a value in megabytes corresponding to the size of the largest file you want to allow to be imported. For example, if you do not want to allow files larger than 34 MB to be imported, set a value of 34. IBM recommends not setting value that exceeds 100 MB. Absolute maximum is 512 MB.

8. Increase max cache object size. For the property...
   resourceserver.maxCacheObjectSize

   ...specify a value of 300 KB or less.

9. Add the transaction.sync.remove property, and specify a value of true.

10. Click...
    Servers | Server Types | WebSphere appservers | portal_server | Server infrastructure | Administration | Custom Properties

    ...and add the property...

    protocol_http_large_data_inbound_buffer

    For the value specify the maximum file size in bytes. This value should correspond to the value you set for the property...

    resource.maxUploadSize

    ...in the WCM WCMConfigService service.

    Note that the protocol_http_large_data_inbound_buffer property uses bytes. So if you specified a value of 34 MB for the resource.maxUploadSize property, you would specify a value of 35651584 bytes for the protocol_http_large_data_inbound_buffer property.

11. Click...
    Resources | JDBC | Data sources | datasource_name | Custom properties

12. Specify the fullyMaterializeLobData property with a value of false.

13. Click...
    Resource | JDBC | Data sources | datasource_name | Connection pool properties

    ...and set pool size to 150.

14. Increase the maximum number of database collections allowed for the application server by increasing the value of the Maximum connections field to a value greater than the default 50 connections.

15. Increase number of prepared statements to 80

    If the portal's policy cache manager indicates that a number of web container threads are hung, to reduce the database connections and load times, and help prevent threads from hanging, from WP CacheManagerService, set..

    cacheinstance.com.ibm.wps.policy.services.PolicyCacheManager.lifetime = -1

16. SQL Server has a lock timeout that could have some bearing on premature error on waiting for a lock in SQL SERVERS: SET LOCK_TIMEOUT 1800;

17. WCM can hold users lock that will not allow empty-portal and may have some impact on WCM lib deletion. There are ways to troubleshoot this issue.

18. Optional: Reset the web content event log

19. Disable rcs and large object off load on network interfaces.

20. Change power options from energy saving to high performance.

Cumulative Fix 20

With the release of CF 19, IBM transitioned the JCR delete code to using the asynchronous deletes added to the 8.5 code. Deletes were by in large much faster, but actual freeing of space took some time to complete, as the deletes are processed in the background. RSI's NC portal had over 460K Nodes waiting to be deleted in the background, compared to only ~225K active nodes. Freeing" up the deleted space would reduce the DB size by over 50%.

As of CF20, IBM backported the JCR Utility to force the cleanup of the deleted items.

The L3 recommendation is to:

1. Create the following index in the Dev server:
   CREATE INDEX ICMSTJCRREMHLPIX3 ON JCR.ICMSTJCRREMOVEHLP ( STATUS, WSID, ORIGWSID, TIID, LID, VID )

2. Install CF20

3. Run the ConfigEngine Task on the primary node:
   jcr-complete-asynchronous-deletes

To monitor the progress of the utility...

SELECT COUNT(1) FROM JCR.ICMSTJCRREMOVEHLP WITH (NOLOCK);

When the value becomes zero, the utility should be complete and the storage reclaimed.