Syndication troubleshooting

Syndication troubleshooting

If we encounter issues when syndicating, there are some common methods available to troubleshoot these issues.

Common issues

Issue Solution
Unable to reach host Ping the syndicator and the subscriber. Try with IP address rather than the domain name.
Syndicator becomes unresponsive during syndication Syndication can require a large amount of resources to run successfully. Consequently, if the server is performing other tasks at the same time as syndication, the process of syndication might slow or stop altogether. Schedule the syndication to occur at times when the server load is at its lowest.
Syndicator status hangs on "Pending", or "Pending, Active"
If we are attempting to update or rebuild a syndicated library containing large number of items, the syndicator status might hang on "Pending", or "Pending, Active". This can occur because the syndicator keeps retrying to syndicate when some items fail to syndicate to the subscriber, or when a system timeout occurs on the subscriber when saving data.
Improving the performance of the database can help avoid these situations. For example, two of the database attributes that DB2 relies upon to perform optimally are the database catalog statistics and the physical organization of the data in the tables. Catalog statistics must be recomputed periodically during the life of the database, particularly after periods of heavy data modifications (inserts, updates, and deletes) such as a population phase. To fix this, run "Runstats" on the JCR database before and after syndication. The DB2 runstats command is used to count and record the statistical details about tables, indexes, and columns. See database performance for information on using the "Runstats" task.
Due to the heavy load of computing these statistics, it is recommend that this maintenance occurs during off hours, periods of low demand, or when the portal is offline.
Time-outs during syndication Time-outs during syndication are often caused by the failure of large items to be saved. Increasing the total transaction lifetime timeout setting of the IBM WebSphere Portal server can address this issue. The total transaction lifetime timeout setting of the subscriber must be at least the same as the syndicator.
The total transaction lifetime timeout setting is changed using the WAS console.
Go to...
Servers | Server Types | WebSphere application servers | portal_server | Container Services | Transaction Service

Subscriber becomes unresponsive during syndication If we are attempting to syndicate a library containing more than 10000 items, the subscriber machine might become unresponsive during the syndication operation. This action can occur due to an insufficient Java heap size setting on the subscriber.
Update the maximum Java heap size used by the portal application server on the subscriber machine, complete the following steps:

In the WAS console, click System administration > dmgr > Java and Process Management > Process Definition > Java Virtual Machine.

Update the value in the Maximum Heap Size field. A value of at least 1024 MB is recommended.

Click OK, and then save the changes.

In addition, ensure that we have at least as much swap space allocated on the subscriber machine as there is physical memory.
500 errors on ext2 and ext3 versions of Linux
If we receive 500 errors on ext2 and ext3 versions of Linux, you exceeded the number of children that a parent folder can hold. We cannot store more than 32768 children under one folder on ext2 and ext3 versions of Linux. Move some content items out of the affected site area to another site area. So that none of the site areas contain more than 32768 children under one folder and then try syndicating again. We can move the content items back to the correct site areas after syndication is complete.

Other solutions

Option Details
Reset the web content event log. To assist in the troubleshooting process, we can reset the web content event log.
Enable the "deployment.enableReport" setting on the subscriber.
We can update the WCM WCMConfigService service and specify the deployment.enableReport property with a value of true. This update enables high level reporting of syndication to the SystemOut log on the subscriber server. It provides a summary of items that were processed, and which items failed syndication to help troubleshoot syndication issues.

WCM syndication errors parent issue

Checking portal log and seeing...
Could not save item with id DepRef(id:c6e601804d873d73a9a0fbb569f96be1 type: com.aptrix.pluto.site.SiteArea nonDraft:true draft:false purged:false parentId:7fab41004d873d72a994fbb569f96be1 timeStamp:1238183752676 stateUpdate: false versions:null moved: false) because it could not find its parent.
Could not save item with id DepRef(id:b25639004d873d74a9a6fbb569f96be1 type: com.aptrix.pluto.site.SiteArea nonDraft:true draft:false purged:false parentId:7fab41004d873d72a994fbb569f96be1 timeStamp:1238183753041 stateUpdate: false versions:null moved: false) because it could not find its parent.
Could not save item with id DepRef(id:88e9dd004d873d75a9acfbb569f96be1 type: com.aptrix.pluto.site.SiteArea nonDraft:true draft:false purged:false parentId:7fab41004d873d72a994fbb569f96be1 timeStamp:1238183753401 stateUpdate: false versions:null moved: false) because it could not find its parent.
Could not save item with id DepRef(id:63a99e804d873d76a9b2fbb569f96be1 type: com.aptrix.pluto.site.SiteArea nonDraft:true draft:false purged:false parentId:7fab41004d873d72a994fbb569f96be1 timeStamp:1238183753764 stateUpdate: false versions:null moved: false) because it could not find its parent.
Could not save item with id DepRef(id:53de8a004d873d77a9b8fbb569f96be1 type: com.aptrix.pluto.site.SiteArea nonDraft:true draft:false purged:false parentId:7fab41004d873d72a994fbb569f96be1 timeStamp:1238183754165 stateUpdate: false versions:null moved: false) because it could not find its parent.

Log on to to IBM Support Tools on syndicator
Go to XPath Query Analyzer and search for parentID. For example...
//element(*, ibmcontentwcm:siteArea)[@jcr:uuid = '7fab41004d873d72a994fbb569f96be1']

Found:
/contentRoot/icm:libraries[6]/Content/professions_site/professions/accounting(7fab41004d873d72a994fbb569f96be1)@1076052003

Go to IBM Support Tools on subscriber
Go to XPath Query Analyzer and search for same parentID. Not found.
On subscriber, delete site area and all children for site area.
On Syndicator, edit site area and all site area children. Save each to update time stamp.
Re-syndicate using "Update" button.

Work with failed items
From time to time items fail to syndicate. Use the failed items view to review a list of failed items and then run syndication again after you fix the issue.

Log on to the syndicator as an administrator.

Click...
Administration | Portal Content | Syndicators

The number of failed items for a syndicator are displayed in the Failed Items column. Click the number of failed items to open the Failed Items view.

Each failed item for the selected syndicator is displayed in the Failed Items view, with information about how the appropriate action required to fix the issue.

The Root and Impact columns are used to find the root cause of a syndication failure, and what secondary items are impacted by the root cause.

The Important Items tab can also be used to narrow down which items are the most crucial to fix.

After you identified and fixed the issues, we can click Retry to initiate syndication for individual items, or use the Retry All in the Important Items tab to try to syndicate all failed items. We can also choose to update or rebuild a syndication relationship.

Example 1
Some WCM items are not syndicating from wcm-01 --> wcm-02.
SystemOut.log has
# grep DepRef SystemOut.log | cut -d" " -f7-100 | sort | uniq
Could not save item with id DepRef(id:127e0f804d814af39094dbb569f96be1 type: com.aptrix.pluto.site.SiteArea nonDraft:true draft:false purged:false parentId:edad42004d81483d9076dbb569f96be1 timeStamp:1238016447593 stateUpdate: false versions:null moved: false) because it could not find its parent.
Could not save item with id DepRef(id:1a6edb004d8158799120dbb569f96be1 type: com.aptrix.pluto.site.SiteArea nonDraft:true draft:false purged:false parentId:edad42004d81483d9076dbb569f96be1 timeStamp:1238017956153 stateUpdate: false versions:null moved: false) because it could not find its parent.
...

I went into IBM Support Tools on wcm-02 and ran the following test query...
//element(*, ibmcontentwcm:siteArea)[@jcr:uuid = 'b745ee804d5009f199bc9b86d280d817']

This returned...
b745ee804d5009f199bc9b86d280d817 /contentRoot/icm:libraries[8]/Content/companies site(b745ee804d5009f199bc9b86d280d817)@2034530628

I then tried query using some of the ids. No surprise, they could not be found...
was_profile_root/PortalServer/wcm/shared/app/config/wcmservices/WCMConfigService.properties

On both syndicator and subscriber I disabled automatic syndication, and rebuilt syndication relationship. Here are results:
Subscriber ID: 78cac3004d90851db616f72461e8faba Name: wcm-02SubLive Description: None Subscriber URL: http://wcm-02.int.myco.com:10040/wps/wcm/connect/?MOD=Subs Syndicator Name: wcm-02SyndLive Syndicator ID: 6692c7804d7209418571bf1eec0f03e1 Syndicator URL: http://wcm-01.int.myco.com:10040/wps/wcm/connect/?MOD=Synd Current State: 7c9649004d50079a99a49b86d280d817:1238605822693,29b855004ba24a639624d71c959cad1d:1238601233357 Enabled: true Type Full update Started Apr 1, 2014 1:12:56 PM Finished Apr 1, 2014 1:24:45 PM Result Update completed with errors Detail IWKSY1030X: Syndication completed with 5 item related errors. Subscriber's old state ICE-ANY Subscriber's new state 7c9649004d50079a99a49b86d280d817:1238605822693,29b855004ba24a639624d71c959cad1d:1238601233357 Update Summary 45736 total updates sent 45695 updates succeeded 41 updates failed Update Details 44218 item updates attempted 43881 items did not require updating 3 items created 324 items updated 10 updates failed 1518 item removals attempted 1479 items were already removed 8 items removed 31 item failed to be removed

On wcm-02, capturing invalid items from logs...
grep re-creating SystemOut.log > recreate.log

Problem uuids
//element(*, ibmcontentwcm:siteArea)[@jcr:uuid = 'edad42004d81483d9076dbb569f96be1']
//element(*, ibmcontentwcm:siteArea)[@jcr:uuid = 'cc8647804d81494a907cdbb569f96be1']
//element(*, ibmcontentwcm:siteArea)[@jcr:uuid = '127e0f804d814af39094dbb569f96be1']
//element(*, ibmcontentwcm:siteArea)[@jcr:uuid = '633c58004d814c1a90aadbb569f96be1']
//element(*, ibmcontentwcm:siteArea)[@jcr:uuid = '1a6edb004d8158799120dbb569f96be1']
//element(*, ibmcontentwcm:siteArea)[@jcr:uuid = 'b71253004d92ee058d22dfb569f96be1']
//element(*, ibmcontentwcm:webContent)[@jcr:uuid = '822a16004d93e1f7bff8ffb569f96be1'] ???
//element(*, ibmcontentwcm:webContent)[@jcr:uuid = '098123004d967f858a0bbe234711edaf']
//element(*, ibmcontentwcm:webContentLink)[@jcr:uuid = 'a93287804d93dbfabf63ffb569f96be1']

Parent id of each of the above is different on wcm-02. We deleted each of the above items on wcm-02 and re-ran syndication.
JCR explorer search on wcm-01
//element(*, ibmcontentwcm:webCategory)[@jcr:uuid = '08da99004d92f6a28e23dfb569f96be1']

That finds...
08da99004d92f6a28e23dfb569f96be1 /contentRoot/icm:libraries[5]/Taxonomies/display_option/featured/featured profession for landing(08da99004d92f6a28e23dfb569f96be1)@1829596429

JCR explorer search on wcm-02
//element(*, ibmcontentwcm:webCategory)[@jcr:uuid = '08da99004d92f6a28e23dfb569f96be1']

On wcm-01:

//element(*, ibmcontentwcm:siteArea)[@jcr:uuid = 'b71253004d92ee058d22dfb569f96be1']

/contentRoot/icm:libraries[6]/Content/other_content/company gold member site area(b71253004d92ee058d22dfb569f96be1)@2057468578

Unlock the wcm-02 libraries...
http://wcm-02:10040/wps/wcm/connect?MOD=UnlockLibrary&library=MyCo_Content_Library
http://wcm-02:10040/wps/wcm/connect?MOD=UnlockLibrary&library=MyCo_Design_Library

Update and save "Other Content" site area on wcm-01. Putting fresh timestamp, and save on item, forces resyndication.
Did the same thing for...
[4/2/09 13:08:14:435 EDT] 000003b6 PackageProces W IWKPD1029X: Could not store doc: featured industry for landing, IdRef: {020380004d92f8748e74dfb569f96be1, com.aptrix.pluto.cmpnt.MenuCmpnt} contentRoot/icm:libraries[5]/Taxonomies/display_option/featured/featured profession for landing

To fix...

Install IBM Support Tools on wcm-02

Use IBM Support Tools to run queries on wcm-02 against bad items. For example...
//element(*, ibmcontentwcm:webCategory)[@jcr:uuid = '8cd9d0804d544ba6ba27be3ed48f2333']
/contentRoot/icm:libraries[5]/Taxonomies/professions_taxonomy/professions/finance/sales

//element(*, ibmcontentwcm:webCategory)[@jcr:uuid = '13a4da804c01a0a49ec09f0de1b348e6']
/contentRoot/icm:libraries[5]/Taxonomies/professions_taxonomy/professions/nonprofit/consulting

//element(*, ibmcontentwcm:webCategory)[@jcr:uuid = '8d16db804c01a0589db09d0de1b348e6']
/contentRoot/icm:libraries[5]/Taxonomies/professions_taxonomy/professions/manufacturing

//element(*, ibmcontentwcm:webCategory)[@jcr:uuid = 'd868d2804c01a0c89f389f0de1b348e6']
/contentRoot/icm:libraries[5]/Taxonomies/professions_taxonomy/professions/personal services

//element(*, ibmcontentwcm:webCategory)[@jcr:uuid = '386277804c01a0cf9f509f0de1b348e6']
/contentRoot/icm:libraries[5]/Taxonomies/professions_taxonomy/professions/project management

//element(*, ibmcontentwcm:webCategory)[@jcr:uuid = '3a0bbe804d80d7859222bfddea9c21ed']
/contentRoot/icm:libraries[5]/Taxonomies/professions_taxonomy/professions/math and science

//element(*, ibmcontentwcm:webCategory)[@jcr:uuid = '5d8546804d80d823923bbfddea9c21ed']
/contentRoot/icm:libraries[5]/Taxonomies/professions_taxonomy/professions/operations

Delete bad items on wcm-02

Edit and save items on wcm-01

Resyndicate

We now have a clean syndication from wcm-01 --> wcm-02
Started Apr 2, 2014 5:55:24 PM Finished Apr 2, 2014 5:55:35 PM Result Update succeeded Detail IWKSY1029X: Syndication completed with no errors. Subscriber's old state 7c9649004d50079a99a49b86d280d817:1238705375303,29b855004ba24a639624d71c959cad1d:1238702912684 Subscriber's new state 7c9649004d50079a99a49b86d280d817:1238705787941,29b855004ba24a639624d71c959cad1d:1238709255196 Update Summary 150 total updates sent 150 updates succeeded 0 updates failed Update Details 141 item updates attempted 52 items did not require updating 12 items created 77 items updated 0 updates failed 9 item removals attempted 0 items were already removed 9 items removed 0 item failed to be removed

Parent Syndication
Related tasks:

Set service configuration properties
Reset the web content event log

Issue	Solution
Unable to reach host	Ping the syndicator and the subscriber. Try with IP address rather than the domain name.
Syndicator becomes unresponsive during syndication	Syndication can require a large amount of resources to run successfully. Consequently, if the server is performing other tasks at the same time as syndication, the process of syndication might slow or stop altogether. Schedule the syndication to occur at times when the server load is at its lowest.
Syndicator status hangs on "Pending", or "Pending, Active"	If we are attempting to update or rebuild a syndicated library containing large number of items, the syndicator status might hang on "Pending", or "Pending, Active". This can occur because the syndicator keeps retrying to syndicate when some items fail to syndicate to the subscriber, or when a system timeout occurs on the subscriber when saving data. Improving the performance of the database can help avoid these situations. For example, two of the database attributes that DB2 relies upon to perform optimally are the database catalog statistics and the physical organization of the data in the tables. Catalog statistics must be recomputed periodically during the life of the database, particularly after periods of heavy data modifications (inserts, updates, and deletes) such as a population phase. To fix this, run "Runstats" on the JCR database before and after syndication. The DB2 runstats command is used to count and record the statistical details about tables, indexes, and columns. See database performance for information on using the "Runstats" task. Due to the heavy load of computing these statistics, it is recommend that this maintenance occurs during off hours, periods of low demand, or when the portal is offline.
Time-outs during syndication	Time-outs during syndication are often caused by the failure of large items to be saved. Increasing the total transaction lifetime timeout setting of the IBM WebSphere Portal server can address this issue. The total transaction lifetime timeout setting of the subscriber must be at least the same as the syndicator. The total transaction lifetime timeout setting is changed using the WAS console. Go to... Servers \| Server Types \| WebSphere application servers \| portal_server \| Container Services \| Transaction Service
Subscriber becomes unresponsive during syndication	If we are attempting to syndicate a library containing more than 10000 items, the subscriber machine might become unresponsive during the syndication operation. This action can occur due to an insufficient Java heap size setting on the subscriber. Update the maximum Java heap size used by the portal application server on the subscriber machine, complete the following steps: In the WAS console, click System administration > dmgr > Java and Process Management > Process Definition > Java Virtual Machine. Update the value in the Maximum Heap Size field. A value of at least 1024 MB is recommended. Click OK, and then save the changes. In addition, ensure that we have at least as much swap space allocated on the subscriber machine as there is physical memory.
500 errors on ext2 and ext3 versions of Linux	If we receive 500 errors on ext2 and ext3 versions of Linux, you exceeded the number of children that a parent folder can hold. We cannot store more than 32768 children under one folder on ext2 and ext3 versions of Linux. Move some content items out of the affected site area to another site area. So that none of the site areas contain more than 32768 children under one folder and then try syndicating again. We can move the content items back to the correct site areas after syndication is complete.

Option	Details
Reset the web content event log.	To assist in the troubleshooting process, we can reset the web content event log.
Enable the "deployment.enableReport" setting on the subscriber.	We can update the WCM WCMConfigService service and specify the deployment.enableReport property with a value of true. This update enables high level reporting of syndication to the SystemOut log on the subscriber server. It provides a summary of items that were processed, and which items failed syndication to help troubleshoot syndication issues.