Syndication issues

Overview

Some WCM items are not syndicating from wcm_stage --> wcm_prod.
In wcm_prod SystemOut.log, messages like...
# grep DepRef SystemOut.log | cut -d" " -f7-100 | sort | uniq
Could not save item with id DepRef(id:127e0f804d814af39094dbb569f96be1 type: com.aptrix.pluto.site.SiteArea nonDraft:true draft:false purged:false parentId:edad42004d81483d9076dbb569f96be1 timeStamp:1238016447593 stateUpdate: false versions:null moved: false) because it could not find its parent.
Could not save item with id DepRef(id:1a6edb004d8158799120dbb569f96be1 type: com.aptrix.pluto.site.SiteArea nonDraft:true draft:false purged:false parentId:edad42004d81483d9076dbb569f96be1 timeStamp:1238017956153 stateUpdate: false versions:null moved: false) because it could not find its parent.
...

I went into IBM Support Tools on and ran the following test query...
//element(*, ibmcontentwcm:siteArea)[@jcr:uuid = 'b745ee804d5009f199bc9b86d280d817']

This returned...
b745ee804d5009f199bc9b86d280d817 /contentRoot/icm:libraries[8]/Content/companies site(b745ee804d5009f199bc9b86d280d817)@2034530628

I then tried query using some of the above ids. No surprise, they could not be found...
was_profile_root/PortalServer/wcm/shared/app/config/wcmservices/WCMConfigService.properties

...on both wcm_stage and wcm_prod and disabled automatic syndication. I then did a Rebuild. Here are results:
Subscriber ID: 
78cac3004d90851db616f72461e8faba 
 
Name: 
wcm_prodSubLive 
 
Description: 
None  
 
Subscriber URL: 
http://wcm_prod.int.clienty.com:10040/wps/wcm/connect/?MOD=Subs 
 
Syndicator Name: 
wcm_prodSyndLive 
 
Syndicator ID: 
6692c7804d7209418571bf1eec0f03e1 
 
Syndicator URL: 
http://wcm_stage.int.clienty.com:10040/wps/wcm/connect/?MOD=Synd 
 
Current State: 
7c9649004d50079a99a49b86d280d817:1238605822693,29b855004ba24a639624d71c959cad1d:1238601233357 
 
Enabled: 
true 
 
Type 
Full update 
 
Result 
Update completed with errors 
 
Detail 
IWKSY1030X: Syndication completed with 5 item related errors. 
 
Subscriber's old state 
ICE-ANY 
 
Subscriber's new state 
7c9649004d50079a99a49b86d280d817:1238605822693,29b855004ba24a639624d71c959cad1d:1238601233357 
 
Update Summary
45736 total updates sent
    45695 updates succeeded
    41 updates failed

Update Details
44218 item updates attempted
    43881 items did not require updating
    3 items created
    324 items updated
    10 updates failed

1518 item removals attempted
    1479 items were already removed
    8 items removed
    31 item failed to be removed
On wcm -s, capturing invalid items from logs...
grep re-creating SystemOut.log > recreate.log

Problem uuids
//element(*, ibmcontentwcm:siteArea)[@jcr:uuid = 'edad42004d81483d9076dbb569f96be1']
//element(*, ibmcontentwcm:siteArea)[@jcr:uuid = 'cc8647804d81494a907cdbb569f96be1']
//element(*, ibmcontentwcm:siteArea)[@jcr:uuid = '127e0f804d814af39094dbb569f96be1']
//element(*, ibmcontentwcm:siteArea)[@jcr:uuid = '633c58004d814c1a90aadbb569f96be1']
//element(*, ibmcontentwcm:siteArea)[@jcr:uuid = '1a6edb004d8158799120dbb569f96be1']
//element(*, ibmcontentwcm:siteArea)[@jcr:uuid = 'b71253004d92ee058d22dfb569f96be1']
//element(*, ibmcontentwcm:webContent)[@jcr:uuid = '822a16004d93e1f7bff8ffb569f96be1'] ???
//element(*, ibmcontentwcm:webContent)[@jcr:uuid = '098123004d967f858a0bbe234711edaf']
//element(*, ibmcontentwcm:webContentLink)[@jcr:uuid = 'a93287804d93dbfabf63ffb569f96be1']

Parent id of each of the above is different on wcm_prod. We deleted each of the above items on wcm_prod and re-ran syndication.
JCR explorer search on wcm_stage
//element(*, ibmcontentwcm:webCategory)[@jcr:uuid = '08da99004d92f6a28e23dfb569f96be1']

That finds...
08da99004d92f6a28e23dfb569f96be1 /contentRoot/icm:libraries[5]/Taxonomies/display_option/featured/featured profession for landing(08da99004d92f6a28e23dfb569f96be1)@1829596429

JCR explorer search on wcm_prod
//element(*, ibmcontentwcm:webCategory)[@jcr:uuid = '08da99004d92f6a28e23dfb569f96be1']

On wcm_stage:

//element(*, ibmcontentwcm:siteArea)[@jcr:uuid = 'b71253004d92ee058d22dfb569f96be1']

/contentRoot/icm:libraries[6]/Content/other_content/company gold member site area(b71253004d92ee058d22dfb569f96be1)@2057468578

Unlocking the wcm_prod libraries...
http://wcm_prod:10040/wps/wcm/connect?MOD=UnlockLibrary&library=My_Content_Library
http://wcm_prod:10040/wps/wcm/connect?MOD=UnlockLibrary&library=My WCM Design Library
Libs now unlocked on wcm_prod. Resyndicated. Failed. Libs did not lock again, so that is good. Updated and saved "Other Content" site area on wcm_stage. Putting fresh timestamp and save on item forces resyndication.
That worked.
Did the same thing for...

[4/2/09 13:08:14:435 EDT] 000003b6 PackageProces W IWKPD1029X: Could not store doc: featured industry for landing, IdRef: {020380004d92f8748e74dfb569f96be1, com.aptrix.pluto.cmpnt.MenuCmpnt}

contentRoot/icm:libraries[5]/Taxonomies/display_option/featured/featured profession for landing
Need to confirm syndication is working for

wcm_stage --> wcm_stage_02
wcm_stage_02 --> portal-01

Here are offenders on wcm_stage_02: wcm-01SystemOut.log
To fix, we need to:

Install IBM Support Tools on wcm_stage_02
Run queries on wcm_stage_02 against bad items to determine name/location
Delete bad items on wcm_stage_02
Edit and save items on wcm_stage
Resyndicate

We now have a clean syndication from wcm_stage --> wcm_stage_02
Result 
Update succeeded 
 
Detail 
IWKSY1029X: Syndication completed with no errors. 
 
Subscriber's old state 
7c9649004d50079a99a49b86d280d817:1238705375303,29b855004ba24a639624d71c959cad1d:1238702912684 
 
Subscriber's new state 
7c9649004d50079a99a49b86d280d817:1238705787941,29b855004ba24a639624d71c959cad1d:1238709255196 
 
Update Summary
150 total updates sent
    150 updates succeeded
    0 updates failed

Update Details
141 item updates attempted
    52 items did not require updating
    12 items created
    77 items updated
    0 updates failed

9 item removals attempted
    0 items were already removed
    9 items removed
    0 item failed to be removed 