+

Search Tips   |   Advanced Search

Delta Detection

Security Directory Integrator provides a number of change, or delta, detection mechanisms and tools:

The Delta engine feature reports specific changes all the way down to the individual values of attributes. This fine degree of change detection is also available when parsing LDIF files. Others components are limited to simply reporting if an entire entry is added, modified or deleted.

By selecting the Allow Duplicate Delta keys check box in the Iterator's Delta tab, you indicate that you allow duplicate delta keys, for those cases of long running AssemblyLines which need to process the same entries more than once. This means that duplicate entries can be handled for AssemblyLines that use both Changelog or Change Detection Connectors, Delta mode connectors and the Delta mode stores, when an entry has already been updated. Attention: There is a possibility to have, for example, an AssemblyLine with a number of Changelog and Delta mode Connectors. In this case, if the Delta mode Connector is pointing to the same underlying system as the Changelog Connector, the Delta operation could trigger the Changelog again. As there is no way to differentiate between newly-received changes and those triggered by the Delta engine, you should carefully consider your scenario in order not to enter into an endless loop.

The delta information computed by the Delta engine is stored in the Work Entry object, and depending on the Change Detection component or feature used can be stored as an Entry-Level operation code, at the Attribute-Level or even at the Attribute Value-Level. As an example, set up a File System Connector with the Delta engine feature enabled. Have it iterate over a simple XML document that you can easily modify in a text editor. For example: 

<?xml version="1.0" encoding="UTF-8"?>
<DocRoot>
    <Entry>
        <Telephone>
            <ValueTag>111-1111</ValueTag>
            <ValueTag>222-2222</ValueTag>
            <ValueTag>333-3333</ValueTag>
        </Telephone>
        <Birthdate>1958-12-24</Birthdate>
        <Title>Full-Time SDI Specialist</Title>
        <uid>jdoe</uid>
        <FullName>John Doe</FullName>
    </Entry> 
</DocRoot>
Be sure to use the special map-all attribute map character, the asterisk (*). This is the only Attribute you need in your map to ensure that all Attributes returned are mapped in to the Work entry object. Now add a Script Component with the following code: 
// Get the names of all Attributes in work as a String array
var attName = work.getAttributeNames();
// Print the Entry-level delta op code
task.logmsg(" Entry (" + 
    work.getString( "FullName" ) + ") : " + 
    work.getOperation() );
// Loop through all the Attributes in work
for (i = 0; i < attName.length; i++) {
  // Grab an Attribute and print the Attribute-level op code
	att = work.getAttribute( attName[ i ] );
	task.logmsg("    Att ( " + attName[i] + ") : " + att.getOperation() );
  // Now loop through all the Attribute's values and print their op codes
	for (j = 0; j < att.size(); j++) {
		task.logmsg( "      Val (" + 
                     att.getValue( j ) + ") : " + 
                     att.getValueOperation( j ) );
	}
}
The first time you run this AL, your Script Component code will create this log output: 
12:46:31   Entry (John Doe) : add
12:46:31      Att ( Telephone) : replace
12:46:31        Val (111-1111) : 
12:46:31        Val (222-2222) : 
12:46:31        Val (333-3333) : 
12:46:31      Att ( Birthdate) : replace
12:46:31        Val (1958-12-24) : 
12:46:31      Att ( Title) : replace
12:46:31        Val (Full-Time SDI Specialist) : 
12:46:31      Att ( uid) : replace
12:46:31        Val (jdoe) : 
12:46:31      Att ( FullName) : replace
12:46:31        Val (John Doe) :
Since this entry was not found in the previously empty Delta Store, it is tagged at the entry-level as new. Furthermore, each of its Attributes has a replace code, meaning that all values have changed (which makes sense because the Delta is telling us that this is new data).

Make the following changes to your XML file:

  1. Change the last Telephone number value to 333-3334.
  2. Delete Birthdate.
  3. Add a new Address Attribute.

Your resulting Config should look like this: 

<?xml version="1.0" encoding="UTF-8"?>
<DocRoot>
    <Entry>
        <Telephone>
            <ValueTag>111-1111</ValueTag>
            <ValueTag>222-2222</ValueTag>
            <ValueTag>333-3334</ValueTag>
        </Telephone>
        <Title>Full-Time SDI Specialist</Title>
        <uid>jdoe</uid>
        <FullName>John Doe</FullName>
        <Address>123 Willowby Lane</Address>
    </Entry> 
</DocRoot>
Run your AL again. This time your log output should look like this: 
13:53:22   Entry (John Doe) : modify
13:53:22      Att ( Telephone) : modify
13:53:22        Val (111-1111) : unchanged
13:53:22        Val (222-2222) : unchanged
13:53:22        Val (333-3334) : add
13:53:22        Val (333-3333) : delete
13:53:22      Att ( Birthdate) : delete
13:53:22        Val (1958-12-24) : delete
13:53:22      Att ( uid) : unchanged
13:53:22        Val (jdoe) : unchanged
13:53:22      Att ( Title) : unchanged
13:53:22        Val (Full-Time SDI Specialist) : unchanged
13:53:22      Att ( Address) : add
13:53:22        Val (123 Willowby Lane) : add
13:53:22      Att ( FullName) : unchanged
13:53:22        Val (John Doe) : unchanged
Now the entry is tagged as modify and the Attributes reflect what the modifications for each of them. As we can see, the Birthdate Attribute is marked as delete and Address as add. That's the reason you used the special map-all character for our Input Map. If you had mapped only the Attributes that existed in the first version of this XML document, we would not have retrieved Address when it appeared in the input.

Note especially the last two value entries under the Telephone Attribute, marked as modify. The change to one of the values of this Attribute resulted in two Delta items: a value delete and then an add.

To build a data synchronization AssemblyLine in earlier versions of IBM Security Directory Integrator, you had to script in order to handle flow control. Although you could be receiving adds, modifies and deletes from your change component or feature, a Connector could only be set to one of the two required output modes: Update or Delete. So either you had two Connectors pointing to the same target system and you put script in the Before Execute Hook of each to ignore the entry if its operation code did not match the mode of this component; or you could have a single Connector (either Update or Delete mode) in Passive state, and then control its execution from script code where you checked the operation code. This still meant that even though you knew what had changed in the case of a modified entry, your Update Mode Connector would still read in the original data before writing the changes back to the data source. This can lead to unwanted network or datasource traffic when you are only changing a single value in a multi-valued group-related Attribute containing thousands of values.

Enter the Connector Delta mode.


Parent topic:

Delta mode