Active Directory Change Detection Connector

IBM Tivoli Directory Integrator

Active Directory Change Detection Connector

The Active Directory Change Detection Connector (hereafter referred to as ADCD Connector) is a specialized instance of the LDAP Connector. It reports changed Active Directory objects so that other repositories can be synchronized with Active Directory.

The LDAP protocol is used for retrieving changed objects.

When run the Connector reports the object changes necessary to synchronize other repositories with Active Directory regardless of whether these changes occurred while the Connector has been offline or they are happening as the Connector is online and operating.

This connector also supports Delta Tagging, at the Entry level only.

The ADCD Connector operates in Iterator mode.

This component is not available in the TDI 7.1 General Purpose Edition.

Tracking changes in Active Directory

Active Directory does not provide a Changelog as IBM Directory Server and some other LDAP Servers do.

The ADCD Connector uses the uSNChanged Active Directory attribute to detect changed objects.

Each Active Directory object has an uSNChanged attribute that corresponds to a directory-global USN (Update Sequence Number) object. Whenever an Active Directory object is created, modified or deleted, the global sequence object value is increased, and the new value is assigned to the object's uSNChanged attribute.

On each AssemblyLine iteration (each call of the getNextEntry() Connector's method) it delivers a single object that has changed in Active Directory. It delivers the changed Active Directory objects as they are, with all their current attributes and also reports the type of object change - whether the object was updated (added or modified) or deleted. The Connector does not report which attributes have changed in this object and the type of attribute change.

Synchronization state is kept by the Connector and saved in the User Property Store - after each reported changed object the Connector saves the USN number necessary to continue from the correct place in case of interruption and restart; when started, the ADCD Connector reads this USN value from the IBM TDI's User Property Store stored from the most recent ADCD Connector session.

Information from MSDN about tracking changes in Active Directory can be found here, and information about polling for changes using the uSNChanged attribute is here.

Deleted objects in Active Directory

When an object is deleted from the directory, Active Directory performs the following steps:

The object's isDeleted attribute is set to TRUE. Objects where isDeleted==TRUE are known as tombstones (not related to TDI tombstones).
All attributes that are not needed by Active Directory are removed. A few key attributes, including objectGUID, objectSID, nTSecurityDescriptor, and uSNChanged are preserved.
Moves the tombstone to the Deleted Objects container, which is a hidden container within the directory partition.

Tombstones or deleted objects are garbage collected some time after the deletion takes place. Two settings on the "cn=Directory Service,cn=Windows NT,cn=Service,cn=Configuration,dc=ForestRootDomain" object determine when and which tombstones are deleted:

The "garbage collection interval" determines the number of hours between garbage collection on a domain controller. The default setting is 12 hours, and the minimum setting is 1 hour.
The "tombstone lifetime" determines the number of days that tombstones persist before they are vulnerable to garbage collection. The default setting is 60 days, and the minimum setting is 2 days.

The above specifics imply the following requirements for synchronization processes that have to handle deleted objects:

Synchronization has to be run on intervals shorter than the "tombstone lifetime" Active Directory setting.
The objectGUID attribute has to be used for object identifier during synchronization. The object's distinguishedName attribute which uniquely identifies the position of an object in the directory tree, cannot be used because after the object is deleted it changes its place in the directory tree - it is moved in the Deleted Objects container and its old distinguished name is irrevocably lost. The objectGUID attribute is however never changed. When a deleted object is found during synchronization, a search in the other repository for an object with the same objectGUID should be made and the found object should be deleted.

Moved objects in Active Directory

When an object is moved from one location of the Active Directory tree to another, its distinguishedName attribute changes. When this object change is detected based on the new increased value of the object's uSNChanged attribute, this change looks like any other modify operation - there is no information about the object's old distinguished name.

A synchronization process that has to handle moved objects properly should use the objectGUID attribute - it doesn't change when objects are moved. A search by the objectGUID attribute in the repository which is synchronized will locate the proper object and then the old and new distinguished names can be compared to check if the object has been moved.

Use objectGUID as the object identifier

When tracking changes in Active Directory the objectGUID attribute should be used for object identifier and not the LDAP distinguished name. This is so because the distinguished name is lost when an object is deleted or moved in Active Directory. The objectGUID attribute is always preserved, it never changes and can be used to identify an object.

When the ADCD Connector reports that an entry is changed, a search by objectGUID value should be performed in the other repository to locate the object that has to be modified or deleted. This means that the objectGUID attribute should be synchronized and stored into the other repository.

Change detection

Change detection mechanism

The ADCD Connector detects and reports changed objects following the chronology of the uSNChanged attribute values: changed objects with lower uSNChanged values will be reported before changed objects with higher uSNChanged values.

The Connector executes an LDAP query of type (usnChanged>=X) where X is the USN number that represents the current synchronization state. Sort and Page LDAP v3 controls are used with the search operation and provide for chronology of changes and ability to process large result sets. The Show Deleted LDAP v3 request control (OID "1.2.840.113556.1.4.417") is used to specify that search results should include deleted objects as well.

The ADCD Connector consecutively reports all changed objects regardless of interruptions, regardless of when it is started and stopped and whether the changes happened while the Connector was online or offline. Synchronization state is kept by the Connector and saved in the User Property Store - after each reported changed object the Connector saves the USN number necessary to continue from the correct place in case of interruption and restart.

The Connector will signal end of data and stop (according to the timeout value) when there are no more changes to report.

When there are no more changed Active Directory objects to retrieve, the Active Directory Connector cycles, waiting for a new object change in Active Directory. The Sleep Interval parameter specifies the number of seconds between two successive polls when the Connector waits for new changes. The Connector loops until a new Active Directory object is retrieved or the timeout (specified by the Timeout parameter) expires. If the timeout expires, the Active Directory Connector returns a null Entry, indicating there are no more Entries to return. If a new Active Directory object is retrieved, it is processed as previously described, and the new Entry is returned by the Active Directory Connector.

In older versions of the Connector, it reported both added and modified entries as updated. Currently, the Connector differentiates between add and modify and reports each operation separately (for details see Offline and Paged results cases.)

The ADCD Connector delivers changed Active Directory objects as they are, with all their current attributes. It does not determine which object attributes have changed, nor how many times an object has been modified. All intermediate changes to an object are irrevocably lost. Each object reported by the Active Directory Connector represents the cumulative effect of all changes performed to that object. The Active Directory Connector, however, recognizes the type of object change that has to be performed on the replicated data source and reports whether the object must be updated or deleted in the replicated data source.

We can retrieve only objects and attributes that we have permission to read. The Connector does not retrieve an object or an attribute that we do not have permission to read, even if it exists in Active Directory. In such a case the ADCD Connector acts as if the object or the attribute does not exist in Active Directory.

Offline and Paged results cases

When the Connector is offline or when Paged results is enabled and an initial search request is made but the page containing modified entry is not retrieved yet, multiple changes made to that entry are merged. In other words the Connector receives only one entry containing the results of all operations that have been applied on it.

In these cases when an entry is added and then deleted in Active Directory the Connector will report "delete" operation for entries that have not been added to the repository being synchronized with Active Directory. This is not a serious restriction because IBM TDI 7.1's Delete Connector mode first checks if the entry to be deleted exists and if it does not exist, the "On No Match" hook is called - this is where we can place code to handle/ignore such unnecessary deletes.

Another scenario is when entry is added and then modified. In that case the Connector will report an "add" operation for that entry, and the entry will contain all the changes made to it after the adding.

In all other possible cases the return entry will contain all the changes and it will be tagged with the last operation made to it.

Using the Active Directory Change Detection Connector

Each delivered entry by the Connector contains the changeType attribute whose value is either "add" (for newly created objects), "modify" (for modified objects) or "delete" (for deleted Active Directory objects). Each entry also contains 2 attributes that represent the objectGUID value:

attribute objectGUID - contains a 16-byte byte array that represents the 128-bit objectGUID of the corresponding Active Directory object.
attribute objectGUIDStr - contains the string representation of the hexadecimal value of the 128-bit objectGUID. It is delivered in the format {xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx}, where each x represents a hexadecimal digit.

If we need to detect and handle moved or deleted objects, you must use the objectGUID value as object identifier instead of the LDAP distinguished name. The LDAP distinguished name changes when an object is moved or deleted, while the objectGUID attribute always remains unchanged. Store the objects' objectGUID attribute in the replicated data source and search by this attribute to locate objects.

Deleted objects in Active Directory live for a configurable period of time (60 days by default), after which they are completely removed. To avoid missing deletions, perform incremental synchronizations more frequently.

The ADCD Connector can be interrupted at any time during the synchronization process. It saves the state of the synchronization process in the User Property Store of the IBM TDI (after each Entry retrieval), and the next time the Active Directory Connector is started, it successfully continues the synchronization from the point the Active Directory Connector was interrupted.

Authentication of the Connector to the directory

Different versions of the LDAP protocol support different authentication methods. LDAP v2 supports three: Anonymous, Simple, Kerberos v4. LDAP v3 supports: Anonymous, Simple and SASL authentication. The AD Change Detection Connector supports Anonymous, Simple and SASL authentication.

Anonymous

If this authentication method is set, it means that the server, to which a client is connected, does not know or care who the client is. The server allows such client to access data that is configured for non-authenticated users. The ADCD connector automatically specifies this authentication method if no username is supplied. If this type of authentication is chosen and a username is specified, then the ADCD connector automatically sets the authentication method to Simple.

Simple

In this case the LDAP server requires that the client (ADCD Connector) sends the fully qualified distinguished name and the client password in cleartext. This is a problem, because the password may be read from the network. You may use this type of authentication in combination with the SSL protocol to avoid exposing the password, if the LDAP server supports it. If Simple mode is specified and neither a username, then mode is automatically set to Anonymous by the ADCD Connector. If Anonymous mode is chosen, but a username is specified, then the mode is set to Simple by the ADCD Connector.

SASL

The client (the ADCD Connector) will use a Simple Authentication and Security Layer (SASL) authentication method when connecting to the LDAP Server. In order to use this authentication mechanism, you need to configure the SASL mechanism to be used manually by specifying additional JNDI parameters using the Extra Provider Parameters option. For more information on SASL authentication, the SASL mechanisms supported by JNDI, and configurable SASL parameters available to JNDI, refer to the following URL: http://java.sun.com/products/jndi/tutorial/ldap/security/sasl.html.

Not all directory servers support all SASL mechanisms and in some cases do not have them enabled by default. Check the documentation and configuration options for the directory server you are connecting to for this information.

Here is an example of the parameters to add to the Extra Provider Parameters parameter to configure the LDAP Connector to use DIGEST-MD5 when the SASL authentication method is selected:

java.naming.security.authentication:DIGEST-MD5

Error flows

A PartialResultException is thrown if the returned entry contains referrals - an Active Directory server is able to generate referrals to external domains. However it is the client's responsibility to be able to follow these referrals because Active Directory does not perform that; it does not support the Manage Referral control. Therefore, the Active Directory Change Detection Connector cannot follow referrals either.
An Exception is thrown if the LDAP URL parameter is missing.
The Connector will throw an exception when there is a problem in storing the USN value in the System Store.

Configuration

The Connector needs the following parameters:

LDAP URL

Specifies the LDAP URL of the Active Directory service we want to access. The LDAP URL has the form ldap://hostname:port or ldap://server_IP_address:port. For example, ldap://localhost:389

The default LDAP port number is 389. When using SSL, the default LDAP port number is 636.

Login username

Specifies the distinguished name used for authentication to the service. For example, cn=administrator,cn=users,dc=your_domain,dc=com.

If we use Anonymous authentication, leave this parameter blank.

Login password

Specifies the credentials (password).

If you use Anonymous authentication, leave this parameter blank.

Authentication Method

Specifies the authentication method to be used. Possible values are:

Anonymous (use no authentication)
Simple (use weak authentication (cleartext password))

Use SSL

Specifies whether to use Secure Sockets Layer for LDAP communication with Active Directory.

Extra Provider Parameters

Allows you to pass a number of extra parameters to the JNDI layer. It is specified as name:value pairs, one pair per line.

Binary Attributes

Specifies a list of parameters that are to be interpreted as binary values instead of strings. The default value for this parameter is objectGUID objectSid.

LDAP Search Base

Specifies the Active Directory sub-tree that is polled for changes. The search base should be an Active Directory Naming Context if detection of deleted objects is required. For example, dc=your_domain,dc=com.

Page Size

Specifies the number of entries per page returned by this request (default value is 500).

Iterator State Key

Specifies the name of the parameter that stores the current synchronization state in the User Property Store of the IBM TDI. This must be a unique name for all parameters stored in one instance of the IBM TDI User Property Store. The Delete button lets you delete this information from the User Property Store.

Start at

Specifies either EOD or 0. EOD means report only changes that occur after the Connector is started. 0 means perform full synchronization, that is, report all objects available in Active Directory Service. This parameter is taken into account only when the parameter specified by the Iterator State Key parameter is not found in the User Property Store.

State Key Persistence

Determines when the Connector's state is written to the System Store. The default (and recommended setting) is End of Cycle, and the choices are:

After read: Updates the System Store when you read an entry from the Active Directory change log, before you continue with the rest of the AssemblyLine.
End of cycle: Updates the System Store with the change log number when all Connectors and other components in the AssemblyLine have been evaluated and executed.
Manual: Switches off the automatic updating of the System Store with this Connector's state information; instead, we will need to save the state by manually calling the ADCD Connector's saveStateKey() method, somewhere in your AssemblyLine.

Use Notifications

Specifies whether to use notification when waiting for new changes in Active Directory. If not enabled, the Connector will poll for new changes.

If enabled, the Connector will not sleep or timeout but instead wait for a Change Notification event (Server Search Notification Control (OID 1.2.840.113556.1.4.528) from the Active Directory server, and the sleep interval and timeout parameters are ignored.

Timeout

Specifies the maximum number of seconds the Connector waits for the next changed Active Directory object. If this parameter is 0, then the Connector waits forever. If the Connector has not retrieved the next changed Active Directory object within timeout seconds, then it returns an empty (null) Entry, indicating that there are no more Entries to return. The default is 5.

Sleep Interval

Specifies the number of seconds the Connector sleeps between successive polls.

Detailed Log

If this field is checked, additional log messages are generated.

Comment

Your comments here.

Changing Timeout or Sleep Interval values will automatically adjust its peer to a valid value after being changed (for example, when timeout is greater than sleep interval the value that was not edited is adjusted to be in line with the other). Adjustment is done when the field editor looses focus.