Populating a large user set

Populate the Profiles database from an LDAP directory with a large user population.

In very large organizations, the number of users in the LDAP directory exceeds the capacity of the Tivoli Directory Integrator assembly lines for Profiles. To overcome this restriction, we can populate the database by using manual TDI assembly lines. We cannot use the Profiles population wizard.

For related information and details, see Tivoli Directory Integrator help.

Limits with large user sets

The LDAP administrator can change the LDAP size limit. The capacity of the standard assembly lines provided with Connections is 100,000 users. In some cases, we can modify the maximum number of entries returned from the LDAP or adjust the source_ldap_page_size parameter in profiles_tdi.properties. For example, set the parameter to the maximum number of records the LDAP repository will return, using the following sample statement:

source_ldap_page_size=1000

If we receive the following, adjust the source_ldap_page_size parameter in profiles_tdi.properties.

LDAP: error code 4 - Sizelimit Exceeded

If neither of these alternatives is successful, use a special set of assembly lines to populate the Profiles database from the LDAP directory.

Alternative population process

If we have a very large set of data, set the source_ldap_iterate_with_filter property in profiles_tdi.properties to true. This uses the collect_ldap_dns_generator.js file to retrieve search criteria for a batch of records. The batch is always smaller than the limit of the LDAP retrieval.

The collect_ldap_dns_generator.js file constructs a search filter with a portion of UIDs but does not modify the search base. It is data-specific so modify it for our own deployment. Modify suppliesSearchBase() or suppliesSearchBase(), depending on which filter is used in the LDAP retrieval.

If one of the filters is changed to return true (in the supplied file, suppliesSearchBase returns true), the corresponding function, either getNextSearchBase() or getNextSearchFilter(), is called in iterations. Each time the function is called it returns a string with the next search base or filter to use. When it reaches the end of the batch, it returns null.

In the sample file, the UID is examined over a range of its first characters. The process first uses some special characters and then examines the first two characters of the UID string, or example aa*, ab*, and so on. After it reaches zz* it returns null and the collect_dns assembly line stops processing. We can then run populate_from_dn_file.

Parent topic:
Populate peopledb