Connectors help you to build your AssemblyLine. Each one is designed to tie a specific data source to our data flow.
The following four sections explain issues with the RAC Connector.
Currently, the Remote Agent Controller (RAC) Connector in Iterator Mode cannot operate with a non-Windows installation of the Agent Controller. This problem is due to the instability of the Agent Controller implementation. Any attempt to run the RAC Connector in Iterator Mode on any non-Windows operating system could cause the Agent Controller process to stop.
If the network connection between the RAC Connector in Iterator mode and the Agent Controller is very slow, the slow network connection may have the following results:
If the RAC Connector is run in AddOnly mode, but it does not register a logging agent in the local Agent Controller, it may be because the RAC cannot locate Agent Controller binaries. The AddOnly mode of the RAC Connector requires that the binaries of the Agent Controller (.dll, .so) be available for the dynamic library loader of the operating system. The preferred way to make the Agent Controller binaries available to the .dll of the operating system is to locate the binaries folder of the Agent Controller in :
We can point to the binaries globally or just for the process of the TDI Server. For example:
LD_LIBRARY_PATH=/AgentController/lib export LD_LIBRARY_PATH
The Agent Controller may be visible using the Log and Trace Analyzer Eclipse tool, but the RAC Connector in Iterator mode may report the following error:
Error: Unable to connect to the Agent Controller.
The reason for this error could be that the Agent Controller installation may not be a new technology Agent Controller. Releases of new technology Agent Controller support the new technology communication protocol. In Iterator mode, the RAC Connector uses the new technology communication protocol, but the Log and Trace analyzer uses the old communication protocol.
This section documents the following issues:
The Axis Easy Web Service Server Connector may generate exceptions if multiple clients are trying to access the server at the same time. The following exception code shows the content of the error:
2007-01-25 13:08:55,828 ERROR [AssemblyLine.AssemblyLines/Square_SimpleClient.753] - [EasyInvokeWebService] at com.ibm.di.fc.webservice.AxisEasyInvokeSoapWS.perform(Unknown Source) at com.ibm.di.server.FunctionComponent.callreply(Unknown Source) at com.ibm.di.server.AssemblyLine.msExecuteNextConnector(Unknown Source) at com.ibm.di.server.AssemblyLine.executeMainStep(Unknown Source) at com.ibm.di.server.AssemblyLine.executeMainLoop(Unknown Source) at com.ibm.di.server.AssemblyLine.executeMainLoop(Unknown Source) at com.ibm.di.server.AssemblyLine.executeAL(Unknown Source) at com.ibm.di.server.AssemblyLine.run(Unknown Source)
On initialization the AxisEasyWSServerConnector opens a server socket to accept connections from clients. In the listening state, the server socket adds each incoming client connection request to an internal queue called a backlog. If client requests arrive at a faster rate than the server program "accepts" them, the backlog starts filling up. Eventually the backlog fills up and newly arrived clients are refused a connection to the server.
Try increasing the value of the Connection Backlog parameter in the server mode connector. The maximum backlog size depends on the platform. For example, on Windows XP Professional the maximum limit for the backlog size is 200, so any backlog size above 200 will have no effect. Slow down the arrival of client connection requests. Use a TDI script to introduce a time delay between clients as they start. The goal is to bring client request arrival speed below the servicing speed of the server.
There is a potential inconsistency across Secure Socket Layer (SSL) clients running TDI7.1. TDI components that use SSL can accept SSL server certificates whose "valid From" dates are not yet valid. Failure to issue an error could be a problem if users expect an exception to be thrown in these cases. This behavior is inconsistent across the platforms.
The JRE being used is accepting not yet "valid From" dates in certificates and is failing to warn the user of invalid certificates, therefore engaging in invalid behavior. If the client system has an earlier system date from the date of the certificate valid date, then the client should not connect to the server over SSL.
Example exception on Solaris when the issue is properly detected by the JRE:
javax.net.ssl.SSLHandshakeException: com.ibm.jsse2.util.h: Certificate not valid yet
The Solaris operating system demonstrates the desired behavior by throwing an exception when facing certificates with not yet valid "valid From" dates. By default, the IBM JRE will not throw the proper exception on any platform except Solaris. The Sun JRE will throw the proper exception on all platforms.
The Trust Manager configured in the JRE must be updated. For IBM JREs we can update the Trust Manager by doing the following:
ssl.TrustManagerFactory.algorithm=
ssl.TrustManagerFactory.algorithm=IbmX509
ssl.TrustManagerFactory.algorithm=PKIX
Some third-party libraries for TDI connectors must be obtained from their proprietary sources, and do not ship with TDI. Using the Configuration Editor, we can configure connectors. However, if a connector is missing its required library (.jar) file, the Connector in the Configuration Editor GUI looks as if it supports all Connector modes. For example, a Connector such as the Domino ChangeDetection Connector may support Iterator mode only, but if you configure the connector in the Configuration Editor with the required .jar file missing, it will show as supporting all other modes, such as Lookup, Update, Delete, etc. To avoid this problem, obtain and supply (to TDI) the library for these connectors. The following connectors have libraries that do not ship with TDI. Obtain the .jar files for these connectors before attempting to configure them in the Configuration Editor GUI:
IBM TDI Component Name | Component Internal Class Name |
---|---|
Domino Change Detection Connector | ibmdi.ChangeDetectionConnector |
Domino Users' Connector | ibmdi.DominoUsersConnector |
Lotus Notes Connector | ibmdi.Notes |
SAP ALE IDoc Connector | ibmdi.SapALEIDocConnector |
TAM Connector | ibmdi.TAM |
Domino AdminP Connector | ibmdi.DominoAdminP |
For information on how to use Connectors, see the section below and the IBM TDI V7.1 Reference Guide.
Before using any of the Domino or Lotus Notes connectors, set environment variables for $PATH and for $LD_LIBRARY_PATH. Add the following two lines to the ibmdisrv and ibmditk scripts. Place the environment variable settings just before the last line of each script:
export PATH={notes.data.dir}:{notes.binary.dir}/notes/latest/linux/:$PATH export
LD_LIBRARY_PATH={notes.binary.dir}/notes/latest/linux/:$LD_LIBRARY_PATH
The notes.data.dir is the directory where the data for the Domino server or for the Lotus Notes client is installed. The
notes.binary.dir
is the directory where the Domino server or for the Lotus Notes client binary and executable files are installed. For example: The default directories for the Domino server on Linux platforms are:
{notes.data.dir} - /local/notesdata
{notes.binary.dir} - /opt/ibm/lotus
The privileges of the TDI server process are determined by the user that starts the server.
For security reasons, the Domino Server forbids execution of commands using root privileges. To run the Domino server, run with the user configured during the installation process, normally the Lotus Notes user. The TDI server is required to run with the user configured during installation only when the Domino libraries enforcing this restriction are loaded. The TDI server is able to run with root privileges only if no Domino or Lotus Notes connectors are used in an AssemblyLine.
It is possible, however, that you require both of the following privileges:
If we need to access a Domino server while executing certain tasks as a root user, :
The [TDIserverRoot] could use either the AssemblyLineConnector or the AssemblyLineFC to access the remote proxy assembly line on the [TDIserverNotes].
If you are running Windows, and trying to execute an internal shell command (such as dir or any command listed by the command), you might have forgotten to prepend cmd /c . For example, the correct syntax for the dir command is cmd /c dir.
When we use the Command Line Connector to run a program on a Windows operating system, the output from the program might be encoded using a DOS code page. This can cause unexpected results, because Windows programs usually use a Windows code page. Because a DOS code page is different from a Windows code page, it might be necessary to set the character encoding in the Command Line Connector's parser to the correct DOS code page for your region; for example: cp850.
When the JDBC Connector is using a JDBC 2.0 driver (or less) for communicating with a database, there may be problems with Custom Prepared Statements. For instance, IBM SolidDB 6.5 provides only a JDBC 2.0 compliant driver. If we want to work with SolidDB and also enable the Use custom SQL prepared statements option of the Connector a java.lang.NullPointerException will be thrown when you try to start the AssemblyLine. The reason is that for handling custom prepared statements the JDBC Connector relies on functionality added in JDBC 3.0 and SolidDB driver is only JDBC 2.0 compliant. To solve this issue, use a JDBC 3.0 driver for your solution. If there is no such driver available for the needed database, as is with SolidDB 6.5, we will not be able to make use of the "Use custom SQL prepared statements" functionality.
A server service named DB2 JDBC Applet Server must be running on the Windows system where the DB2 server is running. If The DB2 JDBC Applet Server service is not running we will get this message.
The remote DB2 server is not configured for DB2 net driver communications. Refer to the FAQ that has more information on connecting to a DB2 server.
If you are getting this when inserting or updating date-fields, you are probably passing the Oracle driver dates as a string that does not match what the driver expects. Problem Scenario: (For more detailed information about a situation where this can happen, skip to the Suggestions section if not interested). We have an AssemblyLine with a JDBC Connector in AddOnly mode that writes some records to an Oracle table with a field of type DATE. Normally, we can insert something like:
INSERT INTO table1 values (to_date('2005/03/01 10:05:13','YYYY/MM/DD HH:MI:SS'))
as part of an INSERT query. However with TDI, we can only do something like this in the output map:
ret.value = '2005/03/01 10:05:13';
But if Oracle fails with the following error:
java.sql.SQLException: ORA-01830:
The Date Format picture ends before converting entire input string.
When dates are supplied as strings (which is what you are doing here) the TDI JDBC Connector will attempt to parse the data using the pattern provided in its Date Format configuration parameter, as explained in the IBM TDI V7.1 Reference Guide. To debug your problem: What is your Data Pattern configuration? Find out how TDI sees this field by checking in the schema tab of the Connector. A fair guess is that your JDBC driver will convert the Oracle Data type into a java.sql.TimeStamp or java.sql.Date type (and note that there are differences between java.util.Date and java.sql.Date, in terms of precision for example). In the case, for example, of a java.sql.Timestamp type, try specifying:
ret.value = java.sql.Timestamp(java.util.Date().getTime());
and see if this helps. Then we will be able to use:
ret.value = java.sql.Timestamp(system.parseDate(work.getString("yourDate"), "yyyyMMddHHmmssz").getTime());
If none of the above helps, run the Connector in detailed log mode and see whether the Connector is able to get the schema from the database. If not, the Connector does not use prepared statements, which makes it less efficient and more error-prone, so you'll have to make sure that the Connector's schema configuration parameter is set correctly.
If your attributes are of CLOB/BLOB type, the Connector does not handle them on output. On input, we can do something like this:
desc = conn.getObject("yourCLOBAttribute"); ret.value = desc.getSubString(1,desc.length());
but it is slow and clumsy. Also, it will only work if the JDBC driver actually returns a java.sql.Blob or java.sql.Clob interface object.
If Prepared Statement is disabled, the JDBC connector attempts to construct a complete SQL query. If the database has a restriction on the length of the SQL query, and the query exceeds the maximum length value, an exception is thrown. This is a common problem with BLOB or binary data types.
Use ojdbc14.jar instead of using classes12.jar when using the JDBC Connector to transfer BLOB data from one table to another table in an Oracle database.
To connect to this DB2 server, obtain a licensed copy of the IBM DB2 Universal Driver for JDBC and SQLJ.
TDI 7.1 comes with updated Derby database (previously known as Cloudscape) and the driver needed for it. 7.1 also comes with a license file that is enables you to connect to Derby, but not to other DB2 databases.
As of DB2 UDB v8.1.2 the Universal JDBC driver requires a license JAR file to be in the CLASSPATH along with the db2jcc.jar file. Here are the names of the required license JAR files:
An appropriate location for this license file to be placed in a TDI system would be <TDI Install Directory>\_jvm\jre\lib\ext directory.
For more information, see http://www-128.ibm.com/developerworks/db2/library/techarticle/0307zikopoulos/0307zikopoulos.html.
Excessive " com.ibm.dsml2.* " log messages may be received in the log of the AssemblyLine.
Examples of the excessive log activity may include:
com.ibm.dsml2.jndi.DSML2DirContext [search] dc=HRLoad (uid=Vox) javax.naming.directory.SearchControls@2f9ad92 com.ibm.dsml2.jndi.SearchMessage [getFilter] filter: (uid=Vox) com.ibm.dsml2.jndi.SearchMessage [getFilter] filterObject: com.ibm.dsml2.parser.Filter@1bc52d92 com.ibm.dsml2.jndi.SearchMessage [checkResponse] reader is java.io.BufferedReader@17792d92 com.ibm.dsml2.jndi.SearchMessage [checkResponse] Starting unmarshaller thread com.ibm.dsml2.jndi.SearchResultEnumeration Creating a search result enumeration com.ibm.dsml2.jndi.SearchResultUnmarshaller [run] Starting unmarshal thread com.ibm.dsml2.jndi.SearchResultUnmarshaller$ResultEnumeration [getNext] got a com.ibm.dsml2.parser.SearchResultEntry
To eliminate the excessive logging of the " com.ibm.dsml2.* " messages seen while using the JNDI Connector, add the following line to the Provider Param parameter of the JNDI Connector:
com.ibm.dsml2.jndi.logLevel:ERROR
When you are processing a very large amount of data in Domino server it is possible to receive errors similar to these:
Both of these errors indicate that your Domino server is running out of memory resources. The first error may occur on servers with very high process local memory usage. An example would be the HTTP server serving up large databases, port compression is enabled and there is a large population of users using the system.
In Domino the private and shared memory must reside in a limited virtual address space, which is usually 4 gigabytes. The error occurs when Domino runs out of virtual memory or out of shared memory. In order to prevent this from occurring we can use either of these two new notes.ini entries:
ConstrainedSHM=1 ConstrainedSHMSizeMB="size in megabytes"
The variable ConstrainedSHM=1 will restrict shared memory to a set of default sizes, as follows:
Windows and Macintosh platforms: 2 gigabytes
AIX platforms: 2.25 gigabytes
Solaris and Linux: 3 gigabytes
iSeries: 2 gigabytes
The ConstrainedSHMSizeMB="size in megabytes" will restrict memory to the "size in megabytes".
This section explains exceptions for Domino User's Connector and provides a workaround.
at com.ibm.di.connector.dominoUsers.DominoUsersConnector.executeCommand(Unknown Source) at com.ibm.di.connector.dominoUsers.DominoUsersConnector.initialize(Unknown Source) at com.ibm.di.server.AssemblyLineComponent.initialize(Unknown Source) at com.ibm.di.server.AssemblyLine.initConnectors(Unknown Source) at com.ibm.di.server.AssemblyLine.msInitConn(Unknown Source) at com.ibm.di.server.AssemblyLine.executeMainStep(Unknown Source) at com.ibm.di.server.AssemblyLine.executeMainLoop(Unknown Source) at com.ibm.di.server.AssemblyLine.executeMainLoop(Unknown Source) at com.ibm.di.server.AssemblyLine.executeAL(Unknown Source) at com.ibm.di.server.AssemblyLine.run(Unknown Source)
The exception can be caused by a wrong directory or misspelling in the LD_LIBRARY_PATH set within the "ibmditk" or "ibmdisrv" startup files.
For example, LD_LIBRARY_PATH=/opt/lotus/notes/latest/linux.
Add the following two lines to the shell script ( "ibmditk" or "ibmdisrv") after the PATH definition and before the startup line:
LD_LIBRARY_PATH=<Domino Binary> export LD_LIBRARY_PATH
where <Domino Binary> is the location of the Domino Binary folder.
Example file: ibmdisrv.sh
#! /bin/sh # start up script for Directory Integrator v6.1 for Unix platforms JRE_PATH=_jvm/bin OS=`uname` if [ $OS = "Linux" -o $OS = "AIX" ];then JRE_PATH=_jvm/jre/bin fi PATH="/opt/IBM/ITDI61/$JRE_PATH:$PATH:/opt/lotus/notes/latest/linux:/local/notesdata:" export PATH LD_LIBRARY_PATH=/opt/lotus/notes/latest/linux: export LD_LIBRARY_PATH # # Only set TDI_SOLDIR if it hasn't been set already in caller's shell # if [ -z "$TDI_SOLDIR" ]; then TDI_SOLDIR="." fi # # -s overrides TDI_SOLDIR env # solnext=0 for s do case $s in -s) solnext=1;; -s*) TDI_SOLDIR="`echo $s | cut -c3-`";; -*) solnext=0;; *) if [ $solnext -eq 1 ]; then TDI_SOLDIR=$s solnext=0 fi;; esac done if [ -n "$TDI_SOLDIR" ]; then cd "$TDI_SOLDIR" fi # Check solution directory files if [ ! -f IDILoader.jar -a ! -f log4j.properties ]; then echo Copying log4j.properties to solution directory cp -f "/opt/IBM/ITDI61/log4j.properties" log4j.properties fi "/opt/IBM/ITDI61/$JRE_PATH/java" -Dos.name=Linux -Djava.library.path=$PATH \ "-Dlog4j.configuration=file:log4j.properties" -jar "/opt/IBM/ITDI61/IDILoader.jar" \ com.ibm.di.server.RS "$@"
For more information, see: IBM TDI: Post Release 6.0 Issue.
This error occurs when you attempt to use the Windows Users and Groups Connector on a non-Windows platform. The Windows Users and Groups Connector is supported on Windows platforms only.
After installation of the sapjco 2.1.7 SAP interface library, connections still fail. When the connector establishes a connection to the R/3 system, you get this JCO.classInitialize exception.
You are unable to start 32-bit programs from SAP Release 6.40 (or higher) because Microsoft runtime DLLs are missing (MSCVR71.dll and MSCVP71.dll).
For more information, see SAP Note 684106 for a procedure to fix this problem.