IBM BPM, V8.0.1, All platforms > Authoring services in Integration Designer > Services and service-related functions > Access external services with adapters > Configure and using adapters > IBM WebSphere Adapters > FTP > Overview of IBM WebSphere Adapter for FTP > Technical overview

Inbound processing

WebSphere Adapter for FTP supports inbound processing of events. The adapter polls a file system associated with an FTP server for events at specified intervals. Each time a file is created in the event directory, the adapter tracks it as an event. When the adapter detects an event, it requests a copy of the file, converts the file data into a business object, and sends it to the consuming service.

The following illustration shows the inbound processing flow for the adapter.

Figure 1. Inbound processing flow

The adapter polls files from the event directory of the FTP server at regular intervals based on the FTPPollFrequency property. When a file arrives in the event directory, the adapter reads the entire file and downloads it to a local event directory on the adapter workstation. The adapter downloads the files from the FTP server, one file at a time, and cannot download all the files simultaneously. After the file is downloaded, the adapter either archives the file in the FTP server in an archive directory given by the FTPArchiveDirectory property or deletes it based on your configuration. The event directory, archive directory, poll frequency, and poll quantity (the number of files to poll in a single poll cycle) are all configurable properties.

If the Remote directory is set to <HOME_DIR>, the adapter polls for event files in user's home directory. The value of an event directory property accepts both the absolute and relative paths of the directory. If the value does not begin with a forward slash (/), the adapter considers the path to be relative to the home directory of the user.

For example, if the value in the remote directory property is set to ftpuser/event, the adapter considers this to be the path relative to the home directory.

If the home directory is set to “/usr/ftp", then the adapter polls the directory /usr/ftp/ftpuser/event for event files.

After the business objects are successfully posted to the export, the events in the local staging directory are either archived in an archive directory on the local file system or deleted, based on your configuration. The adapter must archive or delete the events or they are polled again.

Inbound event processing consists of the following steps:

1. FTP server generates events in the form of files.

2. The adapter polls the event directory.

3. The files are downloaded to the adapter.

4. The files are split based on the SplittingFunctionClassName and SplitCriteria properties. The event file is split into several chunks and each chunk is posted to the export separately. This reduces memory loading during event processing.

   • If splitting is done based on a delimiter, the class that performs this function and the split criteria are provided.

   • If splitting is done based on file size, the class name that performs this function is provided.

   • If splitting is done based on other criteria, you must provide your own file splitting class.

5. The adapter sends information about the location of the polled document and the host name of the system where the file was retrieved, to the export. A function selector invokes the configured data binding, converts the text record into a business object.

Processing of files using FTP scripts

In addition to processing the files downloaded from the event directory during polling, the adapter can also be used to process the files downloaded using the FTP scripts.

You can specify the scripts to be run before or after polling the event directory using the properties, Run FTP script file before downloading files property (ftpScriptFileExecutedBeforeInbound) and Run FTP script file after downloading files property (ftpScriptFileExecutedAfterInbound). The script files can contain FTP commands, such as mget and get, to download the files from the remote directories on the FTP server to the local event directory of the machine where the adapter is installed. The adapter processes the files that are downloaded to the local event directory configured in the activation specification properties and delivers the processed business objects to the consuming service.

Following is an example of a script:

lcd C:\FTPAdapter\localevent
cd /ftpDir1
mget *.txt
cd /ftpDir2
get abc.xml

Where, C:\FTPAdapter\localevent is the local event directory of the adapter, and ftpDir1 and ftpDir2 are directories that exist on the FTP server. The adapter executes the script and downloads the files to the local event directory. The adapter then processes the files and delivers it to the consuming service.

1. You must copy the files downloaded using the script to the configured local event directory for the adapter to process it. Use the FTP command lcd to change the local working directory to the localEventDirectory before you download any files using the script.

2. The files downloaded to the local event directory using the commands, mget or get will be deleted from the FTP server by the adapter after you download the files. This is to ensure that the files are not downloaded again during the next poll cycle.

3. Use the script file to download the files only from remote directories and not from the event directory of the adapter.

Supported inbound operation

The adapter supports the default emitFTPFile operation, during inbound configuration.

Event file locking

File locking behavior is operating system dependent. In Windows, if any of the files being polled by the adapter from the event directory are in use by another application and in the process of being copied to the event directory, they are not made available to the adapter for processing.

However, in UNIX environments, such as AIX , there is no file locking mechanism that prevents applications from accessing files that are being written to. A file that is being copied to the event directory by another application is made available to the adapter for processing, causing erroneous results. There is no platform-independent way in Java™ to check whether a file is being written to.

To prevent this situation from occurring, you can first copy the event file to a staging directory and then move it to the event directory using the move command. Some sample UNIX scripts are provided as part of the adapter. The script file named CheckIfFileIsOpen.sh is available in the Unix-script-file folder in the adapter installer.

Rule-Based filtering of events

The adapter supports the rule-based filtering of events, which is optional for inbound processing. You can filter the events based on multiple rules. You can define a combination of these rules, group them with Boolean logic, and filter the events using the following metadata:

• FileName
• File Size
• Last Modified

For example, you can use FileName "MatchesFilePattern" *.txt, where FileName is the property type, "MatchesFilePattern" is the operator and "*.txt" is the value.

Though using the rule is optional and specifying an event file mask is mandatory, the rule takes a higher precedence over the event file mask, when both a rule and an event file mask are specified. Event file mask is effective only when there is no rule specified. By default, an event file mask has "*.*" as the default value.

Rule-based filtering does not support the logical "OR" operator values between multiple rules.

Adapter does not support rule-based filtering when the EIS is on MVS™ platform.

Metadata filtering properties

Property	Valid operators	Value	Prerequisites
FileName	Matches_File_Pattern	For example: *.txt	Nil
Matches_RegExp	Java Regular Expression
FileSize	Greater than, Less than, Greater than or equal to, Less than or equal to, Equal to, Not equal to.	Numeric value in Bytes. For example: 10000	Nil
LastModified	Greater than, Less than, Greater than or equal to, Less than or equal to, Equal to, Not equal to. Select the 'Equal to' operator when you choose the days of a week.	Day of the week or Time. For example : MONDAY or 20:41:10	Nil
END-OF-RULE	END-OF-RULE	END-OF-RULE	Nil

• File retrieval
  During inbound processing, you can manage the retrieval of the files by using the Time interval for polling unchanged files property. This property helps you to retrieve only those files which are not changed during the specified time interval.

  If the time difference between the last modified timestamp and the current system time is greater than the value set in FileUnchangedTimeInterval, then such files are polled. In addition to verifying the files that are not changed during the time interval, the adapter also verifies if there is a change in the file size before polling a file in the event directory. However, the value of the file size attribute is dependant on the operating system and the FTP server used.

• Function selectors
  During inbound processing, a function selector returns the appropriate operation to be called on the service. You choose a function selector when you configure the adapter for inbound processing in the external service wizard.
• Inbound data transformation
  During inbound communications an adapter transforms an event record created in a native format, such as bytes or a string, into a business object. The process is called data transformation. The adapter uses an adapter-specific data binding and data handlers to accomplish the data transformation.
• Passing files by reference
  

  The adapter also supports a PassByReference feature, where only the event file name is sent to the export. The event file is appended with a time stamp and is available in the local archive directory. This feature should not be used with data transformation. While using this feature, if there are any data handlers configured then an exception occurs.

• Splitting files
  The inbound event processing mode supports an optional file splitting feature, where the event file is split into several business objects, also known as chunks. Each business object is posted to the export separately. This reduces memory loading during event processing. File splitting is performed based on either a delimiter or on a file size specified in the SplitCriteria property.
• Event store
  The event store (event persistence table) is a persistent cache where events are saved until the adapter can process them. The adapter uses event persistence tables to track the inbound requests as they make their way through the system. Each time a file is created in the event directory, the adapter tracks the activity as an event, and updates the status of the event in the event persistence table. The status of each event is continually updated by the adapter for recovery purposes until the events are delivered to a configured export.
• File store
  When the adapter polls the event directory, an entry is created in the file table for each event file that matches the search criteria specified in the activation specification properties. The adapter uses the file table to track the inbound files. Each time a file is created, updated, or deleted, the adapter updates the status of the entry in the file table.
• Event archive
  Archived events are stored in the archive directory with a file extension that is specified in the FTPRenameExt property. Event archiving is an optional feature, which provides you with a record of all the events that have been processed. You can use this information to review whether the events were processed successfully.

Technical overview

Related tasks:

Setting deployment and runtime properties

Related reference:

Custom file splitting

Activation specification properties