IBM BPM, V8.0.1, All platforms > Authoring services in Integration Designer > Services and service-related functions > Access external services with adapters > Configure and using adapters > IBM WebSphere Adapters > Flat Files > Overview of WebSphere Adapter for Flat Files > Technical overview
Inbound processing
WebSphere Adapter for Flat Files supports inbound event processing. It polls the local file system at specified intervals for events, such as the creation of a file. When it detects an event, it converts the event data into a business object and sends it to the module for processing.
The following illustration shows the inbound processing flow for WebSphere Adapter for Flat Files.
Figure 1. Inbound processing
When a change occurs in the local file system, an event file, which is a new or changed file, is created in a specific directory. You configure this directory as the event directory for the adapter. Although an event file can represent one or more events in the file system, it forms a single unit of transfer to the adapter.
The adapter polls the event directory on the file system at regular intervals, based on the value set in the PollPeriod property. When a file arrives in the event directory, the adapter sends the content of the file to the export. The file content can be sent in its entirety or split into several business objects, or chunks. The adapter sends the business objects to the export by using a function selector, which selects an operation to invoke on the component and provides the correct data binding.
The inbound processing flow is described in the following steps.
- Events, in the form of files, are generated in the file system.
- The adapter polls the event directory.
- The adapter assigns each event an event ID and stores the event
ID in the event store. The event store is a persistent cache where event records are saved until a polling adapter can process them.
You must create this database before you configure the adapter. The default name of the database is FFDB.
- The adapter reads each event file as bytes. If file splitting
is enabled, the adapter parses the event file based on the values set in the SplittingFunctionClassName and SplitCriteria properties:
- If splitting is based on a delimiter, the class that performs this functionality and the split criteria are provided.
- If splitting is based on file size, the class name that performs this functionality is provided.
- If the configured data type is object-specific, for example, CustomerWrapper, the data handler is configured on the DataBinding, and the adapter
transforms the data. If the configured data type is FlatFile or FlatFileBG, the adapter passes the content of the file as a byte array within a FlatFile business object, and no transformation is performed.
If file splitting is enabled, the business object contains the file size and the event ID.
- The adapter sends the business object to the export through a function selector, which selects an operation to invoke on the component and provides the correct data binding. After the business object reaches the export, the event is deleted from the event store. If archiving is enabled, the event is moved to an archiving table before it is deleted.
Polling files in subdirectories
By default, when the adapter polls files in the event directory, it polls files from the root directory only and ignores files in the subdirectories.
If you set the PollSubDirectories property in the activation specification to True, the adapter first polls the files in the root directory and then polls the files in the subdirectories. After the adapter retrieves all the files, it sorts them according to the value set for the SortEventFiles property. The adapter then processes the files according to the value set for the PollQuantity property and sends the business objects to the downstream components.
Event archiving
To track successfully polled events, you can configure an archive directory on the file system by using the ArchiveDirectory activation specification property in the external service wizard. The files are copied to the archive directory with either a success or fail extensions, as specified in the activation specification.
Event file locking
File locking behavior is operating system dependent. On Windows, if the adapter polls any file from the event directory that another application is using and the file is in the process of being copied to the event directory, they are not made available to the adapter for processing.
However, in UNIX environments, such as AIX , there is no file locking mechanism that prevents applications from accessing files that are being written to. A file that is being copied to the event directory by another application is made available to the adapter for processing, causing erroneous results. There is no platform independent way in Java™ to check whether a file is being written to.
To prevent this situation from occurring, you can first copy the event file to a staging directory and then move it to the event directory using the move command. Some sample UNIX scripts are provided as part of the adapter. The script file named CheckIfFileIsOpen.sh is available in the Unix-script-file folder in the adapter installer.
Rule-Based filtering of events
The adapter supports the rule-based filtering of events, which is optional for inbound processing. You can filter the events based on multiple rules. You can define a combination of these rules, group them with Boolean logic, and filter the events using the following metadata:
- FileName
- File Size
- Directory
- Last Modified
For example, you can use FileName "MatchesFilePattern" *.txt, where FileName is the property type, "MatchesFilePattern" is the operator and "*.txt" is the value.
The rule is applied to the files that are filtered by the event file mask criteria. By default, event file mask will have "*.*" as default value.
Rule-based filtering does not support the logical "OR" operator values between multiple rules.
Adapter does not support rule-based filtering when the EIS is on MVS™ platform.
| Property | Valid operators | Value | Prerequisites |
|---|---|---|---|
| FileName | Matches_File_Pattern | For example: *.txt | Nil |
| Matches_RegExp | Java Regular Expression | ||
| FileSize | Greater than, Less than, Greater than or equal to, Less than or equal to, Equal to, Not equal to. | Numeric value in Bytes. For example: 10000 | Nil |
| Directory | Matches_RegExp | Java regular expression | pollSubDirs = true |
| LastModified | Greater than, Less than, Greater than or equal
to, Less than or equal to, Equal to, Not equal to.
Select the 'Equal to' operator when you choose the days of a week. | Day of the week or Time. For example : MONDAY or 20:41:10 | Nil |
| END-OF-RULE | END-OF-RULE | END-OF-RULE | Nil |
Event persistence
The adapter supports event persistence for inbound processing in case of abrupt termination. Event persistence (or assured-once delivery) is a way to ensure that events are delivered only once to the export in the case of a failure. During event processing, the adapter maintains the event state in an event store on the data source. You must setup the data source using IBM BPM or WebSphere Enterprise Service Bus before you create the event store. To use the recovery feature provided by IBM BPM or WebSphere Enterprise Service Bus, set the AssuredOnceDelivery property in the activation specification to True. This recovery feature is enabled by default.
The adapter also provides for event persistence by using an in-memory representation of the event store. When you use this feature, you do not need to create a JNDI data source or an external event store, and event processing is faster. However, with this feature there is no support for event recovery. If there is a server failure, the in-memory event stores are lost. To prevent the loss of events in the case of server failure, the recommended approach is to use the database event store.
To use the in-memory event persistence capability of the adapter, you must set the AssuredOnceDelivery property to false, or the adapter will log a warning message.
- Event store
The event store is a persistent cache where event records are saved until the polling adapter can process them. The adapter uses event stores to track inbound events in the system. Each time a file is created, updated, or deleted, the adapter updates the status of the event in an event store. The status of each event is continuously updated by the adapter for recovery purposes until the events are delivered to the export. - File store
The file store is a persistent cache where event files are saved until the polling adapter can process them. The adapter uses file store to track inbound files in the system. Each time a file is created, updated, or deleted the adapter updates the status of the file in the file store. - Function selectors
During inbound processing, a function selector returns the appropriate operation to be called on the service. You choose a function selector when you configure the adapter for inbound processing in the external service wizard. The adapter provides three function selectors, FilenameFunctionSelector, EmbeddedNameFunctionSelector, and RootNameFunctionSelector. - File retrieval
During inbound processing, you can manage the retrieval of the files by using the Poll event files for modified content or the Time interval for polling unchanged files property.You can also use the Include only the newly appended content property to retrieve only the appended file contents.
- File splitting
The adapter supports an optional file splitting feature to reduce memory loading during the event processing. When this feature is used, the adapter divides large event files into smaller chunks, which are then posted separately to the endpoint. - Inbound data transformation
During inbound processing, the adapter performs data transformation based on the adapter-specific data binding and data handler that you select when you configure the module in the external service wizard.
Technical overview of the WebSphere Adapter for Flat Files
Related reference: