+

Search Tips | Advanced Search

MFT resource monitoring concepts

An overview of the key concepts of the Managed File Transfer resource monitoring feature.


Resource monitors

The resource monitor is associated with a Managed File Transfer agent, and is only active when that agent is started. When the monitoring agent stops, so does the resource monitor. If the agent is already started when the resource monitor is created, the resource monitor starts immediately. The monitoring agent must also be the source agent of the task that is initiated by the resource monitor.

Resource monitor names must be unique within their agent. The resource monitor name must be a minimum of one character in length and must not contain asterisk (*), percent (%) or question mark (?) characters. The case in which a resource monitor name is supplied is ignored and the resource monitor name is converted to uppercase. If you try to create a resource monitor with a name that is already present, the request is ignored and the attempt is logged to the resource monitor log topic.

There is no restriction on the number of resource monitors that can be created on an agent, and all run with the same priority. Consider the implications of overlapping monitored resources, conflicting trigger conditions and how frequently the resources are polled.

Resource monitors look at the contents of resources after every poll interval period. The contents of the resource are compared with the trigger conditions and if those conditions are satisfied, the task associated with the resource monitor is called.

The task is started asynchronously. If there is a condition match, and the task is started, the resource monitor continues to poll for further changes to the resource contents. So for example, if a match occurred because a file called reports.go arrived in a monitored directory, the task would be started once. At the next poll interval, even if the file still exists, the task is not started again. However, if the file is deleted and then placed in the directory again, or the file is updated (such that the last modified date attribute is changed), the next trigger condition check causes the task to be called again.

Before Version 9.1.5, if a resource monitor performs a poll that takes longer than the polling interval, this means that the next poll starts as soon as the current one finishes with no gap in between, which could have an effect on how quickly resource monitors submit work to an agent. This could cause performance issues if the items that are found during the first poll are still there when the second one takes place.

From Version 9.1.5, the resource monitor uses the ScheduledExecutorService and initiates the next poll only after the completion of the previous poll plus the configured poll interval time. This means that there will always be a gap in between the poll intervals, rather than having another poll starting straight away after the previous poll if the poll time was longer than the poll interval.

From Version 9.1.3, if a file has failed to transfer, we can clear the resource monitor history, which allows another transfer request to be submitted without the need to either delete the file and place it in the directory again, or update the file to change its last modified date attribute. Clearing the history is useful, for example, in situations where there is need for the file to be transferred but modifying the file is not possible. For more information, see Clearing resource monitor history.


Resources

Resource monitors in Managed File Transfer can poll the contents of the following two types of resource:

    Directories or nested directory structures
    A common scenario is to monitor a directory for the presence of a trigger file. An external application might be processing multiple files and placing them in a known source directory. When the application has completed its processing, it indicates that the files are ready to be transferred, or otherwise acted upon, by placing a trigger file into a monitored location. The trigger file can be detected by a Managed File Transfer resource monitor and the transfer of those files from the source directory to another Managed File Transfer Agent is initiated.
    By default, the specified directory is monitored. To also examine sub-directories set the recursion level in the fteCreateTransfer command.
    Two examples of monitoring a directory are as follows:

    • Monitor for a trigger file (for example trigger.file) and then transfer a wildcard (for example, *.zip).
    • Monitor for *.zip and then transfer ${FilePath} (for example, the file that triggered the transfer). For more information about variable substitution, see Customizing MFT tasks with variable substitution.

    Note: Do not create a monitor that monitors for *.zip, and then transfers *.zip. The monitor tries to start a transfer of *.zip for every .zip file on the system. That is, the monitor generates * number of transfers for *.zip.

    For an example of creating a resource monitor to monitor a directory, see Monitor a directory and using variable substitution.

    IBM MQ queues
    An example of monitoring a queue is that an external application might be generating messages and placing them on a known queue with the same group ID. When the application has completed putting messages on the queue, it indicates that the group is complete. The complete group of messages can be detected by a Managed File Transfer resource monitor and the transfer of the group of messages from the source queue to a file is initiated. For an example of creating a resource monitor to monitor a queue, see Example: Configuring an MFT resource.Note: We can specify only one monitor per queue. If you specify more than one monitor to poll an IBM MQ queue, unpredictable behavior occurs.

Monitor data sets is not supported.


Conditions and trigger conditions

The condition is met when the resource contains a value that matches some other string or pattern. Conditions can be one of the following:

  • Match on file name (pattern)
  • No match on file name (pattern)
  • File size
  • Match if file size remains the same for a number of polls

File name matching can be expressed as:

File names can also be excluded from file name matching by using a wildcard or Java regular expression that identifies file names that are never matched.

When a matching file is detected, its last modified time stamp is retained. If subsequent polls detect that the file has been changed, the trigger condition is satisfied again, and the task is started. If the condition is to detect when a file does not exist, if no file in the monitored directory matches the file name pattern, the task is started. If a file is then added to the directory that does match the file name pattern, the task is only started if the file is then deleted.


Tasks

Managed File Transfer supports the following two types of task that we can configure to be started by resource monitors:

    File transfer tasks
    File transfer tasks are defined in the same way as any other file transfer. A useful way to generate the task XML required by a monitor is to run the fteCreateTransfer command with the -gt parameter. This command generates a task definition as an XML document, including the transfer specification. You then pass the name of the task XML document as the value for the -mt parameter on the fteCreateMonitor command. When the fteCreateMonitor is run, it reads the task XML document. After the fteCreateMonitor is run, any changes that are made to the task XML file are not used by the monitor.
    When using a file transfer task, we can select how many trigger conditions are batched into a task. The default is for one trigger condition to start one task. We can run the fteCreateMonitor command with the -bs option to select the number of trigger conditions that are batched together into one task.

    Command tasks
    Command tasks can run Ant scripts, call executable programs, or run JCL jobs. For more information, see Configure MFT monitor tasks to start commands and scripts.


Trigger files

We can use the contents of a trigger file in a resource monitor to define a set of files to transfer in a single transfer request. Each time a matching trigger file is detected, its contents are parsed for source file paths and optionally for destination file paths. These file paths are then used to define file items in the task transfer XML file that you specify, which is submitted as single transfer request to the agent. The definition of the resource monitor determines whether trigger content is enabled.

The format of each trigger file is a single file path to transfer on each line of text. The default format for the line is either a single source file path or a source and destination file path separated by a comma.

For more information and examples, see Use a trigger file.


Backing up and restoring resource monitors

We can back up the resource monitors that we have already defined so that we can re-use them in the future. There are various options that we can use as follows:

  • Use the fteCreateMonitor command with the -ox parameter to export a resource monitor configuration to an XML file, and with the -ix parameter to restore a resource monitor by importing the resource monitor configuration from an XML file.
  • Use the fteListMonitor command with the -ox to export the definition for a single resource monitor to an XML file.
  • Use the fteListMonitor command with the -od to export multiple resource monitor definitions to a specified directory. Each resource monitor definition is saved to separate XML file. We can also use the -od option to export a single resource monitor definition to a specified directory.

For more information, see Backing up and restoring MFT resource monitors.


Resource monitor logging

From IBM MQ Version 9.1.0, Managed File Transfer includes resource monitor logging. For more information, see Logging MFT resource monitors.

Parent topic: Monitor MFT resources


Related concepts


Related tasks


Related information

Last updated: 2020-10-04