AssemblyLine flow and Hooks

AssemblyLines provide built-in automated behavior for building and deploying data flows. Connectors and Functions have their own behaviors. Connector behavior depends on the Mode setting.

Throughout these built-in logic flows are waypoints where we can add our own scripted logic to extend built-in behavior, or to override it completely. These waypoints are called "Hooks" and are available for customizing under the Hooks tab of all Connectors and Functions, as well as of the AssemblyLine itself.

We can enable and disable Hooks according to whether a particular Hook is applicable to the AssemblyLine we are running. When we disable a Hook, we do not break its inheritance in the connector of which it is a part.

When an AssemblyLine is launched, it goes through three phases:

1. Startup

   Prolog Hooks configure components before they are initialized.

2. Data flow

   Each Work Entry fed into the AssemblyLine is passed down the Flow components for processing.

3. Shutdown

   Epilog Hooks carry out end-of-job work, like checking and reporting on error status, or storing state data for the next time the AssemblyLine is started.

Startup Phase

The server is instructed to load and run an AssemblyLine using the blueprint stored in the Config file. If a TaskCallBlock (TCB) is passed into the AssemblyLine, then its contents are evaluated (which can result in changes to AssemblyLine component parameters). At this point, Prolog Hook flows are initiated.

Global Prologs

First, if any Global Prologs are defined then these are evaluated. Global Prologs are Scripts in the Resources folder in the Project that have been included (in the Solution Logging and Settings window) in the AssemblyLine. This is typically done in the AssemblyLine Settings window of the AssemblyLine by selecting the Scripts to run at AssemblyLine startup. After all Global Prologs are finished, the AssemblyLine Prolog Hooks are called.

AL Prolog Hooks (Before Initialize)

First the AssemblyLine Prolog - Before Initialize Hook is invoked. After this, all Connectors and Functions configured to Initialize "at Startup" go through their initialization phase, which also invokes their Prolog Hooks, as seen in the next point.

Connector/Function Initialization

The initialization sequence is performed for each Connector and Function with Initialization set to "at Startup." These are started in turn as defined by their order in the AssemblyLine. For each Connector or Function the flow is as follows:

1. The Prolog - Before Initialize Hook of the component is called.

2. The component is started; for example, connecting to its underlying data source, target or API.

3. For Connectors in Iterator mode, the Prolog - Before Selection Hook is processed, and the Connector performs the entry selection; this triggers a data source specific call, like performing an SQL SELECT or an LDAP search.

4. For Iterators, the Prolog - After Selection Hook is evaluated.

5. The Prolog - After Initialize Hook is called, ending the sequence.

If initialization fails for a Connector, then AssemblyLine flow passes to the Prolog - On Error Hook where we can deal with this error.

The Reconnect feature allows us to configure a Connector to automatically attempt to re-establish its connection if an error occurs during setup or data access. These settings are found under the Connector's Connection Failure tab.

Script Connectors, that is, Connectors implemented using JavaScript, are evaluated at this stage so that required Connector functions are registered and initialization code is executed.

AL Prolog Hooks (After Initialize)

The AssemblyLine Prolog - After Initialize Hook is executed. Completion of this Hook signals the end of Start up Phase, and the beginning of Data flow Phase.

Data flow Phase

AssemblyLine Start of Cycle Hook

This Hook is invoked at the start of every cycle before Feeds or Flow components.

AssemblyLine Cycle

Control is passed to the first component in the flow, typically a Server or Iterator mode Connector in the Feeds section.

If you have one or more Iterators in the AssemblyLine, then the first one starts the cycle by retrieving the next entry from its result set and mapping Attributes into the Work Entry. The resulting Work Entry is passed to Flow section components, starting at the top of the list as seen in the CE.

For Server mode Connectors, a listener process is launched that waits for incoming client connections. When a connection request is detected, the Connector clones itself, accepts the connection and then switches itself to Iterator mode in order to feed data from the client into the Flow section for processing. Either way, you get an Iterator driving Work Entries to the Flow components. (Meanwhile, the original Server mode Connector waits for additional incoming connection requests.)

If the AssemblyLine neither has an Iterator Mode or Server Mode Connector, then we have a one-shot AssemblyLine typically used to process an Initial Work Entry (IWE) fed by another calling process.

End-of-Cycle

When the last Flow component is executed, one of three things can happen:

• If the current Work Entry came from an Iterator, control is passed back to the Iterator to get the next entry from its source.

• In the case of a server mode Connector, a reply is made to the client.

• For an AssemblyLine that is called in manual cycle mode, the thread is passed back to the caller so that results can be accessed.

There is no specific Hook at this point, although this can be added to our AssemblyLine by inserting a Script at the end.

End-of-Data

End-of-data is an Iterator mode Hook that is called when the end of the input data set is reached. At this point, control is either passed to the next Feeds Connector, or the AssemblyLine goes into Shutdown Phase.

Shutdown Phase

At this point, AssemblyLine processing has either completed normally, or aborted due to an error.

AssemblyLine Epilog - Before Close Hook

The AssemblyLine Hook called Epilog - Before Close is processed.

Connectors/Function Close flow

The Epilog Hooks of each Connectors and Function are called in the order that they appear in the Config Editor:

1. The Before Close Hook.

2. The close operation is carried out; for example, closing a connection or release an API callback.

3. The After Close Hook.

AssemblyLine Epilog - After Close Hook

Finally, the AssemblyLine Epilog - After Close Hook is run.

Server mode Connector Setup

When a Connector in server mode starts, it goes into event listening mode. Once an event is received, it clones the AssemblyLine and resumes waiting for more events. In the clone, meanwhile, the Connector switches itself to Iterator mode and passes control to the next component in the Feeds list. This process allows us to have multiple server mode Connectors active and feeding data into the flow at the same time - one example would be to have several HTTP server Connectors in server mode listening to different ports, but feeding the same AssemblyLine. Although server mode Connectors are part of an AssemblyLine configuration, they run as separate processes, as threads.

There is an additional set of Hooks that is evaluated for Connectors in this mode. The Hooks specific to server mode functionality for dealing with incoming connections are:

Before Accepting connection

This Hook is called before the Connector goes into listening mode.

After Accepting connection

Once a connection is received, this Hook is invoked. Note that no data is available at this time. In order to examine incoming event information, use the Iterator Hooks like After GetNext or GetNext Successful.

Error on Accepting connection

This Hook is executed if an error occurs in any of the server mode Hooks, or received from the data source during event listening.

• As mentioned previously, if we have more than one Connector in Iterator mode (see Multiple Iterators in an AssemblyLine), these Connectors are stacked in the order in which they are displayed in the configuration, from top to bottom. For example, if we have two Iterators, a and b, then a is called until it returns no more entries before the AssemblyLine switches to b.

• If we have no Connectors in Iterator mode, and no Initial Work Entry (IWE) is provided to the AssemblyLine when it is started; for example, by a calling from another AssemblyLine, and if no Work entry is created in an AssemblyLine or Connector Prolog Hook, then the AssemblyLine still performs a single pass.

Finally, there is a Shutdown Request Hook where we can put code that is processed if the AssemblyLine is closed down properly due to an external request to shut down (as opposed to one that crashes), enabling you to make it perform a graceful shutdown.

Special functions are available from the system object to skip or retry the current Work entry, as well as to skip over a Connector, and so forth. See Control the flow of an AssemblyLine for more details.

• Handle termination and cleanup for critical errors
  

Parent topic: The AssemblyLine