Refer to Fig 1. In this generic example:

• State 1 is the "Record Separator" step, which gleans out all records of the desired processiing type.
• State 2 is the "Convert to XML" step, which converts the record data, field by field, into XML tagged data.
• State 3 is the "Load XML to Database" step, which takes the tagged data and loads a temp table in the database.
• State 4 is the S.P. step (stored procedure), which takes the data from the temp table, applies the necessary busines rules/conversions/etc. to the data, and loads the results to the permanent tables.

Now while it's possible to wake up the state machine every few seconds and execute one of these steps in order of appearance, if any process takes longer than the wake interval then this could be problematic. The preferred method of scripting a file process is in Reverse Order. This helps prevent problems with timing. In other words, the FIRST thing that happens when waking up is to try the stored procedure, then that data load, then XML conversion, and so on. This ensures that the file in question actually exists, etc.

For most purposes you will want to set the wakeup interval to longer than the longest processing time. Typically the loading of data takes the longest time (network traffic, inherent inefficiency of DB technology, etc.) so you would set the wakeup interval at longer than the worst case you're likely to encounter.

Here's an example scripting for a "cleared check file" process:

1) c:\test\files;*.PCC;c:\test\fl2xml.exe
2) c:\test\incoming;*.CCF;c:\test\recsep2.exe

In line 1 the "c:\test\files" directory is searched for any file matching a *.PCC file extension. If a file is found (for argument let's say that it finds "myfile.PCC") then the program "c:\test\fl2xml.exe" is executed with the *.PCC filename as an argument. Here's what the command line would look like:

c:\test\fl2xml.exe c:\test\files\myfile.PCC

Line 2 works the same way. The order of execution is in order of appearance in the script (line 1 is done first, in other words.) Overall any *.CCF file (CCF = "cleared check file") found in the "incoming" directory is processed by the RECSEP2 program (RECSEP = RECord SEParator.) The resulting *.PCC file RECSEP2 outputs to the "files" directory is then turned into XML via FL2XML program. Since these are the only two states in the script, the states following FL2XML are "tightly coupled" (see section VI.)

Notes:

Since a state machine by definition is simply a "dumb" device (i.e. if the state is 1 then do this, if the state is 2 then do that) and not aware of what a given state is trying to accomplish, it can handle variation easier. For example, if normally the first 2 states of any process are devoted to turning a data file into XML and the file provider switches to XML format, the state machine would run as before except that state 3 (load XML to server) is now the first state; the original preceeding states are now moot.

It is not always necessary to define each state via the timer. The timer part of the load-process state machine controls a macroscopic function; any of the programs executed can actually chain "micro" states as needed. For instance in the above example (cleared check process) the FL2XML "fixed-length to XML" applet is set to call the XML2DB "xml to database" applet when it completes and the XML2DB applet is set to call the "xmlProcessClearedCheques" stored procedure when it completes. Each of these states is still discrete and could be fired via the state machine timer if need be. Therefore the load-process state machine is in practice a combination of state controls by the timer and process applets.


V. AUTOMATION DETAIL: Post-Process State Machine

This state machine is "target sensitive" in that to operate efficiently it is tied to a Server/Database. The typical implementation uses a table (EDI_STATE) that contains merely a process ID and a state integer. The integer is a flag bitfield allowing up to 32 states, each of which is user defined. The process ID field is also user defined; e.g. process 1 might be the cleared check file process, process 2 deals with files from the Alaska Eskimo Fishing School, and so on. In most cases 8 flags is more than sufficient:

Example Bitfields: cleared check file process

0x01 -- generate HTML report
0x02 -- report disposistion (if needed)
0x04 -- undefined for now
0x08
0x10
0x20
0x40
0x80

As the timer portion wakes up, the lowest nonzero state is executed first, and then the states for that process ID are ignored until the next wakeup.

Here's part of the script for the timer that generates the HTML report. The first entry is the process ID (1 = cleared check process.) The data between the semicolons is (in order) state 1 through 32. In other words, each line in the script is a delimited list of state actions, with each state defined as the ordinal column position.

  |<-                              STATE 1                              ->| 


1;c:\test\htmrpt.exe c:\test\ccf.qry c:\test\files\ccfr c:\test\emailer.exe;;;


^        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^         ^^^


procid                         STATE 1                            STATE 2 - UP

In this case at state 1 HTML reporter applet HTMRPT is started and told to use CCF.QRY as the query, then name any output file with a prefix of CCFR ("c:\test\files\ccfr.") Since HTMRPT has a counter, if it was set to 3 then this will result in a file named CCFR4.HTM. This file is then handed to the EMAILER applet which will then read its own INI file to determine who the report is to be emailed to. As above with the discussion of macroscopic states we could break these steps up to be discrete if required.


VI. Macroscopic vs. Microscopic States: Coupling Issues

As discussed this EDI process allows states to be "loosely" coupled to any following states (macroscopic) or "tightly" coupled as required. The type of coupling depends on the nature of the problem to solve and somewhat on the target.

For example, SQL Server 6.5 locks pages even for a SELECT statement, so it's usually not a good idea to have two processes that could be intensively selecting or writing to the same tables simultaneously: they could collide. As a result processes that could conceivably collide should not use tight coupling. Of course the reverse is true of processes that do not write to often-accessed tables.

In the ongoing cleared check file process example, tight coupling is used because the data in this file will be run against a seldom used table and therefore unlikely to collide with other processes.

In other cases, the decision to use tight coupling will depend on what you are trying to accomplish at that state. For example processes that write to often used tables may have a tightly coupled stored procedure that runs directly after the dataload temp table is loaded; the procedure may do nothing more than set a flag in the EDI_STATE table to indicate a successful load.

In addition, processes that aren't necessarily loading data to a database or manipulating new data can be tightly coupled. For example emailing of reports can be initiated by the report generator directly after the report is written to disk. Reports that needs to be sent as a group however would probably be best served with loose coupling.


VII. EDI_STATE Table

An EDI_STATE table in the simplest form is merely a Process ID and State integer. In the creation of the table the assumption is that states apply to entire processes, but this is not required. States can also be used with discrete objects inside the system. For example you can apply an internal ID as the ProcessID and track/control the processing of entire related objects (e.g. a credit card tracking system would typically have information in multiple tables yet still maintain an entity as "a card.")

The most useful variation would likely be the addition of "next anticipated state" and "last used state." When used properly this allows you to peek at a process and see what it is doing.

For example, if using EDI_STATE to control flow of a credit card application and in state 5 a credit check is run, this is the state configuration:

last_used = 4
current = 5
next_expected = 6

In the case of a process variation such as credit is already on file, state 5 is not required. An application usder this condition would then show up as:

last_used = 4
current = 6
next_expected = 7

This "signature" difference can be used to make it simpler to track process flow.


VIII. Generic Process Construction Issues

Applets discussed thus far in this process are designed to be used in as many copies as required. For example there is but one ( 1 ) FL2XML program whose job is to convert fixed length data to XML. To use it, you need to make a copy of the EXE and the associated configuration files into the directory structure that will be used to process a given file type. When executed Fl2XML will look at the configuration file that details how the fixed length bytefields are to be broken up and what tags to associate with these bytefields. This configuration will change of course and is mapped to a file type. Of course the particular copy of FL2XML that is run at a given time is controlled by the startup; c:\dir1\fl2xml.exe will start the copy of FL2XML in the c:\dir1\ directory whereas c:\data21\fl2xml.exe will start FL2XML in the c:\data21\ directory. The proper place to put the configuration file for each copy is of course the directory that contains the copy, thus even if the two copies are executed simultaneously the copy in c:\dir1\ will be doing a different job than the one in c:\data21\.

Since all of the applets constructed to date are done in this same manner, process control is implemented via multiple copies and location control of the individual copies.

Note also that control of applet instantiation in this manner is not only very network friendly but also cross-platform friendly. Applets do not have to be located in clusters; the only hard and fast rule is that a given applet needs to have the associated configuration file located in the same directory as the applet. Admittedly the logistics of clustering applet copies in one single directory devoted to processing a file type are simpler, but this is not required for the process to work.


IX. Process Change -- Applet Construction

The processing discussed in this document is that which has already been done and is working. However please note that this process is engineered to be as flexible as possible and grow as needed. Because the processing elements are all applets, the proper course of action at any given time is to create applets as needed that do specific tasks. For example there is no existing applet specifically meant to do FTPing of data although should the need arise then creating one should be limited strictly to the functionality of FTP and not specific to what is being transferred.

As this process is broken into component parts, it is NOT required that all applets run on the same machine, against the same database, or even the same operating system.

In general, applets should follow these rules:

1. Applets must be command line capable and never require user input. By definition automation is the process making decisions; as soon as you require human input there is no automation. The alstonlabs.com applets also are coded such that for testing purposes they can present a user interface, but the UI is not required for them to run.
2. Applets should be able to handle either multiple command line parameters or have an associated INI file that can be scripted. This allows an applet to be flexible.
3. Applets must clean up after themselves. If an applet wakes up to find and read file A and writes file B then it should do something with file A so that A isn't processed again. Delete it or archive it, whatever, just so long as the process isn't duplicated.
4. Applets must attempt to be minimally recoverable. When possible this means that deletion or removal of an input file happens at the same time as the closing of an output file. If the power plug gets pulled in the middle of what an applet is creating, when the state resumes the applet will still find the input file. If however the input file is deleted after it is read and the applet does something and the power plug is pulled before the output is written, then when the state is resumed, there is not only no output file, but the input file is missing as well leaving no way to recreate this output.