DEAL stage v1.1

Pipelines v2.1

 

Purpose, Operands, Streams, Usage, Examples, Related

Home

 

Syntax

 
          ┌─STOP─ALLEOF────────┐
>>──DEAL──┼────────────────────┼─────────────────────────────────────────────────────><
          ─STOP─┬─ALLEOF──┐   
               ANYEOF──┼───
               └─streams─┘   
                              
          SECONDARY──────────
                              
               ┌─1-*────────┐ 
          └─KEY─┼────────────┼─┘
                └─inputrange─┘

 

Purpose

 

Use the DEAL stage to read records from its primary input stream and write records to one of its connected output streams in either sequential order; starting with the primary output stream, or some other order specified on the secondary input stream. Unless otherwise specified; DEAL writes the first input record to the connected primary output stream, the second input record to the connected secondary output stream, and so on, until the last connected output stream receives a record, and then the cycle is repeated.

 

When the SECONDARY operand is specified; DEAL sequentially processes a secondary input record and primary input record as a pair. Each secondary input stream record must contain an integer output stream identifier which will control the order in which DEAL writes output records. In this way you can choose your own output stream order rather than use the sequential default. DEAL first reads and inspects a record from the secondary input stream, and retrieves the identity of the output stream that will receive a written record. DEAL then reads a record from its primary input stream. When DEAL determines that the output stream identified in the secondary input stream is connected; DEAL writes a record from the primary input stream to the specified output stream. DEAL then consumes the records from both its primary and secondary input streams.

 

You can also write a contiguous series of primary input stream records that have matching keys to the same output stream. DEAL searches for an identified location in each input record that contains the specified key, comparing the key from one record to the next. If the key location is not specified, or it is specified but is not found, DEAL uses the entire record as the key.

 

Operands

 

STOP

specifies how many output streams must be at end-of-file for DEAL to stop writing output records. STOP ALLEOF is the default.

 

 

ALLEOF

specifies that input records will stop being written when DEAL determines that all output streams are no longer connected. Input records continue to be written as long as at least one output stream is connected. STOP ALLEOF is the default.

 

 

ANYEOF

specifies that input records will stop being written as soon as DEAL determines that any one of its output streams is no longer connected.

 

 

streams

is the number of streams required to reach end-of-file and become disconnected before DEAL stops writing records. Specify a decimal number of 1 or greater. Specifying 1 is the same as specifying ANYEOF.

 

SECONDARY

specifies that DEAL first reads and inspects a secondary input stream record containing a stream identifier before reading a primary input stream record and writing it to that identified output stream. One secondary input stream record is processed with one primary input stream record as a pair, so the order of the records on the secondary input stream controls the order in which DEAL writes primary input stream records to its output streams.

 

KEY

identifies the location in each input record that contains the key of each record. A contiguous group of input records with the same key are written to the same output stream. When the key changes from one input record to the next; DEAL writes the record with the changed key to the next sequential, connected output stream.

 

 

inputrange

an integer column, word or field range that defines the key.

 

Streams

 

The following streams are used by the DEAL stage:

 

Stream

 

Action

 

 

Primary input stream

DEAL reads records in sequential order from its primary input stream.

Secondary input stream

DEAL reads a record from its secondary input stream, if it is connected and the SECONDARY operand is specified. The secondary input stream record is paired with a primary input stream record.

Output streams

DEAL writes each of its primary input stream records to one of its connected output streams, in either sequential order starting with the primary output stream, or the order specified on the secondary input stream.

 

Usage

 

1.

DEAL does not delay the records.

 

2.

If the DEAL stage discovers that its primary or secondary input streams are not connected, the DEAL stage ends.

 

If the SECONDARY or KEY operand is specified and the DEAL stage discovers that the output stream it needs to write the next record to is not connected; DEAL ends and any remaining input records are not fully processed.

 

If the DEAL stage discovers that the number of streams specified by the STOP operand are not connected; DEAL ends. If the number on the STOP operand is greater than the total number of output streams in the pipeline; DEAL processes as though STOP ALLEOF were specified.

 

3.

Leaving an output stream unconnected will not cause records to be dropped; it causes DEAL to end.

 

4.

With any operand other than the SECONDARY operand; DEAL verifies that its primary input stream is the only connected input stream and then begins execution. If the SECONDARY operand is specified; DEAL verifies that both its primary and secondary input streams are connected and then begins execution.

 

Examples

 

1.

The following example pipeline shows how you might use DEAL to control the order in which a particular output stream receives a record. By specifying the SECONDARY operand on the DEAL stage; the pipeline writes input records to specific output streams defined by the LITERAL stage command in the third pipeline.

 

   **** Top of file ****
 1 Address Rxpipe
 2
 3 pipe (endchar ?)',
 4      literal /a j b p l/',        /* Input record data              */
 5   '| split',                      /* Split into individual records. */
 6   '| j: deal secondary',          /* Output stream numbers are on.. */
 7    ,                              /* ..the secondary input stream.  */
 8   '| specs /Stream 0:/ 1 1-* nw', /* Identify the output stream.    */
 9   '| k: faninany',                /* Read from any input stream     */
10   '| console',                    /* Display on the console         */
11   '?',
12   'j:',
13   '| specs /Stream 1:/ 1 1-* nw', /* Identify the output stream.    */
14   '| k:',                         /* Route back to FANINANY.        */
15   '?',
16   'literal /0 1 0 1 1/',          /* Define output streams for DEAL */
17   '| split',                      /* Split into individual records. */
18   '| j:'                          /* Route back to DEAL on its..    */
19                                   /* ..secondary input stream.      */
20
21 Exit 0
   **** End of file ****

 

output:

Stream 0: a

Stream 1: j

Stream 0: b

Stream 1: p

Stream 1: l

 

2.

Similar to the previous example; this next example also uses the SECONDARY operand, with the notable difference that the target output stream/file number is extracted from the input file: salary band.txt, at column 20, and is fed back into the DEAL stage on its secondary input stream. The pipeline produces three salary band output files: level1.txt, level2.txt and level3.txt.

 

salary band.txt (input)

 

...|...+....1....+....2....+....3....+....4....

   **** Top of file ****

 1 IDENT. SALARY     BAND.   JOB TITLE

 2 ------ ---------- ------- ----------------

 3 000111  £4,000.00 L1      Associate

 4 011234 £12,000.00 L2      Staff

 5 096424  £8,000.00 L3      Senior Associate

 6 999990 £16,000.00 L2      Advisory

 7 111111 £99,000.00 L3      Senior

   **** End of file ****

 

   **** Top of file ****
 1 Address Rxpipe
 2
 3 pipe (endchar ?)',
 4   '< salary band.txt',   /* Input records containing employee info. */
 5   '| drop 2',            /* Discard the two header records. */
 6   '| f: fanout',         /* Copy input records to all output streams. */
 7   '| elastic',           /* Buffer output to avoid a pipeline stall. */
 8   '| d: deal secondary', /* Secondary input has the deal order. */
 9   '| console',           /* Stream 0 (primary) will have no records. */
10   '?',
11   'f:',                  /* Secondary output stream connected. */
12   '| specs 25.1 1',      /* Level number in column 25 is stream ID. */
13   'd:',                  /* Secondary output stream connected. */
14   '| > level1.txt',      /* Write record to first output file. */
15   '?',
16   'd:',                  /* Tertiary output stream connected. */
17   '| > level2.txt',      /* Write record to second output file. */
18   '?',
19   'd:',                  /* Quaternary output stream connected. */
20   '| > level3.txt'       /* Write record to third output file. */
21
22 Exit 0
   **** End of file ****

 

 

level1.txt (output)

 

...|...+....1....+....2....+....3....+....4....      

   **** Top of file ****

 1 000111  £4,000.00 L1      Associate

   **** End of file ****

level2.txt (output)

 

...|...+....1....+....2....+....3....+....4....

   **** Top of file ****

 1 011234 £12,000.00 L2      Staff

 2 999990 £16,000.00 L2      Advisory

   **** End of file ****

 

level3.txt (output)

 

...|...+....1....+....2....+....3....+....4....

   **** Top of file ****

 1 096424  £8,000.00 L3      Senior Associate

 2 111111 £99,000.00 L3      Senior

   **** End of file ****

 

 

Related

 

ELASTIC, FANIN, FANINANY, FANOUT

 

History

 

Version

 

Date

Action

Description

Pipelines

1.1

23.12.2021

changed

Application-wide rewrite.

2.1

1.0

09.05.2010

created

First version.

1.7