|
DEAL stage v1.1 |
Pipelines v2.1 |
|
Syntax |
┌─STOP─ALLEOF────────┐>>──DEAL──┼────────────────────┼─────────────────────────────────────────────────────><
├─STOP─┬─ALLEOF──┐ │ │ ├─ANYEOF──┼───┤│ └─streams─┘ │
│ │├─SECONDARY──────────┤
│ ││ ┌─1-*────────┐ │
└─KEY─┼────────────┼─┘
└─inputrange─┘
|
Purpose |
Use the DEAL stage to read records from its primary input stream
and write records to one of its connected output streams in either sequential
order; starting with the primary output stream, or some other order specified
on the secondary input stream. Unless otherwise specified; DEAL writes the
first input record to the connected primary output stream, the second input
record to the connected secondary output stream, and so on, until the last
connected output stream receives a record, and then the cycle is repeated.
When the SECONDARY operand is specified; DEAL sequentially
processes a secondary input record and primary input record as a pair. Each
secondary input stream record must contain an integer output stream identifier
which will control the order in which DEAL writes output records. In this way
you can choose your own output stream order rather than use the sequential
default. DEAL first reads and inspects a record from the secondary input
stream, and retrieves the identity of the output stream that will receive a
written record. DEAL then reads a record from its primary input stream. When
DEAL determines that the output stream identified in the secondary input stream
is connected; DEAL writes a record from the primary input stream to the
specified output stream. DEAL then consumes the records from both its primary
and secondary input streams.
You can also write a contiguous series of primary input stream
records that have matching keys to the same output stream. DEAL searches for an
identified location in each input record that contains the specified key,
comparing the key from one record to the next. If the key location is not
specified, or it is specified but is not found, DEAL uses the entire record as
the key.
|
Operands |
|
● |
STOP specifies how many output streams must be at end-of-file for DEAL to stop writing output records. STOP ALLEOF is the default. |
||
|
|
|
||
|
|
|
||
|
|
|
||
|
● |
SECONDARY specifies that DEAL first reads and
inspects a secondary input stream record containing a stream identifier
before reading a primary input stream record and writing it to that identified
output stream. One secondary input stream record is processed with one
primary input stream record as a pair, so the order of the records on the
secondary input stream controls the order in which DEAL writes primary input
stream records to its output streams. |
||
|
● |
KEY identifies the location in each input record that contains the key of each record. A contiguous group of input records with the same key are written to the same output stream. When the key changes from one input record to the next; DEAL writes the record with the changed key to the next sequential, connected output stream. |
||
|
|
|
|
Streams |
The following streams are used by the DEAL stage:
|
|
Usage |
|
1. |
DEAL does not delay the records. |
|
2. |
If the DEAL stage discovers that
its primary or secondary input streams are not connected, the DEAL stage
ends. If the SECONDARY or KEY operand is
specified and the DEAL stage discovers that the output stream it needs to
write the next record to is not connected; DEAL ends and any remaining input
records are not fully processed. If the DEAL stage discovers that
the number of streams specified by the STOP operand are not connected; DEAL
ends. If the number on the STOP operand is greater than the total number of
output streams in the pipeline; DEAL processes as though STOP ALLEOF were
specified. |
|
3. |
Leaving an output stream
unconnected will not cause records to be dropped; it causes DEAL to end. |
|
4. |
With any operand other than the
SECONDARY operand; DEAL verifies that its primary input stream is the only
connected input stream and then begins execution. If the SECONDARY operand is
specified; DEAL verifies that both its primary and secondary input streams
are connected and then begins execution. |
|
Examples |
|
1. |
The following example pipeline
shows how you might use DEAL to control the order in which a particular
output stream receives a record. By specifying the SECONDARY operand on the
DEAL stage; the pipeline writes input records to specific output streams
defined by the LITERAL stage command in the third pipeline. **** Top of file **** 1 Address Rxpipe 23 pipe (endchar ?)', 4 literal /a j b p l/', /* Input record data */ 5 '| split', /* Split into individual records. */ 6 '| j: deal secondary', /* Output stream numbers are on.. */ 7 , /* ..the secondary input stream. */ 8 '| specs /Stream 0:/ 1 1-* nw', /* Identify the output stream. */ 9 '| k: faninany', /* Read from any input stream */ 10 '| console', /* Display on the console */ 11 '?', 12 'j:', 13 '| specs /Stream 1:/ 1 1-* nw', /* Identify the output stream. */ 14 '| k:', /* Route back to FANINANY. */ 15 '?', 16 'literal /0 1 0 1 1/', /* Define output streams for DEAL */ 17 '| split', /* Split into individual records. */ 18 '| j:' /* Route back to DEAL on its.. */ 19 /* ..secondary input stream. */ 2021 Exit 0 **** End of file ****
output: Stream 0: a Stream 1: j Stream 0: b Stream 1: p Stream 1: l |
||||
|
2. |
Similar to the
previous example; this next example also uses the SECONDARY operand, with the
notable difference that the target output stream/file number is extracted
from the input file: salary band.txt,
at column 20, and is fed back into the DEAL stage on its secondary input
stream. The pipeline produces three salary band output files: level1.txt, level2.txt and level3.txt. salary band.txt (input) ...|...+....1....+....2....+....3....+....4.... **** Top of file **** 1
IDENT. SALARY BAND. JOB TITLE 2
------ ---------- ------- ---------------- 3
000111 £4,000.00 L1 Associate 4 011234
£12,000.00 L2 Staff 5
096424 £8,000.00 L3 Senior Associate 6
999990 £16,000.00 L2 Advisory 7
111111 £99,000.00 L3 Senior **** End of file **** **** Top of file ****1 Address Rxpipe 23 pipe (endchar ?)', 4 '< salary band.txt', /* Input records containing employee info. */ 5 '| drop 2', /* Discard the two header records. */ 6 '| f: fanout', /* Copy input records to all output streams. */ 7 '| elastic', /* Buffer output to avoid a pipeline stall. */ 8 '| d: deal secondary', /* Secondary input has the deal order. */ 9 '| console', /* Stream 0 (primary) will have no records. */ 10 '?', 11 'f:', /* Secondary output stream connected. */ 12 '| specs 25.1 1', /* Level number in column 25 is stream ID. */ 13 'd:', /* Secondary output stream connected. */ 14 '| > level1.txt', /* Write record to first output file. */ 15 '?', 16 'd:', /* Tertiary output stream connected. */ 17 '| > level2.txt', /* Write record to second output file. */ 18 '?', 19 'd:', /* Quaternary output stream connected. */ 20 '| > level3.txt' /* Write record to third output file. */ 2122 Exit 0 **** End of file **** |
||||
|
|
|
|
Related |
ELASTIC,
FANIN,
FANINANY, FANOUT