« Back to index
Table of Contents
Pipelines form the heart of KSML streams logic. They take one or more input streams and apply processing logic to them. Output is passed on from operation to operation.
Pipelines are contained in the
pipelines section. Each pipeline has a name, which is the name of the YAML tag.
As an example, the following defines a pipeline called
copy_pipeline, which consumes messages from
some_input_stream and outputs the same messages to
pipelines: copy_pipeline: from: some_input_stream to: some_output_stream
All pipelines contain three elements:
The source of a pipeline is marked with the
from keyword. The value of the YAML node is the name of a defined
See Streams for more information.
Once a source was selected, a list of operations can be applied to the input. The list is started through the keyword
via, below which a list of operations is defined.
pipelines: copy_pipeline: from: some_input_stream via: - type: peek forEach: my_peek_function - type: transformKeyValue mapper: my_transform_function to: some_output_stream
Each operation must have a
type field, which indicates the type of operations applied to the input.
See Operations for a full list of operations that can be applied.
After all operations are applied, a pipeline can define a sink to which all messages are sent. There are four sink types in KSML: