TopologyDefinition
KSML definition
Properties
functions(object): (optional) Functions that can be referenced in producers and pipelines Default: {}.^[a-zA-Z0-9_]+$- Any of
- : Refer to #/definitions/AggregatorDefinition.
- : Refer to #/definitions/ForEachActionDefinition.
- : Refer to #/definitions/ForeignKeyExtractorDefinition.
- : Refer to #/definitions/GeneratorDefinition.
- : Refer to #/definitions/GenericFunctionDefinition.
- : Refer to #/definitions/InitializerDefinition.
- : Refer to #/definitions/KeyTransformerDefinition.
- : Refer to #/definitions/KeyValueMapperDefinition.
- : Refer to #/definitions/KeyValuePrinterDefinition.
- : Refer to #/definitions/KeyValueToKeyValueListTransformerDefinition.
- : Refer to #/definitions/KeyValueToValueListTransformerDefinition.
- : Refer to #/definitions/KeyValueTransformerDefinition.
- : Refer to #/definitions/MergerDefinition.
- : Refer to #/definitions/PredicateDefinition.
- : Refer to #/definitions/ReducerDefinition.
- : Refer to #/definitions/StreamPartitionerDefinition.
- : Refer to #/definitions/TopicNameExtractorDefinition.
- : Refer to #/definitions/ValueJoinerDefinition.
- : Refer to #/definitions/ValueTransformerDefinition.
- Any of
globalTables(object): (optional) GlobalTables that can be referenced in producers and pipelines Default: {}.^[a-zA-Z0-9_]+$: Refer to #/definitions/GlobalTableDefinition.
pipelines(object): (optional) Collection of named pipelines Default: {}.^[a-zA-Z0-9_]+$: Refer to #/definitions/PipelineDefinition.
producers(object): (optional) Collection of named producers Default: {}.^[a-zA-Z0-9_]+$: Refer to #/definitions/ProducerDefinition.
stores(object): (optional) State stores that can be referenced in producers and pipelines Default: {}.^[a-zA-Z0-9_]+$- Any of
- : Refer to #/definitions/KeyValueStateStoreDefinition.
- : Refer to #/definitions/SessionStateStoreDefinition.
- : Refer to #/definitions/WindowStateStoreDefinition.
- Any of
streams(object): (optional) Streams that can be referenced in producers and pipelines Default: {}.^[a-zA-Z0-9_]+$: Refer to #/definitions/StreamDefinition.
tables(object): (optional) Tables that can be referenced in producers and pipelines Default: {}.^[a-zA-Z0-9_]+$: Refer to #/definitions/TableDefinition.Definitions
AggregateOperation(object): An aggregate operation. Cannot contain additional properties.adder: (optional) (GroupedTable) A function that adds a record to the aggregation result.- Any of
- string
- : Refer to #/definitions/AggregatorDefinition.
- Any of
aggregator: (optional) (GroupedStream, SessionWindowedStream, TimeWindowedStream) The aggregator function, which combines a value with the previous aggregation result and outputs a new aggregation result.- Any of
- string
- : Refer to #/definitions/AggregatorDefinition.
- Any of
initializer: The initializer function, which generates an initial value for every set of aggregated records.- Any of
- string
- : Refer to #/definitions/InitializerDefinition.
- Any of
merger: (optional) (SessionWindowedStream, SessionWindowedCogroupedStream) A function that combines two aggregation results.- Any of
- string
- : Refer to #/definitions/MergerDefinition.
- Any of
name(string, required): The name of the operation processor Default: aggregate.store: (optional) Materialized view of the aggregation.- Any of
- string
- : Refer to #/definitions/WindowStateStoreDefinitionWithImplicitType.
- Any of
subtractor: (optional) (GroupedTable) A function that removes a record from the aggregation result.- Any of
- string
- : Refer to #/definitions/AggregatorDefinition.
- Any of
type: The type of the operation, fixed value “aggregate” Default: aggregate. Must be one of:["aggregate"].
AggregatorDefinition(object): Defines a aggregator function, that gets injected into the Kafka Streams topology. Cannot contain additional properties.code: (optional) The (multiline) code of the aggregator.- Any of
- boolean
- string
- Any of
expression: (optional) The expression returned by the aggregator. Only required for functions that return values.- Any of
- boolean
- string
- Any of
globalCode: (optional) Global (multiline) code that gets loaded into the Python context outside of the aggregator. Can be used for defining eg. global variables.- Any of
- boolean
- string
- Any of
name(string): (optional) The name of the aggregator. If this field is not defined, then the name is derived from the context.parameters(array): (optional) A list of parameters to be passed into the aggregator Default: [].- Items: Refer to #/definitions/ParameterDefinition.
resultType(string): (optional) The data type returned by the aggregator. Only required for function types, which are not pre-defined.stores(array): (optional) A list of store names that the aggregator uses. Only required if the function wants to use a state store. Default: [].- Items (string)
type: (optional) The type of the function, fixed value “aggregator” Default: aggregator. Must be one of:["aggregator"].
BranchDefinition(object): Defines one branch in a BranchOperation. Cannot contain additional properties.as(string): (optional) The name to register the pipeline result under, which can be used as source by follow-up pipelines.branch(array): (optional) Defines a single branch, consisting of a condition and a pipeline to execute for messages that fulfil the predicate Default: [].- Items: Refer to #/definitions/BranchDefinition.
forEach: (optional) A function that gets called for every message in the stream.- Any of
- string
- : Refer to #/definitions/ForEachActionDefinition.
- Any of
if: (optional) Defines the condition under which messages get sent down this branch.- Any of
- string
- : Refer to #/definitions/PredicateDefinition.
- Any of
name(string): (optional) The name of the operation processor Default: forEach.print: (optional) The specification of where to print messages to. Refer to #/definitions/PrintOperation.stores(array): (optional) The names of all state stores used by the function Default: [].- Items (string)
to: (optional) Ends the pipeline by sending all messages to a fixed topic, or to a topic returned by a topic name extractor function.- Any of
- string
- : Refer to #/definitions/ToOperation.
- Any of
via(array): (optional) A series of operations performed on the input stream Default: [].- Items
- Any of
- : Refer to #/definitions/AggregateOperation.
- : Refer to #/definitions/CogroupOperation.
- : Refer to #/definitions/ConvertKeyOperation.
- : Refer to #/definitions/ConvertKeyValueOperation.
- : Refer to #/definitions/ConvertValueOperation.
- : Refer to #/definitions/CountOperation.
- : Refer to #/definitions/FilterOperation.
- : Refer to #/definitions/FilterNotOperation.
- : Refer to #/definitions/TransformKeyValueToKeyValueListOperation.
- : Refer to #/definitions/TransformKeyValueToValueListOperation.
- : Refer to #/definitions/GroupByOperation.
- : Refer to #/definitions/GroupByKeyOperation.
- : Refer to #/definitions/JoinOperation.
- : Refer to #/definitions/JoinOperation.
- : Refer to #/definitions/TransformKeyValueOperation.
- : Refer to #/definitions/TransformKeyOperation.
- : Refer to #/definitions/TransformKeyValueOperation.
- : Refer to #/definitions/TransformValueOperation.
- : Refer to #/definitions/TransformValueOperation.
- : Refer to #/definitions/MergeOperation.
- : Refer to #/definitions/JoinOperation.
- : Refer to #/definitions/PeekOperation.
- : Refer to #/definitions/ReduceOperation.
- : Refer to #/definitions/RepartitionOperation.
- : Refer to #/definitions/TransformKeyOperation.
- : Refer to #/definitions/SuppressOperation.
- : Refer to #/definitions/ToStreamOperation.
- : Refer to #/definitions/ToTableOperation.
- : Refer to #/definitions/TransformKeyOperation.
- : Refer to #/definitions/TransformKeyValueOperation.
- : Refer to #/definitions/TransformKeyValueToKeyValueListOperation.
- : Refer to #/definitions/TransformKeyValueToValueListOperation.
- : Refer to #/definitions/TransformValueOperation.
- : Refer to #/definitions/WindowBySessionOperation.
- : Refer to #/definitions/WindowByTimeOperation.
- Any of
- Items
CogroupOperation(object): A cogroup operation. Cannot contain additional properties.aggregator: (GroupedStream, SessionWindowedStream, TimeWindowedStream) The aggregator function, which combines a value with the previous aggregation result and outputs a new aggregation result.- Any of
- string
- : Refer to #/definitions/AggregatorDefinition.
- Any of
name(string, required): The name of the operation processor Default: cogroup.store: (optional) Materialized view of the cogroup.- Any of
- string
- : Refer to #/definitions/WindowStateStoreDefinitionWithImplicitType.
- Any of
type: The type of the operation, fixed value “cogroup” Default: cogroup. Must be one of:["cogroup"].
ConvertKeyOperation(object): An operation to convert the stream key type to another type. Conversion is only syntactic, eg. from Avro to XML. Cannot contain additional properties.into(string, required): The type to convert the stream key into.name(string, required): The name of the operation processor Default: convertKey.type: The type of the operation, fixed value “convertKey” Default: convertKey. Must be one of:["convertKey"].
ConvertKeyValueOperation(object): An operation to convert the stream key and value types to other types. Conversion is only syntactic, eg. from Avro to XML. Cannot contain additional properties.into(string, required): The tuple type to convert the stream key/value into.name(string, required): The name of the operation processor Default: convertKeyValue.type: The type of the operation, fixed value “convertKeyValue” Default: convertKeyValue. Must be one of:["convertKeyValue"].
ConvertValueOperation(object): An operation to convert the stream value type to another type. Conversion is only syntactic, eg. from Avro to XML. Cannot contain additional properties.into(string, required): The type to convert the stream value into.name(string, required): The name of the operation processor Default: convertValue.type: The type of the operation, fixed value “convertValue” Default: convertValue. Must be one of:["convertValue"].
CountOperation(object): Count the number of times a key is seen in a given window. Cannot contain additional properties.name(string, required): The name of the operation processor Default: count.store: (optional) Materialized view of the count operation’s result.- Any of
- string
- : Refer to #/definitions/KeyValueStateStoreDefinition.
- : Refer to #/definitions/SessionStateStoreDefinition.
- : Refer to #/definitions/WindowStateStoreDefinition.
- Any of
type: The type of the operation, fixed value “count” Default: count. Must be one of:["count"].
FilterNotOperation(object): Filter records based on the inverse result of a predicate function. Cannot contain additional properties.if: A function that returns “false” when records are accepted, “true” otherwise.- Any of
- string
- : Refer to #/definitions/PredicateDefinition.
- Any of
name(string, required): The name of the operation processor Default: filterNot.store: (optional) Materialized view of the filtered table.- Any of
- string
- : Refer to #/definitions/KeyValueStateStoreDefinitionWithImplicitType.
- Any of
stores(array): (optional) The names of all state stores used by the function Default: [].- Items (string)
type: The type of the operation, fixed value “filterNot” Default: filterNot. Must be one of:["filterNot"].
FilterOperation(object): Filter records based on a predicate function. Cannot contain additional properties.if: A function that returns “true” when records are accepted, “false” otherwise.- Any of
- string
- : Refer to #/definitions/PredicateDefinition.
- Any of
name(string, required): The name of the operation processor Default: filter.store: (optional) Materialized view of the filtered table.- Any of
- string
- : Refer to #/definitions/KeyValueStateStoreDefinitionWithImplicitType.
- Any of
stores(array): (optional) The names of all state stores used by the function Default: [].- Items (string)
type: The type of the operation, fixed value “filter” Default: filter. Must be one of:["filter"].
ForEachActionDefinition(object): Defines a foreach action function, that gets injected into the Kafka Streams topology. Cannot contain additional properties.code: (optional) The (multiline) code of the foreach action.- Any of
- boolean
- string
- Any of
expression: (optional) The expression returned by the foreach action. Only required for functions that return values.- Any of
- boolean
- string
- Any of
globalCode: (optional) Global (multiline) code that gets loaded into the Python context outside of the foreach action. Can be used for defining eg. global variables.- Any of
- boolean
- string
- Any of
name(string): (optional) The name of the foreach action. If this field is not defined, then the name is derived from the context.parameters(array): (optional) A list of parameters to be passed into the foreach action Default: [].- Items: Refer to #/definitions/ParameterDefinition.
resultType(string): (optional) The data type returned by the foreach action. Only required for function types, which are not pre-defined.stores(array): (optional) A list of store names that the foreach action uses. Only required if the function wants to use a state store. Default: [].- Items (string)
type: (optional) The type of the function, fixed value “forEach” Default: forEach. Must be one of:["forEach"].
ForeignKeyExtractorDefinition(object): Defines a foreign key extractor function, that gets injected into the Kafka Streams topology. Cannot contain additional properties.code: (optional) The (multiline) code of the foreign key extractor.- Any of
- boolean
- string
- Any of
expression: (optional) The expression returned by the foreign key extractor. Only required for functions that return values.- Any of
- boolean
- string
- Any of
globalCode: (optional) Global (multiline) code that gets loaded into the Python context outside of the foreign key extractor. Can be used for defining eg. global variables.- Any of
- boolean
- string
- Any of
name(string): (optional) The name of the foreign key extractor. If this field is not defined, then the name is derived from the context.parameters(array): (optional) A list of parameters to be passed into the foreign key extractor Default: [].- Items: Refer to #/definitions/ParameterDefinition.
resultType(string): (optional) The data type returned by the foreign key extractor. Only required for function types, which are not pre-defined.stores(array): (optional) A list of store names that the foreign key extractor uses. Only required if the function wants to use a state store. Default: [].- Items (string)
type: (optional) The type of the function, fixed value “forEach” Default: forEach. Must be one of:["forEach"].
GeneratorDefinition(object): Defines a message generator function, that gets injected into the Kafka Streams topology. Cannot contain additional properties.code: (optional) The (multiline) code of the message generator.- Any of
- boolean
- string
- Any of
expression: (optional) The expression returned by the message generator. Only required for functions that return values.- Any of
- boolean
- string
- Any of
globalCode: (optional) Global (multiline) code that gets loaded into the Python context outside of the message generator. Can be used for defining eg. global variables.- Any of
- boolean
- string
- Any of
name(string): (optional) The name of the message generator. If this field is not defined, then the name is derived from the context.parameters(array): (optional) A list of parameters to be passed into the message generator Default: [].- Items: Refer to #/definitions/ParameterDefinition.
resultType(string): (optional) The data type returned by the message generator. Only required for function types, which are not pre-defined.stores(array): (optional) A list of store names that the message generator uses. Only required if the function wants to use a state store. Default: [].- Items (string)
type: (optional) The type of the function, fixed value “generator” Default: generator. Must be one of:["generator"].
GenericFunctionDefinition(object): Defines a generic function function, that gets injected into the Kafka Streams topology. Cannot contain additional properties.code: (optional) The (multiline) code of the generic function.- Any of
- boolean
- string
- Any of
expression: (optional) The expression returned by the generic function. Only required for functions that return values.- Any of
- boolean
- string
- Any of
globalCode: (optional) Global (multiline) code that gets loaded into the Python context outside of the generic function. Can be used for defining eg. global variables.- Any of
- boolean
- string
- Any of
name(string): (optional) The name of the generic function. If this field is not defined, then the name is derived from the context.parameters(array): (optional) A list of parameters to be passed into the generic function Default: [].- Items: Refer to #/definitions/ParameterDefinition.
resultType(string): (optional) The data type returned by the generic function. Only required for function types, which are not pre-defined.stores(array): (optional) A list of store names that the generic function uses. Only required if the function wants to use a state store. Default: [].- Items (string)
type: (optional) The type of the function, fixed value “generic” Default: generic. Must be one of:["generic"].
GlobalTableDefinition(object): Contains a definition of a GlobalTable, which can be referenced by producers and pipelines. Cannot contain additional properties.keyType(string, required): The key type of the global table.store: (optional) KeyValue state store definition.- Any of
- string
- : Refer to #/definitions/KeyValueStateStoreDefinitionWithImplicitType.
- Any of
topic(string, required): The name of the Kafka topic for this global table.valueType(string, required): The value type of the global table.
GlobalTableDefinitionWithOptionalTypes(object): Contains a definition of a GlobalTable, which can be referenced by producers and pipelines. Cannot contain additional properties.keyType(string): (optional) The key type of the global table.store: (optional) KeyValue state store definition.- Any of
- string
- : Refer to #/definitions/KeyValueStateStoreDefinitionWithImplicitType.
- Any of
topic(string, required): The name of the Kafka topic for this global table.valueType(string): (optional) The value type of the global table.
GroupByKeyOperation(object): Operation to group all messages with the same key together. Cannot contain additional properties.name(string, required): The name of the operation processor Default: groupByKey.store: (optional) Materialized view of the grouped stream.- Any of
- string
- : Refer to #/definitions/KeyValueStateStoreDefinitionWithImplicitType.
- Any of
type: The type of the operation, fixed value “groupByKey” Default: groupByKey. Must be one of:["groupByKey"].
GroupByOperation(object): Operation to group all messages with together based on a keying function. Cannot contain additional properties.mapper: Function to map records to a key they can be grouped on.- Any of
- string
- : Refer to #/definitions/KeyValueMapperDefinition.
- Any of
name(string, required): The name of the operation processor Default: groupBy.store: (optional) Materialized view of the grouped stream or table.- Any of
- string
- : Refer to #/definitions/KeyValueStateStoreDefinitionWithImplicitType.
- Any of
type: The type of the operation, fixed value “groupBy” Default: groupBy. Must be one of:["groupBy"].
InitializerDefinition(object): Defines a initializer function, that gets injected into the Kafka Streams topology. Cannot contain additional properties.code: (optional) The (multiline) code of the initializer.- Any of
- boolean
- string
- Any of
expression: (optional) The expression returned by the initializer. Only required for functions that return values.- Any of
- boolean
- string
- Any of
globalCode: (optional) Global (multiline) code that gets loaded into the Python context outside of the initializer. Can be used for defining eg. global variables.- Any of
- boolean
- string
- Any of
name(string): (optional) The name of the initializer. If this field is not defined, then the name is derived from the context.parameters(array): (optional) A list of parameters to be passed into the initializer Default: [].- Items: Refer to #/definitions/ParameterDefinition.
resultType(string): (optional) The data type returned by the initializer. Only required for function types, which are not pre-defined.stores(array): (optional) A list of store names that the initializer uses. Only required if the function wants to use a state store. Default: [].- Items (string)
type: (optional) The type of the function, fixed value “initializer” Default: initializer. Must be one of:["initializer"].
JoinOperation(object): Defines a join operation. Cannot contain additional properties.foreignKeyExtractor: (Table joins) A function that can translate the join table value to a primary key.- Any of
- string
- : Refer to #/definitions/ValueJoinerDefinition.
- Any of
globalTable: (Required for GlobalTable joins) A reference to the GlobalTable, or an inline definition of the GlobalTable to join with.- Any of
- string
- : Refer to #/definitions/GlobalTableDefinitionWithOptionalTypes.
- Any of
grace: (optional) (Table joins) The window grace period (the time to admit out-of-order events after the end of the window).- Any of
- number
- string
- Any of
mapper: (GlobalTable joins) A function that maps the key value from the stream with the primary key of the GlobalTable.- Any of
- string
- : Refer to #/definitions/ValueJoinerDefinition.
- Any of
name(string, required): The name of the operation processor Default: join.otherPartitioner: (optional) (Table joins) A function that partitions the records on the join table.- Any of
- string
- : Refer to #/definitions/StreamPartitionerDefinition.
- Any of
partitioner: (optional) (Table joins) A function that partitions the records on the primary table.- Any of
- string
- : Refer to #/definitions/StreamPartitionerDefinition.
- Any of
store: (optional) Materialized view of the joined streams.- Any of
- string
- : Refer to #/definitions/KeyValueStateStoreDefinition.
- : Refer to #/definitions/SessionStateStoreDefinition.
- : Refer to #/definitions/WindowStateStoreDefinition.
- Any of
stream: (Required for Stream joins) A reference to the Stream, or an inline definition of the Stream to join with.- Any of
- string
- : Refer to #/definitions/StreamDefinitionWithOptionalTypes.
- Any of
table: (Required for Table joins) A reference to the Table, or an inline definition of the Table to join with.- Any of
- string
- : Refer to #/definitions/TableDefinitionWithOptionalTypes.
- Any of
type(string, required): The type of the operation, fixed value “join”.valueJoiner: (GlobalTable joins) A function that joins two values.- Any of
- string
- : Refer to #/definitions/ValueJoinerDefinition.
- Any of
window: (Stream joins) The maximum time difference for a join over two streams on the same key.- Any of
- number
- string
- Any of
KeyTransformerDefinition(object): Defines a key transformer function, that gets injected into the Kafka Streams topology. Cannot contain additional properties.code: (optional) The (multiline) code of the key transformer.- Any of
- boolean
- string
- Any of
expression: (optional) The expression returned by the key transformer. Only required for functions that return values.- Any of
- boolean
- string
- Any of
globalCode: (optional) Global (multiline) code that gets loaded into the Python context outside of the key transformer. Can be used for defining eg. global variables.- Any of
- boolean
- string
- Any of
name(string): (optional) The name of the key transformer. If this field is not defined, then the name is derived from the context.parameters(array): (optional) A list of parameters to be passed into the key transformer Default: [].- Items: Refer to #/definitions/ParameterDefinition.
resultType(string): (optional) The data type returned by the key transformer. Only required for function types, which are not pre-defined.stores(array): (optional) A list of store names that the key transformer uses. Only required if the function wants to use a state store. Default: [].- Items (string)
type: (optional) The type of the function, fixed value “keyTransformer” Default: keyTransformer. Must be one of:["keyTransformer"].
KeyValueMapperDefinition(object): Defines a keyvalue mapper function, that gets injected into the Kafka Streams topology. Cannot contain additional properties.code: (optional) The (multiline) code of the keyvalue mapper.- Any of
- boolean
- string
- Any of
expression: (optional) The expression returned by the keyvalue mapper. Only required for functions that return values.- Any of
- boolean
- string
- Any of
globalCode: (optional) Global (multiline) code that gets loaded into the Python context outside of the keyvalue mapper. Can be used for defining eg. global variables.- Any of
- boolean
- string
- Any of
name(string): (optional) The name of the keyvalue mapper. If this field is not defined, then the name is derived from the context.parameters(array): (optional) A list of parameters to be passed into the keyvalue mapper Default: [].- Items: Refer to #/definitions/ParameterDefinition.
resultType(string): (optional) The data type returned by the keyvalue mapper. Only required for function types, which are not pre-defined.stores(array): (optional) A list of store names that the keyvalue mapper uses. Only required if the function wants to use a state store. Default: [].- Items (string)
type: (optional) The type of the function, fixed value “keyValueMapper” Default: keyValueMapper. Must be one of:["keyValueMapper"].
KeyValuePrinterDefinition(object): Defines a keyvalue printer function, that gets injected into the Kafka Streams topology. Cannot contain additional properties.code: (optional) The (multiline) code of the keyvalue printer.- Any of
- boolean
- string
- Any of
expression: (optional) The expression returned by the keyvalue printer. Only required for functions that return values.- Any of
- boolean
- string
- Any of
globalCode: (optional) Global (multiline) code that gets loaded into the Python context outside of the keyvalue printer. Can be used for defining eg. global variables.- Any of
- boolean
- string
- Any of
name(string): (optional) The name of the keyvalue printer. If this field is not defined, then the name is derived from the context.parameters(array): (optional) A list of parameters to be passed into the keyvalue printer Default: [].- Items: Refer to #/definitions/ParameterDefinition.
resultType(string): (optional) The data type returned by the keyvalue printer. Only required for function types, which are not pre-defined.stores(array): (optional) A list of store names that the keyvalue printer uses. Only required if the function wants to use a state store. Default: [].- Items (string)
type: (optional) The type of the function, fixed value “keyValuePrinter” Default: keyValuePrinter. Must be one of:["keyValuePrinter"].
KeyValueStateStoreDefinition(object): Definition of a keyValue state store. Cannot contain additional properties.caching(boolean): (optional) “true” if changed to the keyValue store need to be buffered and periodically released, “false” to emit all changes directly Default: false.historyRetention: (optional) (Versioned only) The duration for which old record versions are available for query (cannot be negative).- Any of
- number
- string
- Any of
keyType(string): (optional) The key type of the keyValue store.logging(boolean): (optional) “true” if a changelog topic should be set up on Kafka for this keyValue store, “false” otherwise Default: false.name(string): (optional) The name of the keyValue store. If this field is not defined, then the name is derived from the context.persistent(boolean): (optional) “true” if this keyValue store needs to be stored on disk, “false” otherwise Default: false.segmentInterval: (optional) Size of segments for storing old record versions (must be positive). Old record versions for the same key in a single segment are stored (updated and accessed) together. The only impact of this parameter is performance. If segments are large and a workload results in many record versions for the same key being collected in a single segment, performance may degrade as a result. On the other hand, historical reads (which access older segments) and out-of-order writes may slow down if there are too many segments.- Any of
- number
- string
- Any of
timestamped(boolean): (optional) “true” if elements in the store are timestamped, “false” otherwise Default: false.type: The type of the state store, fixed value “keyValue” Default: keyValue. Must be one of:["keyValue"].valueType(string): (optional) The value type of the keyValue store.versioned(boolean): (optional) “true” if elements in the store are versioned, “false” otherwise Default: false.
KeyValueStateStoreDefinitionWithImplicitType(object): Definition of a keyValue state store. Cannot contain additional properties.caching(boolean): (optional) “true” if changed to the keyValue store need to be buffered and periodically released, “false” to emit all changes directly Default: false.historyRetention: (optional) (Versioned only) The duration for which old record versions are available for query (cannot be negative).- Any of
- number
- string
- Any of
keyType(string): (optional) The key type of the keyValue store.logging(boolean): (optional) “true” if a changelog topic should be set up on Kafka for this keyValue store, “false” otherwise Default: false.name(string): (optional) The name of the keyValue store. If this field is not defined, then the name is derived from the context.persistent(boolean): (optional) “true” if this keyValue store needs to be stored on disk, “false” otherwise Default: false.segmentInterval: (optional) Size of segments for storing old record versions (must be positive). Old record versions for the same key in a single segment are stored (updated and accessed) together. The only impact of this parameter is performance. If segments are large and a workload results in many record versions for the same key being collected in a single segment, performance may degrade as a result. On the other hand, historical reads (which access older segments) and out-of-order writes may slow down if there are too many segments.- Any of
- number
- string
- Any of
timestamped(boolean): (optional) “true” if elements in the store are timestamped, “false” otherwise Default: false.type: (optional) The type of the state store, fixed value “keyValue” Default: keyValue. Must be one of:["keyValue"].valueType(string): (optional) The value type of the keyValue store.versioned(boolean): (optional) “true” if elements in the store are versioned, “false” otherwise Default: false.
KeyValueToKeyValueListTransformerDefinition(object): Defines a keyvalue-to-keyvaluelist transformer function, that gets injected into the Kafka Streams topology. Cannot contain additional properties.code: (optional) The (multiline) code of the keyvalue-to-keyvaluelist transformer.- Any of
- boolean
- string
- Any of
expression: (optional) The expression returned by the keyvalue-to-keyvaluelist transformer. Only required for functions that return values.- Any of
- boolean
- string
- Any of
globalCode: (optional) Global (multiline) code that gets loaded into the Python context outside of the keyvalue-to-keyvaluelist transformer. Can be used for defining eg. global variables.- Any of
- boolean
- string
- Any of
name(string): (optional) The name of the keyvalue-to-keyvaluelist transformer. If this field is not defined, then the name is derived from the context.parameters(array): (optional) A list of parameters to be passed into the keyvalue-to-keyvaluelist transformer Default: [].- Items: Refer to #/definitions/ParameterDefinition.
resultType(string): (optional) The data type returned by the keyvalue-to-keyvaluelist transformer. Only required for function types, which are not pre-defined.stores(array): (optional) A list of store names that the keyvalue-to-keyvaluelist transformer uses. Only required if the function wants to use a state store. Default: [].- Items (string)
type: (optional) The type of the function, fixed value “keyValueToKeyValueListTransformer” Default: keyValueToKeyValueListTransformer. Must be one of:["keyValueToKeyValueListTransformer"].
KeyValueToValueListTransformerDefinition(object): Defines a keyvalue-to-valuelist transformer function, that gets injected into the Kafka Streams topology. Cannot contain additional properties.code: (optional) The (multiline) code of the keyvalue-to-valuelist transformer.- Any of
- boolean
- string
- Any of
expression: (optional) The expression returned by the keyvalue-to-valuelist transformer. Only required for functions that return values.- Any of
- boolean
- string
- Any of
globalCode: (optional) Global (multiline) code that gets loaded into the Python context outside of the keyvalue-to-valuelist transformer. Can be used for defining eg. global variables.- Any of
- boolean
- string
- Any of
name(string): (optional) The name of the keyvalue-to-valuelist transformer. If this field is not defined, then the name is derived from the context.parameters(array): (optional) A list of parameters to be passed into the keyvalue-to-valuelist transformer Default: [].- Items: Refer to #/definitions/ParameterDefinition.
resultType(string): (optional) The data type returned by the keyvalue-to-valuelist transformer. Only required for function types, which are not pre-defined.stores(array): (optional) A list of store names that the keyvalue-to-valuelist transformer uses. Only required if the function wants to use a state store. Default: [].- Items (string)
type: (optional) The type of the function, fixed value “keyValueToValueListTransformer” Default: keyValueToValueListTransformer. Must be one of:["keyValueToValueListTransformer"].
KeyValueTransformerDefinition(object): Defines a keyvalue transformer function, that gets injected into the Kafka Streams topology. Cannot contain additional properties.code: (optional) The (multiline) code of the keyvalue transformer.- Any of
- boolean
- string
- Any of
expression: (optional) The expression returned by the keyvalue transformer. Only required for functions that return values.- Any of
- boolean
- string
- Any of
globalCode: (optional) Global (multiline) code that gets loaded into the Python context outside of the keyvalue transformer. Can be used for defining eg. global variables.- Any of
- boolean
- string
- Any of
name(string): (optional) The name of the keyvalue transformer. If this field is not defined, then the name is derived from the context.parameters(array): (optional) A list of parameters to be passed into the keyvalue transformer Default: [].- Items: Refer to #/definitions/ParameterDefinition.
resultType(string): (optional) The data type returned by the keyvalue transformer. Only required for function types, which are not pre-defined.stores(array): (optional) A list of store names that the keyvalue transformer uses. Only required if the function wants to use a state store. Default: [].- Items (string)
type: (optional) The type of the function, fixed value “keyValueTransformer” Default: keyValueTransformer. Must be one of:["keyValueTransformer"].
MergeOperation(object): A merge operation to join two Streams. Cannot contain additional properties.name(string, required): The name of the operation processor Default: merge.stream: The stream to merge with.- Any of
- string
- : Refer to #/definitions/StreamDefinition.
- Any of
type: The type of the operation, fixed value “merge” Default: merge. Must be one of:["merge"].
MergerDefinition(object): Defines a merger function, that gets injected into the Kafka Streams topology. Cannot contain additional properties.code: (optional) The (multiline) code of the merger.- Any of
- boolean
- string
- Any of
expression: (optional) The expression returned by the merger. Only required for functions that return values.- Any of
- boolean
- string
- Any of
globalCode: (optional) Global (multiline) code that gets loaded into the Python context outside of the merger. Can be used for defining eg. global variables.- Any of
- boolean
- string
- Any of
name(string): (optional) The name of the merger. If this field is not defined, then the name is derived from the context.parameters(array): (optional) A list of parameters to be passed into the merger Default: [].- Items: Refer to #/definitions/ParameterDefinition.
resultType(string): (optional) The data type returned by the merger. Only required for function types, which are not pre-defined.stores(array): (optional) A list of store names that the merger uses. Only required if the function wants to use a state store. Default: [].- Items (string)
type: (optional) The type of the function, fixed value “merger” Default: merger. Must be one of:["merger"].
ParameterDefinition(object): Defines a parameter for a user function. Cannot contain additional properties.defaultValue(string): (optional) The default value for the parameter.name(string, required): The name of the parameter.type(string, required): The type of the parameter.
PeekOperation(object): Operation to peek into a stream, without modifying the stream contents. Cannot contain additional properties.forEach: A function that gets called for every message in the stream.- Any of
- string
- : Refer to #/definitions/ForEachActionDefinition.
- Any of
name(string, required): The name of the operation processor Default: peek.stores(array): (optional) The names of all state stores used by the function Default: [].- Items (string)
type: The type of the operation, fixed value “peek” Default: peek. Must be one of:["peek"].
PipelineDefinition(object): Defines a pipeline through a source, a series of operations to perform on it and a sink operation to close the stream with. Cannot contain additional properties.as(string): (optional) The name to register the pipeline result under, which can be used as source by follow-up pipelines.branch(array): (optional) Defines a single branch, consisting of a condition and a pipeline to execute for messages that fulfil the predicate Default: [].- Items: Refer to #/definitions/BranchDefinition.
forEach: (optional) A function that gets called for every message in the stream.- Any of
- string
- : Refer to #/definitions/ForEachActionDefinition.
- Any of
from: Pipeline source.- Any of
- string
- : Refer to #/definitions/TopicDefinition.
- Any of
name(string): (optional) The name of the operation processor Default: forEach.print: (optional) The specification of where to print messages to. Refer to #/definitions/PrintOperation.stores(array): (optional) The names of all state stores used by the function Default: [].- Items (string)
to: (optional) Ends the pipeline by sending all messages to a fixed topic, or to a topic returned by a topic name extractor function.- Any of
- string
- : Refer to #/definitions/ToOperation.
- Any of
via(array): (optional) A series of operations performed on the input stream Default: [].- Items
- Any of
- : Refer to #/definitions/AggregateOperation.
- : Refer to #/definitions/CogroupOperation.
- : Refer to #/definitions/ConvertKeyOperation.
- : Refer to #/definitions/ConvertKeyValueOperation.
- : Refer to #/definitions/ConvertValueOperation.
- : Refer to #/definitions/CountOperation.
- : Refer to #/definitions/FilterOperation.
- : Refer to #/definitions/FilterNotOperation.
- : Refer to #/definitions/TransformKeyValueToKeyValueListOperation.
- : Refer to #/definitions/TransformKeyValueToValueListOperation.
- : Refer to #/definitions/GroupByOperation.
- : Refer to #/definitions/GroupByKeyOperation.
- : Refer to #/definitions/JoinOperation.
- : Refer to #/definitions/JoinOperation.
- : Refer to #/definitions/TransformKeyValueOperation.
- : Refer to #/definitions/TransformKeyOperation.
- : Refer to #/definitions/TransformKeyValueOperation.
- : Refer to #/definitions/TransformValueOperation.
- : Refer to #/definitions/TransformValueOperation.
- : Refer to #/definitions/MergeOperation.
- : Refer to #/definitions/JoinOperation.
- : Refer to #/definitions/PeekOperation.
- : Refer to #/definitions/ReduceOperation.
- : Refer to #/definitions/RepartitionOperation.
- : Refer to #/definitions/TransformKeyOperation.
- : Refer to #/definitions/SuppressOperation.
- : Refer to #/definitions/ToStreamOperation.
- : Refer to #/definitions/ToTableOperation.
- : Refer to #/definitions/TransformKeyOperation.
- : Refer to #/definitions/TransformKeyValueOperation.
- : Refer to #/definitions/TransformKeyValueToKeyValueListOperation.
- : Refer to #/definitions/TransformKeyValueToValueListOperation.
- : Refer to #/definitions/TransformValueOperation.
- : Refer to #/definitions/WindowBySessionOperation.
- : Refer to #/definitions/WindowByTimeOperation.
- Any of
- Items
PredicateDefinition(object): Defines a Function that returns true or false based on key/value input function, that gets injected into the Kafka Streams topology. Cannot contain additional properties.code: (optional) The (multiline) code of the Function that returns true or false based on key/value input.- Any of
- boolean
- string
- Any of
expression: (optional) The expression returned by the Function that returns true or false based on key/value input. Only required for functions that return values.- Any of
- boolean
- string
- Any of
globalCode: (optional) Global (multiline) code that gets loaded into the Python context outside of the Function that returns true or false based on key/value input. Can be used for defining eg. global variables.- Any of
- boolean
- string
- Any of
name(string): (optional) The name of the Function that returns true or false based on key/value input. If this field is not defined, then the name is derived from the context.parameters(array): (optional) A list of parameters to be passed into the Function that returns true or false based on key/value input Default: [].- Items: Refer to #/definitions/ParameterDefinition.
resultType(string): (optional) The data type returned by the Function that returns true or false based on key/value input. Only required for function types, which are not pre-defined.stores(array): (optional) A list of store names that the Function that returns true or false based on key/value input uses. Only required if the function wants to use a state store. Default: [].- Items (string)
type: (optional) The type of the function, fixed value “predicate” Default: predicate. Must be one of:["predicate"].
PrintOperation(object): Operation to print the contents of a pipeline on the screen or to write them to a file. Cannot contain additional properties.filename(string): (optional) The filename to output records to. If nothing is specified, then messages will be printed on stdout.label(string): (optional) A label to attach to the output records.mapper: (optional) A function to convert record into a string for output.- Any of
- string
- : Refer to #/definitions/KeyValuePrinterDefinition.
- Any of
name(string, required): The name of the operation processor Default: print.
ProducerDefinition(object): Definition of a Producer that regularly generates messages for a topic. Cannot contain additional properties.condition: (optional) A function that validates the generator’s result message. Returns “true” when the message may be produced on the topic, “false” otherwise.- Any of
- string
- : Refer to #/definitions/PredicateDefinition.
- Any of
generator: The function that generates records.- Any of
- string
- : Refer to #/definitions/GeneratorDefinition.
- Any of
interval: The interval with which the generator is called.- Any of
- number
- string
- Any of
to: The topic to produce to.- Any of
- string
- : Refer to #/definitions/TopicDefinitionWithOptionalTypes.
- Any of
ReduceOperation(object): Operation to reduce a series of records into a single aggregate result. Cannot contain additional properties.adder: A function that adds a record to the aggregate result.- Any of
- string
- : Refer to #/definitions/ReducerDefinition.
- Any of
name(string, required): The name of the operation processor Default: reduce.reducer: A function that computes a new aggregate result.- Any of
- string
- : Refer to #/definitions/ReducerDefinition.
- Any of
store: (optional) Materialized view of the aggregation.- Any of
- string
- : Refer to #/definitions/WindowStateStoreDefinitionWithImplicitType.
- Any of
subtractor: A function that removes a record from the aggregate result.- Any of
- string
- : Refer to #/definitions/ReducerDefinition.
- Any of
type: The type of the operation, fixed value “reduce” Default: reduce. Must be one of:["reduce"].
ReducerDefinition(object): Defines a reducer function, that gets injected into the Kafka Streams topology. Cannot contain additional properties.code: (optional) The (multiline) code of the reducer.- Any of
- boolean
- string
- Any of
expression: (optional) The expression returned by the reducer. Only required for functions that return values.- Any of
- boolean
- string
- Any of
globalCode: (optional) Global (multiline) code that gets loaded into the Python context outside of the reducer. Can be used for defining eg. global variables.- Any of
- boolean
- string
- Any of
name(string): (optional) The name of the reducer. If this field is not defined, then the name is derived from the context.parameters(array): (optional) A list of parameters to be passed into the reducer Default: [].- Items: Refer to #/definitions/ParameterDefinition.
resultType(string): (optional) The data type returned by the reducer. Only required for function types, which are not pre-defined.stores(array): (optional) A list of store names that the reducer uses. Only required if the function wants to use a state store. Default: [].- Items (string)
type: (optional) The type of the function, fixed value “reducer” Default: reducer. Must be one of:["reducer"].
RepartitionOperation(object): Operation to (re)partition a stream. Cannot contain additional properties.name(string, required): The name of the operation processor Default: repartition.partitioner: A function that partitions stream records.- Any of
- string
- : Refer to #/definitions/StreamPartitionerDefinition.
- Any of
type: The type of the operation, fixed value “repartition” Default: repartition. Must be one of:["repartition"].
SessionStateStoreDefinition(object): Definition of a session state store. Cannot contain additional properties.caching(boolean): (optional) “true” if changed to the session store need to be buffered and periodically released, “false” to emit all changes directly Default: false.keyType(string): (optional) The key type of the session store.logging(boolean): (optional) “true” if a changelog topic should be set up on Kafka for this session store, “false” otherwise Default: false.name(string): (optional) The name of the session store. If this field is not defined, then the name is derived from the context.persistent(boolean): (optional) “true” if this session store needs to be stored on disk, “false” otherwise Default: false.retention: (optional) The duration for which elements in the session store are retained.- Any of
- number
- string
- Any of
timestamped(boolean): (optional) “true” if elements in the store are timestamped, “false” otherwise Default: false.type: The type of the state store, fixed value “session” Default: session. Must be one of:["session"].valueType(string): (optional) The value type of the session store.
StreamDefinition(object): Contains a definition of a Stream, which can be referenced by producers and pipelines. Cannot contain additional properties.keyType(string, required): The key type of the stream.topic(string, required): The name of the Kafka topic for this stream.valueType(string, required): The value type of the stream.
StreamDefinitionWithOptionalTypes(object): Contains a definition of a Stream, which can be referenced by producers and pipelines. Cannot contain additional properties.keyType(string): (optional) The key type of the stream.topic(string, required): The name of the Kafka topic for this stream.valueType(string): (optional) The value type of the stream.
StreamPartitionerDefinition(object): Defines a stream partitioner function, that gets injected into the Kafka Streams topology. Cannot contain additional properties.code: (optional) The (multiline) code of the stream partitioner.- Any of
- boolean
- string
- Any of
expression: (optional) The expression returned by the stream partitioner. Only required for functions that return values.- Any of
- boolean
- string
- Any of
globalCode: (optional) Global (multiline) code that gets loaded into the Python context outside of the stream partitioner. Can be used for defining eg. global variables.- Any of
- boolean
- string
- Any of
name(string): (optional) The name of the stream partitioner. If this field is not defined, then the name is derived from the context.parameters(array): (optional) A list of parameters to be passed into the stream partitioner Default: [].- Items: Refer to #/definitions/ParameterDefinition.
resultType(string): (optional) The data type returned by the stream partitioner. Only required for function types, which are not pre-defined.stores(array): (optional) A list of store names that the stream partitioner uses. Only required if the function wants to use a state store. Default: [].- Items (string)
type: (optional) The type of the function, fixed value “streamPartitioner” Default: streamPartitioner. Must be one of:["streamPartitioner"].
SuppressOperation(object): Operation to suppress messages in the source stream until a certain limit is reached. Cannot contain additional properties.bufferFullStrategy(string): (optional) What to do when the buffer is full, either “emitEarlyWhenFull”, or “shutdownWhenFull”.duration: The duration for which messages are suppressed.- Any of
- number
- string
- Any of
maxBytes(string): (optional) The maximum number of bytes in the buffer.maxRecords(string): (optional) The maximum number of records in the buffer.name(string, required): The name of the operation processor Default: suppress.type: The type of the operation, fixed value “suppress” Default: suppress. Must be one of:["suppress"].until(string, required): The method by which messages are held, either “timeLimit”, or “windowCloses”.
TableDefinition(object): Contains a definition of a Table, which can be referenced by producers and pipelines. Cannot contain additional properties.keyType(string, required): The key type of the table.store: (optional) KeyValue state store definition.- Any of
- string
- : Refer to #/definitions/KeyValueStateStoreDefinitionWithImplicitType.
- Any of
topic(string, required): The name of the Kafka topic for this table.valueType(string, required): The value type of the table.
TableDefinitionWithOptionalTypes(object): Contains a definition of a Table, which can be referenced by producers and pipelines. Cannot contain additional properties.keyType(string): (optional) The key type of the table.store: (optional) KeyValue state store definition.- Any of
- string
- : Refer to #/definitions/KeyValueStateStoreDefinitionWithImplicitType.
- Any of
topic(string, required): The name of the Kafka topic for this table.valueType(string): (optional) The value type of the table.
ToOperation(object): Either a topic or topic name extractor that defines where to write pipeline messages to. Cannot contain additional properties.code: (optional) The (multiline) code of the stream partitioner.- Any of
- boolean
- string
- Any of
expression: (optional) The expression returned by the stream partitioner. Only required for functions that return values.- Any of
- boolean
- string
- Any of
globalCode: (optional) Global (multiline) code that gets loaded into the Python context outside of the stream partitioner. Can be used for defining eg. global variables.- Any of
- boolean
- string
- Any of
keyType(string): (optional) The key type of the topic.name(string): (optional) The name of the stream partitioner. If this field is not defined, then the name is derived from the context.parameters(array): (optional) A list of parameters to be passed into the stream partitioner Default: [].- Items: Refer to #/definitions/ParameterDefinition.
resultType(string): (optional) The data type returned by the stream partitioner. Only required for function types, which are not pre-defined.stores(array): (optional) A list of store names that the stream partitioner uses. Only required if the function wants to use a state store. Default: [].- Items (string)
topic(string): (optional) The name of the Kafka topic.topicNameExtractor: (optional) Reference to a pre-defined topic name extractor, or an inline definition of a topic name extractor and an optional stream partitioner.- Any of
- string
- : Refer to #/definitions/TopicNameExtractorDefinition.
- Any of
type: (optional) The type of the function, fixed value “streamPartitioner” Default: streamPartitioner. Must be one of:["streamPartitioner"].valueType(string): (optional) The value type of the topic.
ToStreamOperation(object): Convert a Table into a Stream, optionally through a custom key transformer. Cannot contain additional properties.mapper: A function that computes the output key for every record.- Any of
- string
- : Refer to #/definitions/KeyTransformerDefinition.
- Any of
name(string, required): The name of the operation processor Default: toStream.type: The type of the operation, fixed value “toStream” Default: toStream. Must be one of:["toStream"].
ToTableOperation(object): Convert a Stream into a Table. Cannot contain additional properties.name(string, required): The name of the operation processor Default: toTable.store: (optional) Materialized view of the Table.- Any of
- string
- : Refer to #/definitions/KeyValueStateStoreDefinitionWithImplicitType.
- Any of
type: The type of the operation, fixed value “toTable” Default: toTable. Must be one of:["toTable"].
TopicDefinition(object): Contains a definition of a Kafka topic, to be used by producers and pipelines. Cannot contain additional properties.keyType(string, required): The key type of the topic.topic(string, required): The name of the Kafka topic.valueType(string, required): The value type of the topic.
TopicDefinitionWithOptionalTypes(object): Contains a definition of a Kafka topic, to be used by producers and pipelines. Cannot contain additional properties.keyType(string): (optional) The key type of the topic.topic(string, required): The name of the Kafka topic.valueType(string): (optional) The value type of the topic.
TopicNameExtractorDefinition(object): Defines a topic name extractor function, that gets injected into the Kafka Streams topology. Cannot contain additional properties.code: (optional) The (multiline) code of the topic name extractor.- Any of
- boolean
- string
- Any of
expression: (optional) The expression returned by the topic name extractor. Only required for functions that return values.- Any of
- boolean
- string
- Any of
globalCode: (optional) Global (multiline) code that gets loaded into the Python context outside of the topic name extractor. Can be used for defining eg. global variables.- Any of
- boolean
- string
- Any of
name(string): (optional) The name of the topic name extractor. If this field is not defined, then the name is derived from the context.parameters(array): (optional) A list of parameters to be passed into the topic name extractor Default: [].- Items: Refer to #/definitions/ParameterDefinition.
resultType(string): (optional) The data type returned by the topic name extractor. Only required for function types, which are not pre-defined.stores(array): (optional) A list of store names that the topic name extractor uses. Only required if the function wants to use a state store. Default: [].- Items (string)
type: (optional) The type of the function, fixed value “topicNameExtractor” Default: topicNameExtractor. Must be one of:["topicNameExtractor"].
TransformKeyOperation(object): Convert the key of every record in the stream to another key. Cannot contain additional properties.mapper: A function that computes a new key for each record.- Any of
- string
- : Refer to #/definitions/KeyTransformerDefinition.
- Any of
name(string, required): The name of the operation processor Default: transformKey.stores(array): (optional) The names of all state stores used by the function Default: [].- Items (string)
type: The type of the operation, fixed value “transformKey” Default: transformKey. Must be one of:["transformKey"].
TransformKeyValueOperation(object): Convert the key/value of every record in the stream to another key/value. Cannot contain additional properties.mapper: A function that computes a new key/value for each record.- Any of
- string
- : Refer to #/definitions/KeyValueTransformerDefinition.
- Any of
name(string, required): The name of the operation processor Default: transformKeyValue.stores(array): (optional) The names of all state stores used by the function Default: [].- Items (string)
type: The type of the operation, fixed value “transformKeyValue” Default: transformKeyValue. Must be one of:["transformKeyValue"].
TransformKeyValueToKeyValueListOperation(object): Convert a stream by transforming every record into a list of derived records. Cannot contain additional properties.mapper: A function that converts every record of a stream to a list of output records.- Any of
- string
- : Refer to #/definitions/KeyValueToKeyValueListTransformerDefinition.
- Any of
name(string, required): The name of the operation processor Default: transformKeyValueToKeyValueList.stores(array): (optional) The names of all state stores used by the function Default: [].- Items (string)
type: The type of the operation, fixed value “transformKeyValueToKeyValueList” Default: transformKeyValueToKeyValueList. Must be one of:["transformKeyValueToKeyValueList"].
TransformKeyValueToValueListOperation(object): Convert every record in the stream to a list of output records with the same key. Cannot contain additional properties.mapper: A function that converts every key/value into a list of result values, which will be combined with the original key in the output stream.- Any of
- string
- : Refer to #/definitions/KeyValueToValueListTransformerDefinition.
- Any of
name(string, required): The name of the operation processor Default: transformKeyValueToValueList.stores(array): (optional) The names of all state stores used by the function Default: [].- Items (string)
type: The type of the operation, fixed value “transformKeyValueToValueList” Default: transformKeyValueToValueList. Must be one of:["transformKeyValueToValueList"].
TransformValueOperation(object): Convert the value of every record in the stream to another value. Cannot contain additional properties.mapper: A function that converts the value of every record into another value.- Any of
- string
- : Refer to #/definitions/ValueTransformerDefinition.
- Any of
name(string, required): The name of the operation processor Default: transformValue.store: (optional) Materialized view of the transformed table.- Any of
- string
- : Refer to #/definitions/KeyValueStateStoreDefinitionWithImplicitType.
- Any of
stores(array): (optional) The names of all state stores used by the function Default: [].- Items (string)
type: The type of the operation, fixed value “transformValue” Default: transformValue. Must be one of:["transformValue"].
ValueJoinerDefinition(object): Defines a value joiner function, that gets injected into the Kafka Streams topology. Cannot contain additional properties.code: (optional) The (multiline) code of the value joiner.- Any of
- boolean
- string
- Any of
expression: (optional) The expression returned by the value joiner. Only required for functions that return values.- Any of
- boolean
- string
- Any of
globalCode: (optional) Global (multiline) code that gets loaded into the Python context outside of the value joiner. Can be used for defining eg. global variables.- Any of
- boolean
- string
- Any of
name(string): (optional) The name of the value joiner. If this field is not defined, then the name is derived from the context.parameters(array): (optional) A list of parameters to be passed into the value joiner Default: [].- Items: Refer to #/definitions/ParameterDefinition.
resultType(string): (optional) The data type returned by the value joiner. Only required for function types, which are not pre-defined.stores(array): (optional) A list of store names that the value joiner uses. Only required if the function wants to use a state store. Default: [].- Items (string)
type: (optional) The type of the function, fixed value “valueJoiner” Default: valueJoiner. Must be one of:["valueJoiner"].
ValueTransformerDefinition(object): Defines a value transformer function, that gets injected into the Kafka Streams topology. Cannot contain additional properties.code: (optional) The (multiline) code of the value transformer.- Any of
- boolean
- string
- Any of
expression: (optional) The expression returned by the value transformer. Only required for functions that return values.- Any of
- boolean
- string
- Any of
globalCode: (optional) Global (multiline) code that gets loaded into the Python context outside of the value transformer. Can be used for defining eg. global variables.- Any of
- boolean
- string
- Any of
name(string): (optional) The name of the value transformer. If this field is not defined, then the name is derived from the context.parameters(array): (optional) A list of parameters to be passed into the value transformer Default: [].- Items: Refer to #/definitions/ParameterDefinition.
resultType(string): (optional) The data type returned by the value transformer. Only required for function types, which are not pre-defined.stores(array): (optional) A list of store names that the value transformer uses. Only required if the function wants to use a state store. Default: [].- Items (string)
type: (optional) The type of the function, fixed value “valueTransformer” Default: valueTransformer. Must be one of:["valueTransformer"].
WindowBySessionOperation(object): Operation to window messages by session, configured by an inactivity gap. Cannot contain additional properties.grace: (optional) (Tumbling, Hopping) The grace period, during which out-of-order records can still be processed.- Any of
- number
- string
- Any of
inactivityGap: The inactivity gap, below which two messages are considered to be of the same session.- Any of
- number
- string
- Any of
name(string, required): The name of the operation processor Default: windowBySession.type: The type of the operation, fixed value “windowBySession” Default: windowBySession. Must be one of:["windowBySession"].
WindowByTimeOperation(object): Operation to reduce a series of records into a single aggregate result. Cannot contain additional properties.advanceBy: (optional) (Hopping) The amount of time to increase time windows by.- Any of
- number
- string
- Any of
duration: (optional) (Tumbling) The duration of time windows.- Any of
- number
- string
- Any of
grace: (optional) (Tumbling, Hopping, Sliding) The grace period, during which out-of-order records can still be processed.- Any of
- number
- string
- Any of
name(string, required): The name of the operation processor Default: windowByTime.timeDifference: (optional) (Sliding) The maximum amount of time difference between two records.- Any of
- number
- string
- Any of
type: The type of the operation, fixed value “windowByTime” Default: windowByTime. Must be one of:["windowByTime"].windowType(string, required): The type of the operation, either “tumbling”, or “hopping”, or “sliding”.
WindowStateStoreDefinition(object): Definition of a window state store. Cannot contain additional properties.caching(boolean): (optional) “true” if changed to the window store need to be buffered and periodically released, “false” to emit all changes directly Default: false.keyType(string): (optional) The key type of the window store.logging(boolean): (optional) “true” if a changelog topic should be set up on Kafka for this window store, “false” otherwise Default: false.name(string): (optional) The name of the window store. If this field is not defined, then the name is derived from the context.persistent(boolean): (optional) “true” if this window store needs to be stored on disk, “false” otherwise Default: false.retainDuplicates(boolean): (optional) Whether or not to retain duplicates Default: false.retention: (optional) The duration for which elements in the window store are retained.- Any of
- number
- string
- Any of
timestamped(boolean): (optional) “true” if elements in the store are timestamped, “false” otherwise Default: false.type: The type of the state store, fixed value “window” Default: window. Must be one of:["window"].valueType(string): (optional) The value type of the window store.windowSize: (optional) Size of the windows (cannot be negative).- Any of
- number
- string
- Any of
WindowStateStoreDefinitionWithImplicitType(object): Definition of a window state store. Cannot contain additional properties.caching(boolean): (optional) “true” if changed to the window store need to be buffered and periodically released, “false” to emit all changes directly Default: false.keyType(string): (optional) The key type of the window store.logging(boolean): (optional) “true” if a changelog topic should be set up on Kafka for this window store, “false” otherwise Default: false.name(string): (optional) The name of the window store. If this field is not defined, then the name is derived from the context.persistent(boolean): (optional) “true” if this window store needs to be stored on disk, “false” otherwise Default: false.retainDuplicates(boolean): (optional) Whether or not to retain duplicates Default: false.retention: (optional) The duration for which elements in the window store are retained.- Any of
- number
- string
- Any of
timestamped(boolean): (optional) “true” if elements in the store are timestamped, “false” otherwise Default: false.type: (optional) The type of the state store, fixed value “window” Default: window. Must be one of:["window"].valueType(string): (optional) The value type of the window store.windowSize: (optional) Size of the windows (cannot be negative).- Any of
- number
- string
- Any of