Class KafkaPipelineSource<T, TRecordKey, TRecordValue>

Namespace

Pipelinez.Kafka.Source

Assembly

Pipelinez.Kafka.dll

Consumes records from Kafka and publishes them into a Pipelinez pipeline.

public class KafkaPipelineSource<T, TRecordKey, TRecordValue> : PipelineSourceBase<T>, IPipelineExecutionConfigurable, IDistributedPipelineSource<T>, IPipelineSource<T>, IFlowSource<PipelineContainer<T>> where T : PipelineRecord where TRecordKey : class where TRecordValue : class

Type Parameters

T

The pipeline record type.

TRecordKey

The Kafka key type.

TRecordValue

The Kafka value type.

Inheritance

object

PipelineSourceBase<T>

KafkaPipelineSource<T, TRecordKey, TRecordValue>

Implements

IPipelineExecutionConfigurable

IDistributedPipelineSource<T>

IPipelineSource<T>

IFlowSource<PipelineContainer<T>>

Inherited Members

PipelineSourceBase<T>.Logger

PipelineSourceBase<T>.ConnectTo(IFlowDestination<PipelineContainer<T>>, DataflowLinkOptions)

PipelineSourceBase<T>.StartAsync(CancellationTokenSource)

PipelineSourceBase<T>.PublishAsync(T)

PipelineSourceBase<T>.PublishAsync(T, PipelinePublishOptions)

PipelineSourceBase<T>.PublishAsync(T, MetadataCollection)

PipelineSourceBase<T>.PublishAsync(T, MetadataCollection, PipelinePublishOptions)

PipelineSourceBase<T>.Complete()

PipelineSourceBase<T>.Completion

PipelineSourceBase<T>.Initialize(Pipeline<T>)

PipelineSourceBase<T>.ConfigureExecutionOptions(PipelineExecutionOptions)

PipelineSourceBase<T>.GetExecutionOptions()

PipelineSourceBase<T>.ParentPipeline

object.Equals(object)

object.Equals(object, object)

object.GetHashCode()

object.GetType()

object.MemberwiseClone()

object.ReferenceEquals(object, object)

object.ToString()

Constructors

KafkaPipelineSource(string, KafkaSourceOptions, Func<TRecordKey, TRecordValue, T>)

Initializes a new Kafka-backed pipeline source.

public KafkaPipelineSource(string pipelineName, KafkaSourceOptions options, Func<TRecordKey, TRecordValue, T> recordMapper)

Parameters

pipelineName string

The owning pipeline name.

options KafkaSourceOptions

The Kafka source configuration.

recordMapper Func<TRecordKey, TRecordValue, T>

Maps Kafka key and value payloads into pipeline records.

Properties

SupportsDistributedExecution

Gets a value indicating whether the source supports distributed execution.

public bool SupportsDistributedExecution { get; }

Property Value

bool

TransportName

Gets the transport name reported in distributed metadata.

public string TransportName { get; }

Property Value

string

Methods

GetOwnedPartitions()

Gets the partitions currently owned by the source.

public IReadOnlyList<PipelinePartitionLease> GetOwnedPartitions()

Returns

IReadOnlyList<PipelinePartitionLease>

The owned partition leases.

GetPartitionExecutionStates()

Gets the current execution state for owned partitions.

public IReadOnlyList<PipelinePartitionExecutionState> GetPartitionExecutionStates()

Returns

IReadOnlyList<PipelinePartitionExecutionState>

The partition execution state snapshots.

Initialize()

Provides an opportunity for the source to initialize transport-specific state.

protected override void Initialize()

MainLoop(CancellationTokenSource)

Executes the main source loop.

protected override Task MainLoop(CancellationTokenSource cancellationToken)

Parameters

cancellationToken CancellationTokenSource

The runtime cancellation source used to stop the source.

Returns

Task

A task that completes when the source loop exits.

OnPipelineContainerComplete(object, PipelineContainerCompletedEventHandlerArgs<PipelineContainer<T>>)

Handles notification that a published container completed successfully.

public override void OnPipelineContainerComplete(object sender, PipelineContainerCompletedEventHandlerArgs<PipelineContainer<T>> e)

Parameters

sender object

The event sender.

e PipelineContainerCompletedEventHandlerArgs<PipelineContainer<T>>

The completed container event arguments.