Package io.deephaven.kafka.ingest

Class KafkaStreamPublisher

java.lang.Object

io.deephaven.util.referencecounting.ReferenceCounted

io.deephaven.engine.liveness.ReferenceCountedLivenessReferent

io.deephaven.engine.liveness.ReferenceCountedLivenessNode

io.deephaven.kafka.StreamPublisherBase

io.deephaven.kafka.ingest.KafkaStreamPublisher

All Implemented Interfaces:

LogOutputAppendable, LivenessManager, LivenessNode, LivenessReferent, ConsumerRecordToStreamPublisherAdapter, StreamPublisher

public class KafkaStreamPublisher
extends StreamPublisherBase
implements ConsumerRecordToStreamPublisherAdapter

An adapter that maps keys and values, possibly each with multiple fields, to single Deephaven columns. Each Kafka record produces one Deephaven row.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	KafkaStreamPublisher.Parameters

Field Summary

Fields

Modifier and Type	Field	Description
static final int	NULL_COLUMN_INDEX

Fields inherited from class io.deephaven.kafka.StreamPublisherBase

chunks, consumer

Method Summary

Modifier and Type	Method	Description
long	consumeRecords(long receiveTime, @NotNull List<? extends org.apache.kafka.clients.consumer.ConsumerRecord<?,?>> records)	Consume a List of Kafka records, producing zero or more rows in the output.
static ConsumerRecordToStreamPublisherAdapter	make(@NotNull KafkaStreamPublisher.Parameters parameters, @NotNull Runnable shutdownCallback)
void	propagateFailure(@NotNull Throwable cause)	Propagate a failure from the Kafka consumer to this StreamPublisher.
void	shutdown()	Shutdown this StreamPublisher.

Methods inherited from class io.deephaven.kafka.StreamPublisherBase

flush, getChunksToFill, register

Methods inherited from class io.deephaven.engine.liveness.ReferenceCountedLivenessNode

getWeakReference, initializeTransientFieldsForLiveness, onReferenceCountAtZero, tryManage, tryUnmanage, tryUnmanage

Methods inherited from class io.deephaven.engine.liveness.ReferenceCountedLivenessReferent

destroy, dropReference, tryRetainReference

Methods inherited from class io.deephaven.util.referencecounting.ReferenceCounted

append, decrementReferenceCount, forceReferenceCountToZero, getReferenceCountDebug, incrementReferenceCount, resetReferenceCount, toString, tryDecrementReferenceCount, tryIncrementReferenceCount

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface io.deephaven.engine.liveness.LivenessManager

manage, unmanage, unmanage

Methods inherited from interface io.deephaven.engine.liveness.LivenessReferent

dropReference, getReferentDescription, retainReference, tryRetainReference

Methods inherited from interface io.deephaven.stream.StreamPublisher

flush, register

Field Details

NULL_COLUMN_INDEX

public static final int NULL_COLUMN_INDEX

See Also:

• Constant Field Values

Method Details

make

public static ConsumerRecordToStreamPublisherAdapter make(@NotNull
 @NotNull KafkaStreamPublisher.Parameters parameters,
 @NotNull
 @NotNull Runnable shutdownCallback)

propagateFailure

public void propagateFailure(@NotNull
 @NotNull Throwable cause)

Description copied from interface: ConsumerRecordToStreamPublisherAdapter

Propagate a failure from the Kafka consumer to this StreamPublisher.

Specified by:

propagateFailure in interface ConsumerRecordToStreamPublisherAdapter

Parameters:

cause - The failure to propagate

consumeRecords

public long consumeRecords(long receiveTime,
 @NotNull
 @NotNull List<? extends org.apache.kafka.clients.consumer.ConsumerRecord<?,?>> records)

Description copied from interface: ConsumerRecordToStreamPublisherAdapter

Consume a List of Kafka records, producing zero or more rows in the output.

Specified by:

consumeRecords in interface ConsumerRecordToStreamPublisherAdapter

Parameters:

receiveTime - the time, in nanoseconds since the epoch, the records were received in this process

records - the records received from KafkaConsumer.poll(Duration).

Returns:

the number of bytes processed

shutdown

public void shutdown()

Description copied from interface: StreamPublisher

Shutdown this StreamPublisher. Implementations should stop publishing new data and release any related resources as soon as practicable.

Specified by:

shutdown in interface StreamPublisher