Package io.deephaven.kafka.ingest

Class ProtobufConsumerRecordToTableWriterAdapter

java.lang.Object

io.deephaven.kafka.ingest.ProtobufConsumerRecordToTableWriterAdapter

All Implemented Interfaces:

ConsumerRecordToTableWriterAdapter

@ScriptApi
public class ProtobufConsumerRecordToTableWriterAdapter
extends Object
implements ConsumerRecordToTableWriterAdapter

Converts a consumer record containing ProtoBuf serialized objects to a Deephaven row.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	ProtobufConsumerRecordToTableWriterAdapter.Builder	A builder to map key and value fields to table columns.

Method Summary

Modifier and Type	Method	Description
static ProtobufConsumerRecordToTableWriterAdapter.Builder	builder(String protoDescriptor)
static ProtobufConsumerRecordToTableWriterAdapter.Builder	builder(Function<byte[],com.google.protobuf.Message> protoConstructor)
void	close()	Provides an implementation with a way to terminate any potential threads it has spawned
void	consumeRecord(org.apache.kafka.clients.consumer.ConsumerRecord<?,?> record, DBDateTime recvTime)	Consume a Kafka record, producing zero or more rows in the output.
void	consumeRecord(org.apache.kafka.clients.consumer.ConsumerRecord<?,?> record, DBDateTime recvTime, ProcessDataLocker processor, Runnable afterConsumed)	Consume a Kafka record, producing zero or more rows in the output.
ConsumerRecordToTableWriterAdapter	setAsyncErrorHandler(Consumer<Exception> exceptionHandler)	Provides a mechanism for helper threads to provide an Exception back to the main KafkaConsumer.poll(Duration) loop

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface io.deephaven.kafka.ingest.ConsumerRecordToTableWriterAdapter

defaultConsumeRecord, setAsyncStatusUpdater

Method Details

builder

@ScriptApi
public static ProtobufConsumerRecordToTableWriterAdapter.Builder builder(String protoDescriptor)

builder

@ScriptApi
public static ProtobufConsumerRecordToTableWriterAdapter.Builder builder(Function<byte[],com.google.protobuf.Message> protoConstructor)

consumeRecord

public void consumeRecord(org.apache.kafka.clients.consumer.ConsumerRecord<?,?> record,
 DBDateTime recvTime)
                   throws IOException

Description copied from interface: ConsumerRecordToTableWriterAdapter

Consume a Kafka record, producing zero or more rows in the output.

Specified by:

consumeRecord in interface ConsumerRecordToTableWriterAdapter

Parameters:

record - the record received from KafkaConsumer.poll(Duration).

recvTime - the time the record was received from KafkaConsumer.poll(Duration).

Throws:

IOException - if there was an error writing to the output table

consumeRecord

public void consumeRecord(org.apache.kafka.clients.consumer.ConsumerRecord<?,?> record,
 DBDateTime recvTime,
 ProcessDataLocker processor,
 Runnable afterConsumed)

Description copied from interface: ConsumerRecordToTableWriterAdapter

Consume a Kafka record, producing zero or more rows in the output. Defaulted to an implementation that parses and writes the row(s) to disk within the `processor.processData()` lock, which is a reasonable default for single threaded implementations. Implementations which parse inbound messages on multiple threads must overwrite this method

Specified by:

consumeRecord in interface ConsumerRecordToTableWriterAdapter

Parameters:

record - the record received from KafkaConsumer.poll(Duration).

recvTime - the time the record was received from KafkaConsumer.poll(Duration).

processor - a ProcessDataLocker instance, which is used as a write-lock to serialize table-write operations

afterConsumed - a Runnable executed after successful completion of the potential write(s), which is run within the context of the write-lock

setAsyncErrorHandler

public ConsumerRecordToTableWriterAdapter setAsyncErrorHandler(Consumer<Exception> exceptionHandler)

Description copied from interface: ConsumerRecordToTableWriterAdapter

Provides a mechanism for helper threads to provide an Exception back to the main KafkaConsumer.poll(Duration) loop

Specified by:

setAsyncErrorHandler in interface ConsumerRecordToTableWriterAdapter

Parameters:

exceptionHandler - an Exception handler which may process Exceptions thrown by an asynchronous message parser or by the table-writer

Returns:

this ConsumerRecordToTableWriterAdapter instance

close

public void close()

Description copied from interface: ConsumerRecordToTableWriterAdapter

Provides an implementation with a way to terminate any potential threads it has spawned

Specified by:

close in interface ConsumerRecordToTableWriterAdapter