Package io.deephaven.kafka.ingest

Class JsonConsumerRecordToTableWriterAdapter

java.lang.Object

io.deephaven.kafka.ingest.JsonConsumerRecordToTableWriterAdapter

All Implemented Interfaces:

ConsumerRecordToTableWriterAdapter

public class JsonConsumerRecordToTableWriterAdapter
extends Object
implements ConsumerRecordToTableWriterAdapter

Converts a consumer record containing JSON serialized objects to a Deephaven row.

After processing explicit mappings for fields; the JSON record is searched for exact matches of the key.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	JsonConsumerRecordToTableWriterAdapter.Builder	A builder to map key and value fields to table columns.

Method Summary

Modifier and Type	Method	Description
static JsonConsumerRecordToTableWriterAdapter.Builder	builder()
void	close()	Provides an implementation with a way to terminate any potential threads it has spawned
void	consumeRecord(org.apache.kafka.clients.consumer.ConsumerRecord<?,?> record, DBDateTime recvTime)	Consume a Kafka record, producing zero or more rows in the output.
void	consumeRecord(org.apache.kafka.clients.consumer.ConsumerRecord<?,?> record, DBDateTime recvTime, ProcessDataLocker processor, Runnable afterConsumed)	Consume a Kafka record, producing zero or more rows in the output.
static JsonConsumerRecordToTableWriterAdapter.Builder.NestedBuilder	nestedBuilder()
ConsumerRecordToTableWriterAdapter	setAsyncErrorHandler(Consumer<Exception> exceptionHandler)	Provides a mechanism for helper threads to provide an Exception back to the main KafkaConsumer.poll(Duration) loop
ConsumerRecordToTableWriterAdapter	setAsyncStatusUpdater(BiConsumer<Integer,Long> updateHandler)	Provides a mechanism for notification of status of parallel-parser, if one exists.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface io.deephaven.kafka.ingest.ConsumerRecordToTableWriterAdapter

defaultConsumeRecord

Method Details

builder

@ScriptApi
public static JsonConsumerRecordToTableWriterAdapter.Builder builder()

nestedBuilder

@ScriptApi
public static JsonConsumerRecordToTableWriterAdapter.Builder.NestedBuilder nestedBuilder()

consumeRecord

public void consumeRecord(org.apache.kafka.clients.consumer.ConsumerRecord<?,?> record,
 DBDateTime recvTime)
                   throws IOException

Description copied from interface: ConsumerRecordToTableWriterAdapter

Consume a Kafka record, producing zero or more rows in the output.

Specified by:

consumeRecord in interface ConsumerRecordToTableWriterAdapter

Parameters:

record - the record received from KafkaConsumer.poll(Duration).

recvTime - the time the record was received from KafkaConsumer.poll(Duration).

Throws:

IOException - if there was an error writing to the output table

consumeRecord

public void consumeRecord(org.apache.kafka.clients.consumer.ConsumerRecord<?,?> record,
 DBDateTime recvTime,
 ProcessDataLocker processor,
 Runnable afterConsumed)

Description copied from interface: ConsumerRecordToTableWriterAdapter

Consume a Kafka record, producing zero or more rows in the output. Defaulted to an implementation that parses and writes the row(s) to disk within the `processor.processData()` lock, which is a reasonable default for single threaded implementations. Implementations which parse inbound messages on multiple threads must overwrite this method

Specified by:

consumeRecord in interface ConsumerRecordToTableWriterAdapter

Parameters:

record - the record received from KafkaConsumer.poll(Duration).

recvTime - the time the record was received from KafkaConsumer.poll(Duration).

processor - a ProcessDataLocker instance, which is used as a write-lock to serialize table-write operations

afterConsumed - a Runnable executed after successful completion of the potential write(s), which is run within the context of the write-lock

setAsyncErrorHandler

public ConsumerRecordToTableWriterAdapter setAsyncErrorHandler(Consumer<Exception> exceptionHandler)

Description copied from interface: ConsumerRecordToTableWriterAdapter

Provides a mechanism for helper threads to provide an Exception back to the main KafkaConsumer.poll(Duration) loop

Specified by:

setAsyncErrorHandler in interface ConsumerRecordToTableWriterAdapter

Parameters:

exceptionHandler - an Exception handler which may process Exceptions thrown by an asynchronous message parser or by the table-writer

Returns:

this ConsumerRecordToTableWriterAdapter instance

setAsyncStatusUpdater

public ConsumerRecordToTableWriterAdapter setAsyncStatusUpdater(BiConsumer<Integer,Long> updateHandler)

Description copied from interface: ConsumerRecordToTableWriterAdapter

Provides a mechanism for notification of status of parallel-parser, if one exists. The number of items currently waiting for execution and the total number of records successfully processed are handed to the consumer

Specified by:

setAsyncStatusUpdater in interface ConsumerRecordToTableWriterAdapter

Parameters:

updateHandler - a BiConsumer, which is handed the number of items currently waiting for execution and the total number of records successfully processed

Returns:

this ConsumerRecordToTableWriterAdapter instance

close

public void close()

Description copied from interface: ConsumerRecordToTableWriterAdapter

Provides an implementation with a way to terminate any potential threads it has spawned

Specified by:

close in interface ConsumerRecordToTableWriterAdapter