Package com.illumon.iris.db.tables.dataimport

Class BatchedImportRowBufferProcessor

java.lang.Object

com.illumon.iris.db.tables.dataimport.BatchedImportRowBufferProcessor

All Implemented Interfaces:

BinaryStoreV2RowBufferProcessor, TableListener

public abstract class BatchedImportRowBufferProcessor
extends Object
implements BinaryStoreV2RowBufferProcessor, TableListener

Base class for generated BinaryStoreV2RowBufferProcessor implementations that also support part of the TableListener interface for integration with the BinaryStoreImportProcessor and use a RowBatch to accumulate rows and enable vectorizable transformations and bulk writing.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
protected class	BatchedImportRowBufferProcessor.BooleanComputedSetter
protected class	BatchedImportRowBufferProcessor.BooleanFillSetter
protected class	BatchedImportRowBufferProcessor.ByteComputedSetter
protected class	BatchedImportRowBufferProcessor.ByteFillSetter
protected class	BatchedImportRowBufferProcessor.CharComputedSetter
protected class	BatchedImportRowBufferProcessor.CharFillSetter
protected class	BatchedImportRowBufferProcessor.DoubleComputedSetter
protected class	BatchedImportRowBufferProcessor.DoubleFillSetter
protected class	BatchedImportRowBufferProcessor.EncodedFillSetter
protected class	BatchedImportRowBufferProcessor.FloatComputedSetter
protected class	BatchedImportRowBufferProcessor.FloatFillSetter
protected class	BatchedImportRowBufferProcessor.ImportStateUpdater
static interface	BatchedImportRowBufferProcessor.IndexToBooleanFunction
static interface	BatchedImportRowBufferProcessor.IndexToByteFunction
static interface	BatchedImportRowBufferProcessor.IndexToCharFunction
static interface	BatchedImportRowBufferProcessor.IndexToDoubleFunction
static interface	BatchedImportRowBufferProcessor.IndexToFloatFunction
static interface	BatchedImportRowBufferProcessor.IndexToIntFunction
static interface	BatchedImportRowBufferProcessor.IndexToLongFunction
static interface	BatchedImportRowBufferProcessor.IndexToShortFunction
protected class	BatchedImportRowBufferProcessor.IntComputedSetter
protected class	BatchedImportRowBufferProcessor.IntFillSetter
protected class	BatchedImportRowBufferProcessor.LongComputedSetter
protected class	BatchedImportRowBufferProcessor.LongFillSetter
protected class	BatchedImportRowBufferProcessor.ObjectComputedSetter
protected class	BatchedImportRowBufferProcessor.ShortComputedSetter
protected class	BatchedImportRowBufferProcessor.ShortFillSetter

Nested classes/interfaces inherited from interface com.illumon.iris.binarystore.BinaryStoreV2RowBufferProcessor

BinaryStoreV2RowBufferProcessor.Uninitialized

Field Summary

Fields

Modifier and Type	Field	Description
protected int[]	rowSizes
protected long	rxTimestamp	The receiving timestamp recorded locally at this destination, in nanoseconds.
protected long	txTimestamp	The sending timestamp passed along from our source, in nanoseconds.

Constructor Summary

Constructors

Modifier	Constructor	Description
protected	BatchedImportRowBufferProcessor(int rowBatchCapacity, boolean requiresInstrumentation)	Construct a BatchedImportRowBufferProcessor with no parallelism.
protected	BatchedImportRowBufferProcessor(int rowBatchCapacity, int parallelism, boolean requiresInstrumentation)	Construct a BatchedImportRowBufferProcessor.

Method Summary

Modifier and Type	Method	Description
protected abstract void	assignRowBatchColumns(RowBatch rowBatch)	Implementations should use this as an opportunity to extract columns from the new RowBatch for use in writeRowBatch().
void	close()	Close this listener when no further rows will be supplied.
void	doDeferredProcessing()	Do any row processing previously deferred by this implementation's BinaryStoreV2RowBufferProcessor.processRow(Row.Flags, int).
protected long	getInstRxTime()
protected int	getInstSize(int vi)
protected long	getInstTxTime()
abstract TableDefinition	getTableDefinition()	Get the TableDefinition that specifies the columns this table listener expects to supply to its writer.
void	init(TableReader reader)	Specify the reader to be used for subsequent rows.
protected void	invokeParallelTasks(ForkJoinTask<Void>[] tasks)	Tool for implementations to invoke parallel tasks.
boolean	isDirty()	Check whether this buffer processor has any deferred row data to process via BinaryStoreV2RowBufferProcessor.doDeferredProcessing().
void	onNewRow()	Called by the driving application whenever a new row that is available from the reader.
void	processRow(Row.Flags flags, int rowSize)	Process a binary row entry.
void	setDataBuffer(ByteBuffer dataBuffer, long txTimestamp, long rxTimestamp)	Advise this buffer processor of the incoming data buffer and its associated timestamps to be used in subsequent calls to BinaryStoreV2RowBufferProcessor.processRow(Row.Flags, int).
void	setDataImportProfiler(DataImportProfiler dataImportProfiler)	Set a DataImportProfiler to use in instrumentation.
void	setWriter(TableWriter writer, DbTableLocation tableLocation)	Supply the writer for output.
protected abstract void	updateSetters(LocalTableWriter localTableWriter)	Implementations should use this as an opportunity to extract columns from the new LocalTableWriter for use in writeRowBatch() and construct appropriate setters.
protected abstract void	writeRowBatch()	Implementations should write the data accumulated in rowBatch to localTableWriter's LocalAppendableColumns.
protected abstract void	writeRowBatchParallel()	Implementations should write the data accumulated in rowBatch to localTableWriter's LocalAppendableColumns, utilizing invokeParallelTasks(ForkJoinTask[]) for parallelism.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface com.illumon.iris.db.tables.dataimport.TableListener

resumeReading, setWriter

Field Details

txTimestamp

protected long txTimestamp

The sending timestamp passed along from our source, in nanoseconds.

rxTimestamp

protected long rxTimestamp

The receiving timestamp recorded locally at this destination, in nanoseconds.

rowSizes

protected final int[] rowSizes

Constructor Details

BatchedImportRowBufferProcessor

protected BatchedImportRowBufferProcessor(int rowBatchCapacity, int parallelism, boolean requiresInstrumentation)

Construct a BatchedImportRowBufferProcessor.

Parameters:

rowBatchCapacity - The capacity for row batches

parallelism - The number of parallel writing threads to use, if > 1

BatchedImportRowBufferProcessor

protected BatchedImportRowBufferProcessor(int rowBatchCapacity, boolean requiresInstrumentation)

Construct a BatchedImportRowBufferProcessor with no parallelism.

Parameters:

rowBatchCapacity - The capacity for row batches

Method Details

setDataBuffer

public final void setDataBuffer(@NotNull ByteBuffer dataBuffer, long txTimestamp, long rxTimestamp)

Description copied from interface: BinaryStoreV2RowBufferProcessor

Advise this buffer processor of the incoming data buffer and its associated timestamps to be used in subsequent calls to BinaryStoreV2RowBufferProcessor.processRow(Row.Flags, int). This buffer processor should record this information, and may need to clear cached objects (e.g. duplicate buffers) linked to the buffer.

Specified by:

setDataBuffer in interface BinaryStoreV2RowBufferProcessor

Parameters:

dataBuffer - The data buffer

txTimestamp - The transmit timestamp from the source

rxTimestamp - The receive timestamp at this destination

processRow

public final void processRow(@NotNull Row.Flags flags, int rowSize) throws IOException

Description copied from interface: BinaryStoreV2RowBufferProcessor

Process a binary row entry. Data is guaranteed to be complete and to have passed checksum verification if such verification is enabled. The data buffer will be positioned to begin reading variable columns.

Implementations may internally "defer" part or all of this processing until a later time, for example in order to batch transformation or columnar writing tasks. They may assume that BinaryStoreV2RowBufferProcessor.doDeferredProcessing() will be called before the contents of the current data buffer are mutated.

Specified by:

processRow in interface BinaryStoreV2RowBufferProcessor

Parameters:

flags - The flags for this row

Throws:

IOException

doDeferredProcessing

public final void doDeferredProcessing() throws IOException

Description copied from interface: BinaryStoreV2RowBufferProcessor

Do any row processing previously deferred by this implementation's BinaryStoreV2RowBufferProcessor.processRow(Row.Flags, int).

This must be invoked explicitly:

• before changing the contents of the current data buffer
• before processing a record

Row buffer processors are internally responsible for making sure deferred processing results in correct operation, e.g. starting/ending transactions as appropriate and respecting internal capacity restrictions.

Specified by:

doDeferredProcessing in interface BinaryStoreV2RowBufferProcessor

Throws:

IOException

isDirty

public final boolean isDirty()

Description copied from interface: BinaryStoreV2RowBufferProcessor

Check whether this buffer processor has any deferred row data to process via BinaryStoreV2RowBufferProcessor.doDeferredProcessing().

Specified by:

isDirty in interface BinaryStoreV2RowBufferProcessor

Returns:

Whether this buffer processor has any deferred row data to process

writeRowBatch

protected abstract void writeRowBatch() throws IOException

Implementations should write the data accumulated in rowBatch to localTableWriter's LocalAppendableColumns.

Throws:

IOException

writeRowBatchParallel

protected abstract void writeRowBatchParallel()

Implementations should write the data accumulated in rowBatch to localTableWriter's LocalAppendableColumns, utilizing invokeParallelTasks(ForkJoinTask[]) for parallelism.

invokeParallelTasks

protected final void invokeParallelTasks(@NotNull ForkJoinTask<Void>[] tasks)

Tool for implementations to invoke parallel tasks. Will ForkJoinTask.reinitialize() all tasks upon completion.

Parameters:

tasks - The tasks to execute and reinitialize

setDataImportProfiler

public void setDataImportProfiler(@NotNull DataImportProfiler dataImportProfiler)

Set a DataImportProfiler to use in instrumentation. This should be done before any calls to init(TableReader) or setWriter(TableWriter, DbTableLocation).

Parameters:

dataImportProfiler - The DataImportProfiler to use

init

public final void init(@NotNull TableReader reader)

Description copied from interface: TableListener

Specify the reader to be used for subsequent rows. Typically allows listeners to obtain RowGetters for each relevant column.

Specified by:

init in interface TableListener

Parameters:

reader - The reader

assignRowBatchColumns

protected abstract void assignRowBatchColumns(@NotNull RowBatch rowBatch)

Implementations should use this as an opportunity to extract columns from the new RowBatch for use in writeRowBatch().

Parameters:

rowBatch - The new RowBatch

onNewRow

public final void onNewRow()

Description copied from interface: TableListener

Called by the driving application whenever a new row that is available from the reader.

Specified by:

onNewRow in interface TableListener

setWriter

public final void setWriter(@NotNull TableWriter writer, @Nullable DbTableLocation tableLocation)

Description copied from interface: TableListener

Supply the writer for output. Typically allows listeners to obtain RowSetters for each relevant column.

Specified by:

setWriter in interface TableListener

Parameters:

writer - The writer

tableLocation - The table location object, typically used only for import details output on close

updateSetters

protected abstract void updateSetters(@NotNull LocalTableWriter localTableWriter)

Implementations should use this as an opportunity to extract columns from the new LocalTableWriter for use in writeRowBatch() and construct appropriate setters. assignRowBatchColumns(RowBatch) must have been called at least once as a precondition.

Parameters:

localTableWriter - The new LocalTableWriter

close

public final void close() throws IOException

Description copied from interface: TableListener

Close this listener when no further rows will be supplied.

Specified by:

close in interface TableListener

Throws:

IOException

getTableDefinition

public abstract TableDefinition getTableDefinition()

Description copied from interface: TableListener

Get the TableDefinition that specifies the columns this table listener expects to supply to its writer.

Note that this should only include writable columns, and that listener implementations are only required to accurately describe the column names (DefaultColumnDefinition.getName()) and data types (ColumnDefinition.getDataType()) they expect to supply.

Specified by:

getTableDefinition in interface TableListener

Returns:

The supported TableDefinition.

getInstRxTime

protected long getInstRxTime()

getInstTxTime

protected long getInstTxTime()

getInstSize

protected int getInstSize(int vi)