Package com.illumon.iris.db.tables.dataimport

Class BatchedImportRowBufferProcessor

java.lang.Object
com.illumon.iris.db.tables.dataimport.BatchedImportRowBufferProcessor
All Implemented Interfaces:
BinaryStoreV2RowBufferProcessor, TableListener
public abstract class BatchedImportRowBufferProcessor extends Object implements BinaryStoreV2RowBufferProcessor, TableListener
Base class for generated BinaryStoreV2RowBufferProcessor implementations that also support part of the TableListener interface for integration with the BinaryStoreImportProcessor and use a RowBatch to accumulate rows and enable vectorizable transformations and bulk writing.

  • Field Details

    • txTimestamp

      protected long txTimestamp
      The sending timestamp passed along from our source, in nanoseconds.

    • rxTimestamp

      protected long rxTimestamp
      The receiving timestamp recorded locally at this destination, in nanoseconds.

    • rowSizes

      protected final int[] rowSizes

  • Constructor Details

    • BatchedImportRowBufferProcessor

      protected BatchedImportRowBufferProcessor(int rowBatchCapacity, int parallelism, boolean requiresInstrumentation)
      Construct a BatchedImportRowBufferProcessor.
      Parameters:
      rowBatchCapacity - The capacity for row batches
      parallelism - The number of parallel writing threads to use, if > 1

    • BatchedImportRowBufferProcessor

      protected BatchedImportRowBufferProcessor(int rowBatchCapacity, boolean requiresInstrumentation)
      Construct a BatchedImportRowBufferProcessor with no parallelism.
      Parameters:
      rowBatchCapacity - The capacity for row batches

  • Method Details

    • setDataBuffer

      public final void setDataBuffer(@NotNull ByteBuffer dataBuffer, long txTimestamp, long rxTimestamp)
      Description copied from interface: BinaryStoreV2RowBufferProcessor
      Advise this buffer processor of the incoming data buffer and its associated timestamps to be used in subsequent calls to BinaryStoreV2RowBufferProcessor.processRow(Row.Flags, int). This buffer processor should record this information, and may need to clear cached objects (e.g. duplicate buffers) linked to the buffer.
      Specified by:
      setDataBuffer in interface BinaryStoreV2RowBufferProcessor
      Parameters:
      dataBuffer - The data buffer
      txTimestamp - The transmit timestamp from the source
      rxTimestamp - The receive timestamp at this destination

    • processRow

      public final void processRow(@NotNull Row.Flags flags, int rowSize) throws IOException
      Description copied from interface: BinaryStoreV2RowBufferProcessor
      Process a binary row entry. Data is guaranteed to be complete and to have passed checksum verification if such verification is enabled. The data buffer will be positioned to begin reading variable columns.

      Implementations may internally "defer" part or all of this processing until a later time, for example in order to batch transformation or columnar writing tasks. They may assume that BinaryStoreV2RowBufferProcessor.doDeferredProcessing() will be called before the contents of the current data buffer are mutated.

      Specified by:
      processRow in interface BinaryStoreV2RowBufferProcessor
      Parameters:
      flags - The flags for this row
      Throws:
      IOException

    • doDeferredProcessing

      public final void doDeferredProcessing() throws IOException
      Description copied from interface: BinaryStoreV2RowBufferProcessor
      Do any row processing previously deferred by this implementation's BinaryStoreV2RowBufferProcessor.processRow(Row.Flags, int).

      This must be invoked explicitly:

      • before changing the contents of the current data buffer
      • before processing a record
      Row buffer processors are internally responsible for making sure deferred processing results in correct operation, e.g. starting/ending transactions as appropriate and respecting internal capacity restrictions.
      Specified by:
      doDeferredProcessing in interface BinaryStoreV2RowBufferProcessor
      Throws:
      IOException

    • isDirty

      public final boolean isDirty()
      Description copied from interface: BinaryStoreV2RowBufferProcessor
      Check whether this buffer processor has any deferred row data to process via BinaryStoreV2RowBufferProcessor.doDeferredProcessing().
      Specified by:
      isDirty in interface BinaryStoreV2RowBufferProcessor
      Returns:
      Whether this buffer processor has any deferred row data to process

    • writeRowBatch

      protected abstract void writeRowBatch() throws IOException
      Implementations should write the data accumulated in rowBatch to localTableWriter's LocalAppendableColumns.
      Throws:
      IOException

    • writeRowBatchParallel

      protected abstract void writeRowBatchParallel()
      Implementations should write the data accumulated in rowBatch to localTableWriter's LocalAppendableColumns, utilizing invokeParallelTasks(ForkJoinTask[]) for parallelism.

    • invokeParallelTasks

      protected final void invokeParallelTasks(@NotNull ForkJoinTask<Void>[] tasks)
      Tool for implementations to invoke parallel tasks. Will ForkJoinTask.reinitialize() all tasks upon completion.
      Parameters:
      tasks - The tasks to execute and reinitialize

    • setDataImportProfiler

      public void setDataImportProfiler(@NotNull DataImportProfiler dataImportProfiler)
      Set a DataImportProfiler to use in instrumentation. This should be done before any calls to init(TableReader) or setWriter(TableWriter, DbTableLocation).
      Parameters:
      dataImportProfiler - The DataImportProfiler to use

    • init

      public final void init(@NotNull TableReader reader)
      Description copied from interface: TableListener
      Specify the reader to be used for subsequent rows. Typically allows listeners to obtain RowGetters for each relevant column.
      Specified by:
      init in interface TableListener
      Parameters:
      reader - The reader

    • assignRowBatchColumns

      protected abstract void assignRowBatchColumns(@NotNull RowBatch rowBatch)
      Implementations should use this as an opportunity to extract columns from the new RowBatch for use in writeRowBatch().
      Parameters:
      rowBatch - The new RowBatch

    • onNewRow

      public final void onNewRow()
      Description copied from interface: TableListener
      Called by the driving application whenever a new row that is available from the reader.
      Specified by:
      onNewRow in interface TableListener

    • setWriter

      public final void setWriter(@NotNull TableWriter writer, @Nullable DbTableLocation tableLocation)
      Description copied from interface: TableListener
      Supply the writer for output. Typically allows listeners to obtain RowSetters for each relevant column.
      Specified by:
      setWriter in interface TableListener
      Parameters:
      writer - The writer
      tableLocation - The table location object, typically used only for import details output on close

    • updateSetters

      protected abstract void updateSetters(@NotNull LocalTableWriter localTableWriter)
      Implementations should use this as an opportunity to extract columns from the new LocalTableWriter for use in writeRowBatch() and construct appropriate setters. assignRowBatchColumns(RowBatch) must have been called at least once as a precondition.
      Parameters:
      localTableWriter - The new LocalTableWriter

    • close

      public final void close() throws IOException
      Description copied from interface: TableListener
      Close this listener when no further rows will be supplied.
      Specified by:
      close in interface TableListener
      Throws:
      IOException

    • getTableDefinition

      public abstract TableDefinition getTableDefinition()
      Description copied from interface: TableListener
      Get the TableDefinition that specifies the columns this table listener expects to supply to its writer.

      Note that this should only include writable columns, and that listener implementations are only required to accurately describe the column names (DefaultColumnDefinition.getName()) and data types (ColumnDefinition.getDataType()) they expect to supply.

      Specified by:
      getTableDefinition in interface TableListener
      Returns:
      The supported TableDefinition.

    • getInstRxTime

      protected long getInstRxTime()

    • getInstTxTime

      protected long getInstTxTime()

    • getInstSize

      protected int getInstSize(int vi)