Package io.deephaven.importers.csv.sink

Class AppendableColumnSinkHolder

java.lang.Object
io.deephaven.importers.csv.sink.BaseAppendableColumnSink<Object,Object[]>
io.deephaven.importers.csv.sink.AppendableColumnSinkHolder
All Implemented Interfaces:
LogOutputAppendable, io.deephaven.csv.sinks.Sink<Object[]>, io.deephaven.csv.sinks.Source<Object[]>, PartitionUpdatesObserver, RowUpdateObservable, AppendableColumnSink<Object,Object[]>, AppendableSink<Object,Object[]>, ColumnSinkHolder<Object,Object[]>
public final class AppendableColumnSinkHolder extends BaseAppendableColumnSink<Object,Object[]> implements ColumnSinkHolder<Object,Object[]>
This column sink is a wrapper that holds multiple column sinks that are sourced from the same source column.

This by itself will not write anything to disk but acts as the sink that DHC parser attaches to the source column.

The parser for this column will delegate to the parsers of member column sinks.

  • Constructor Details

    • AppendableColumnSinkHolder

      public AppendableColumnSinkHolder(@NotNull @NotNull Logger log, @NotNull @NotNull String columnName)
      Constructor.
      Parameters:
      log - The passed in logger needed in Base class
      columnName - The source column name

  • Method Details

    • addColumnSink

      public void addColumnSink(@NotNull @NotNull BaseAppendableColumnSink<?,?> columnSink)
      Description copied from interface: ColumnSinkHolder
      Adds the column sink to the holder list of column sinks that are source by SinkHolder's Source column
      Specified by:
      addColumnSink in interface ColumnSinkHolder<Object,Object[]>
      Parameters:
      columnSink - columnSink that is sourced by SinkHolder's Source column

    • getColumnSinks

      public List<BaseAppendableColumnSink<?,?>> getColumnSinks()
      Description copied from interface: ColumnSinkHolder
      Returns the list of ColumnSinks that are part of this SinkHolder.
      Specified by:
      getColumnSinks in interface ColumnSinkHolder<Object,Object[]>
      Returns:
      list of ColumnSinks that are part of this SinkHolder

    • isColumnInSource

      public boolean isColumnInSource()
      Description copied from interface: AppendableColumnSink
      Returns true when the source name attribute in ImporterColumnDefinition is not null and the columns AppendableColumnSink.isConstantColumn() attribute is not true.
      Specified by:
      isColumnInSource in interface AppendableColumnSink<Object,Object[]>
      Overrides:
      isColumnInSource in class BaseAppendableColumnSink<Object,Object[]>
      Returns:
      true if the column name or source column name in schema is present in the source csv file.

    • isColumnInSchema

      public boolean isColumnInSchema()
      Description copied from interface: AppendableColumnSink
      Returns true if the column is defined in schema.

      A source csv may have columns that are not defined in the schema. Those columns need to be identified, so they can be handled appropriately to satisfy dhc plumbing requirements. In addition, if such a column is designated to be RowUpdateObserver then row updates need to be published.

      Specified by:
      isColumnInSchema in interface AppendableColumnSink<Object,Object[]>
      Overrides:
      isColumnInSchema in class BaseAppendableColumnSink<Object,Object[]>
      Returns:
      true if the column is defined in schema.

    • isPartitionCol

      public boolean isPartitionCol()
      Description copied from interface: AppendableColumnSink
      Returns true when the ImporterColumnDefinition.isPartitioning() attribute for the column is set to true.
      Specified by:
      isPartitionCol in interface AppendableColumnSink<Object,Object[]>
      Overrides:
      isPartitionCol in class BaseAppendableColumnSink<Object,Object[]>
      Returns:
      true if the column is defined as a partition column in the schema

    • supportsTransformations

      public boolean supportsTransformations()
      Description copied from interface: AppendableColumnSink
      Data transformation is considered to be supported when at least one of transform attribute or formula attribute are defined for the column in schema. When data transformation is supported the corresponding parser should use the generated transformer to transform the source file values.
      Specified by:
      supportsTransformations in interface AppendableColumnSink<Object,Object[]>
      Overrides:
      supportsTransformations in class BaseAppendableColumnSink<Object,Object[]>
      Returns:
      true when at least one of formula or transform attribute is defined in schema for the column

    • isConstantColumn

      public boolean isConstantColumn()
      Description copied from interface: AppendableColumnSink
      Returns false if column type is not set to ImporterColumnDefinition.IrisImportConstant in schema
      Specified by:
      isConstantColumn in interface AppendableColumnSink<Object,Object[]>
      Overrides:
      isConstantColumn in class BaseAppendableColumnSink<Object,Object[]>
      Returns:
      true if the ColumnDataTransformer has the hasConstant attribute as true

    • add

      public void add(@NotNull @NotNull Object[] values, int startIndex, int length, long destEnd, boolean isSingleValue)
      Description copied from interface: AppendableColumnSink
      The wrapper method around the BaseAppendableColumn.add(Object[], int, int) The method should make sure the received values are persisted to the right partitions when the table schema supports partitions. To allow for this, the persisting of values may be blocked until the partition column is processed for the same chunk. In addition, if other columns have registered an interest in receiving current chunk details then that information should be published as the first operation.
      Specified by:
      add in interface AppendableColumnSink<Object,Object[]>
      Parameters:
      values - The values to be persisted in the chunk
      startIndex - The start index at which to persist
      length - The length of the values that should be persisted as part of this chunk
      destEnd - The total no of rows processed up to the end of this chunk
      isSingleValue - If all the values for the update are the same constant value

    • addToAppendableColumn

      protected void addToAppendableColumn(@NotNull @NotNull io.deephaven.shadow.enterprise.com.illumon.iris.db.tables.appendable.local.LocalAppendableColumn<Object> appendableColumn, Object @NotNull [] values, int chunkStartIndex, int chunkSize, boolean isSingleValue)
      Description copied from class: BaseAppendableColumnSink
      Writes to disk the contents of values array from chunkStartIndex to chunkSize in the appendable column of a particular partition
      Specified by:
      addToAppendableColumn in class BaseAppendableColumnSink<Object,Object[]>
      Parameters:
      appendableColumn - The LocalAppendableColumn of the current partition
      values - The array whose contents need to be written to disk for this column and partition
      chunkStartIndex - The start index in the values array
      chunkSize - The length of data from the start index that need to be written to disk
      isSingleValue - This indicates that entire chunk needs to be updated using a single value which is the constant value

    • onPartitionParserUpdate

      public void onPartitionParserUpdate(@NotNull @NotNull PartitionParserUpdate parserUpdate)
      Description copied from interface: PartitionUpdatesObserver
      Updates are published by the source (CsvPartitionColumnParser or AppendableTableWrapper) by invoking this method.
      Specified by:
      onPartitionParserUpdate in interface PartitionUpdatesObserver
      Overrides:
      onPartitionParserUpdate in class BaseAppendableColumnSink<Object,Object[]>
      Parameters:
      parserUpdate - Partition for current chunk.

    • getStringBuilderIncludingBasicSinkAttributes

      @NotNull protected @NotNull StringBuilder getStringBuilderIncludingBasicSinkAttributes()
      Overrides:
      getStringBuilderIncludingBasicSinkAttributes in class BaseAppendableColumnSink<Object,Object[]>

    • nullFlagsToValues

      public void nullFlagsToValues(Object @NotNull [] values, boolean[] isNull, int size)
      Description copied from interface: AppendableSink
      The method allows the appropriate null values to be populated in the chunk.
      Specified by:
      nullFlagsToValues in interface AppendableSink<Object,Object[]>
      Parameters:
      values - The chunk to populate null values if 'isNull' param is true for the index
      isNull - Indicates if the cell should be null value
      size - The size of the values array that should be persisted

    • writeToLocal

      public void writeToLocal(Object @NotNull [] values, int size, long destEnd)
      Description copied from interface: AppendableSink
      The method will save the values chunk to disk.
      Specified by:
      writeToLocal in interface AppendableSink<Object,Object[]>
      Parameters:
      values - The current chunk ready to be persisted
      size - The size of the values array that should be persisted
      destEnd - The exclusive end index of the destination range.

    • publishToCustomSetter

      public void publishToCustomSetter(Object @NotNull [] values, boolean[] isNull, int size)
      Description copied from interface: AppendableSink
      The method will publish the values chunk to CustomSetterSinkDataProcessor if it is present and applicable
      Specified by:
      publishToCustomSetter in interface AppendableSink<Object,Object[]>
      Parameters:
      values - The current chunk ready to be persisted
      isNull - A boolean array, with the same range of valid elements. A "true" value at position i means that src[i] should be ignored and the element should be considered as the "null value".
      size - The size of the values array that should be persisted

    • updateRowChunk

      public void updateRowChunk(int size, long destEnd)
      Description copied from interface: AppendableSink
      Allows columns not in csv source file to apply updates to next chunk of rows. For example constant column values are not part of the csv source file, they will receive current row chunk details by registering with a row that is in csv source file.
      Specified by:
      updateRowChunk in interface AppendableSink<Object,Object[]>
      Parameters:
      size - The current chunk length
      destEnd - The exclusive end index of the destination range.

    • updateCustomSetterData

      public void updateCustomSetterData(int size, long destEnd)
      Description copied from interface: AppendableSink
      Allows custom setter columns to process updates to next chunk of rows. These are the steps that are expected to be done in implementing classes
      1. Pull data from CustomSetterSinkDataProcessor.getAllColumnDataMap(String)
      2. loop across individual row and invoke BaseCsvFieldWriter.processValues(Map, int, long)
      3. Pull and populate columns value array at the index using processed value from BaseCsvFieldWriter.getSetterValue()
      4. invoke appropriate type based add call, with singleValue flag set to false
      Specified by:
      updateCustomSetterData in interface AppendableSink<Object,Object[]>
      Parameters:
      size - The current chunk length
      destEnd - The exclusive end index of the destination range.