Package com.illumon.iris.importers.csv.sink

Interface AppendableColumnSink<DATA_TYPE,TARRAY>

Type Parameters:
DATA_TYPE - The column data type
TARRAY - The stored values array data type (for example Integer DataType column would have int[] array data type) on occasion this may be different for example AppendableBooleanAsByteColumnSink
All Superinterfaces:
AppendableSink<DATA_TYPE,TARRAY>, PartitionUpdatesObserver, RowUpdateObservable, io.deephaven.csv.sinks.Sink<TARRAY>
All Known Subinterfaces:
ColumnSinkHolder<DATA_TYPE,TARRAY>
All Known Implementing Classes:
AppendableColumnSinkHolder, BaseAppendableColumnSink
public interface AppendableColumnSink<DATA_TYPE,TARRAY> extends AppendableSink<DATA_TYPE,TARRAY>
The interface encapsulates the table column properties and provides wrapper methods to invoke persist calls on underlying LocalAppendableColumn

  • Method Details

    • getColumnName

      String getColumnName()
      Returns the column name retrieved from the TableDefinition of the column for this sink. This may or may not match the column name in the source csv file. See getCsvSourceColumnName()
      Returns:
      the associated TableDefinition column name.

    • getCsvSourceColumnName

      String getCsvSourceColumnName()
      Returns the source name mapping defined for the column in the schema. When source name is not explicitly defined it defaults to TableDefinition column name. When the attribute isColumnInSource() is true, this should match a column header in source file.
      Returns:
      The csv source file column header mapping of this column sink.

    • getColumnDataTransformer

      ImportColumnDataTransformer getColumnDataTransformer()
      Returns an optional Data Transformer instance if data transformation is defined for the column in schema. See supportsTransformations().
      Returns:
      an instance of ImportColumnDataTransformer when data transformation is defined for column or null.

    • supportsTransformations

      boolean supportsTransformations()
      Data transformation is considered to be supported when at least one of transform attribute or formula attribute are defined for the column in schema. When data transformation is supported the corresponding parser should use the generated transformer to transform the source file values.
      Returns:
      true when at least one of formula or transform attribute is defined in schema for the column

    • isPartitionCol

      boolean isPartitionCol()
      Returns true when the ColumnDefinition.isPartitioning() attribute for the column is set to true.
      Returns:
      true if the column is defined as a partition column in the schema

    • isColumnInSource

      boolean isColumnInSource()
      Returns true when the source name attribute in ImporterColumnDefinition is not null and the columns isConstantColumn() attribute is not true.
      Returns:
      true if the column name or source column name in schema is present in the source csv file.

    • isColumnInSchema

      boolean isColumnInSchema()
      Returns true if the column is defined in schema.

      A source csv may have columns that are not defined in the schema. Those columns need to be identified, so they can be handled appropriately to satisfy dhc plumbing requirements. In addition, if such a column is designated to be RowUpdateObserver then row updates need to be published.

      Returns:
      true if the column is defined in schema.

    • isConstantColumn

      boolean isConstantColumn()
      Returns false if column type is not set to ImporterColumnDefinition.IrisImportConstant in schema
      Returns:
      true if the ColumnDataTransformer has the hasConstant attribute as true

    • isNotConsideredPartSourceFileMapping

      boolean isNotConsideredPartSourceFileMapping()
      Returns true when the column is one of multiple columns mapped to a single source file column in the schema. In these instances a ColumnSink defined as a ColumnSinkHolder will be the Column Sink that will be passed to DHC parsers infrastructure as the ColumnSink that is mapped to the source csv column. This column sink instance should be part of the collection of sinks that is managed by the ColumnSinkHolder
      Returns:
      true if the column is part of many-to-one mapping as defined in schema w.r.t to source column

    • publishRowUpdate

      void publishRowUpdate(int size, long end)
      Delegates to RowUpdateObserver.publishRowUpdate(int, long) if the column sink was registered as one. See BaseAppendableColumnSink.registerRowUpdateObserver(RowUpdateObserver).
      Parameters:
      size - The size of the update
      end - The destination End parameter of the update

    • isColumnOnlyInSchema

      default boolean isColumnOnlyInSchema()
      Returns true if the column is defined in the schema but not mapped to the source csv file column.

      Constant columns are an example as well as Partition Columns when they do not have a source column mapping. When this attribute is true for any column in schema then one of the source csv columns will be designated as RowUpdateObserver and will be used to publish row updates. See publishRowUpdate(int, long)

      Returns:
      true when the column is in schema but is not mapped to a source column.

    • addAppendableColumn

      void addAppendableColumn(String partition, LocalAppendableColumn<DATA_TYPE> appendableColumn)
      The method provides the partition and its associated LocalAppendableColumn This information should be cached in the column sink implementation and should be used when updating the column values based on PartitionParserUpdate that will be pushed to the column when the table has a partitioning column and the values of the column should be persisted in the appropriate partition.
      Parameters:
      partition - Partition value as a string
      appendableColumn - The LocalAppendableColumn of the column associated for this partition

    • evict

      void evict(String partition)
      The provided partition should be evicted from the current partition column cache. This should be registered as one of the evicted partitions.
      Parameters:
      partition - Partition value as a string, that should be evicted from the local column cache of partitions

    • addBytes

      default void addBytes(@NotNull byte[] values, int startIndex, int length, long destEnd, boolean isSingleValue)
      The wrapper method around the LocalAppendableColumn.addBytes(byte[], int, int) The method should make sure the received values are persisted to the right partitions when the table schema supports partitions. To allow for this, the persisting of values may be blocked until the partition column is processed for the same chunk. In addition, if other columns have registered an interest in receiving current chunk details then that information should be published as the first operation.
      Parameters:
      values - The values to be persisted in the chunk
      startIndex - The start index at which to persist
      length - The length of the values that should be persisted as part of this chunk
      destEnd - The total no of rows processed up to the end of this chunk
      isSingleValue - If all the values for the update are the same constant value

    • addBooleans

      default void addBooleans(@NotNull Boolean[] values, int startIndex, int length, long destEnd, boolean isSingleValue)
      The wrapper method around the BaseAppendableColumn.addBooleans(Boolean[], int, int) The method should make sure the received values are persisted to the right partitions when the table schema supports partitions. To allow for this, the persisting of values may be blocked until the partition column is processed for the same chunk. In addition, if other columns have registered an interest in receiving current chunk details then that information should be published as the first operation.
      Parameters:
      values - The values to be persisted in the chunk
      startIndex - The start index at which to persist
      length - The length of the values that should be persisted as part of this chunk
      destEnd - The total no of rows processed up to the end of this chunk
      isSingleValue - If all the values for the update are the same constant value

    • addChars

      default void addChars(@NotNull char[] values, int startIndex, int length, long destEnd, boolean isSingleValue)
      The wrapper method around the LocalAppendableColumn.addChars(char[], int, int) The method should make sure the received values are persisted to the right partitions when the table schema supports partitions. To allow for this, the persisting of values may be blocked until the partition column is processed for the same chunk. In addition, if other columns have registered an interest in receiving current chunk details then that information should be published as the first operation.
      Parameters:
      values - The values to be persisted in the chunk
      startIndex - The start index at which to persist
      length - The length of the values that should be persisted as part of this chunk
      destEnd - The total no of rows processed up to the end of this chunk
      isSingleValue - If all the values for the update are the same constant value

    • addDoubles

      default void addDoubles(@NotNull double[] values, int startIndex, int length, long destEnd, boolean isSingleValue)
      The wrapper method around the LocalAppendableColumn.addDouble(double, long) The method should make sure the received values are persisted to the right partitions when the table schema supports partitions. To allow for this, the persisting of values may be blocked until the partition column is processed for the same chunk. In addition, if other columns have registered an interest in receiving current chunk details then that information should be published as the first operation.
      Parameters:
      values - The values to be persisted in the chunk
      startIndex - The start index at which to persist
      length - The length of the values that should be persisted as part of this chunk
      destEnd - The total no of rows processed up to the end of this chunk
      isSingleValue - If all the values for the update are the same constant value

    • addFloats

      default void addFloats(@NotNull float[] values, int startIndex, int length, long destEnd, boolean isSingleValue)
      The wrapper method around the LocalAppendableColumn.addFloats(float[], int, int) The method should make sure the received values are persisted to the right partitions when the table schema supports partitions. To allow for this, the persisting of values may be blocked until the partition column is processed for the same chunk. In addition, if other columns have registered an interest in receiving current chunk details then that information should be published as the first operation.
      Parameters:
      values - The values to be persisted in the chunk
      startIndex - The start index at which to persist
      length - The length of the values that should be persisted as part of this chunk
      destEnd - The total no of rows processed up to the end of this chunk
      isSingleValue - If all the values for the update are the same constant value

    • addInts

      default void addInts(@NotNull int[] values, int startIndex, int length, long destEnd, boolean isSingleValue)
      The wrapper method around the LocalAppendableColumn.addInts(int[], int, int) The method should make sure the received values are persisted to the right partitions when the table schema supports partitions. To allow for this, the persisting of values may be blocked until the partition column is processed for the same chunk. In addition, if other columns have registered an interest in receiving current chunk details then that information should be published as the first operation.
      Parameters:
      values - The values to be persisted in the chunk
      startIndex - The start index at which to persist
      length - The length of the values that should be persisted as part of this chunk
      destEnd - The total no of rows processed up to the end of this chunk
      isSingleValue - If all the values for the update are the same constant value

    • addLongs

      default void addLongs(@NotNull long[] values, int startIndex, int length, long destEnd, boolean isSingleValue)
      The wrapper method around the LocalAppendableColumn.addLongs(long[], int, int) The method should make sure the received values are persisted to the right partitions when the table schema supports partitions. To allow for this, the persisting of values may be blocked until the partition column is processed for the same chunk. In addition, if other columns have registered an interest in receiving current chunk details then that information should be published as the first operation.
      Parameters:
      values - The values to be persisted in the chunk
      startIndex - The start index at which to persist
      length - The length of the values that should be persisted as part of this chunk
      destEnd - The total no of rows processed up to the end of this chunk
      isSingleValue - If all the values for the update are the same constant value

    • addShorts

      default void addShorts(@NotNull short[] values, int startIndex, int length, long destEnd, boolean isSingleValue)
      The wrapper method around the LocalAppendableColumn.addShorts(short[], int, int) The method should make sure the received values are persisted to the right partitions when the table schema supports partitions. To allow for this, the persisting of values may be blocked until the partition column is processed for the same chunk. In addition, if other columns have registered an interest in receiving current chunk details then that information should be published as the first operation.
      Parameters:
      values - The values to be persisted in the chunk
      startIndex - The start index at which to persist
      length - The length of the values that should be persisted as part of this chunk
      destEnd - The total no of rows processed up to the end of this chunk
      isSingleValue - If all the values for the update are the same constant value

    • add

      default void add(@NotNull DATA_TYPE[] values, int startIndex, int length, long destEnd, boolean isSingleValue)
      The wrapper method around the BaseAppendableColumn.add(Object[], int, int) The method should make sure the received values are persisted to the right partitions when the table schema supports partitions. To allow for this, the persisting of values may be blocked until the partition column is processed for the same chunk. In addition, if other columns have registered an interest in receiving current chunk details then that information should be published as the first operation.
      Parameters:
      values - The values to be persisted in the chunk
      startIndex - The start index at which to persist
      length - The length of the values that should be persisted as part of this chunk
      destEnd - The total no of rows processed up to the end of this chunk
      isSingleValue - If all the values for the update are the same constant value