Package com.illumon.iris.importers.csv.sink

Class AppendableColumnSinkHolder

java.lang.Object

com.illumon.iris.importers.csv.sink.BaseAppendableColumnSink<Object,Object[]>

com.illumon.iris.importers.csv.sink.AppendableColumnSinkHolder

All Implemented Interfaces:

com.fishlib.base.log.LogOutputAppendable, PartitionUpdatesObserver, RowUpdateObservable, AppendableColumnSink<Object,Object[]>, AppendableSink<Object,Object[]>, ColumnSinkHolder<Object,Object[]>, io.deephaven.csv.sinks.Sink<Object[]>, io.deephaven.csv.sinks.Source<Object[]>

public final class AppendableColumnSinkHolder
extends BaseAppendableColumnSink<Object,Object[]>
implements ColumnSinkHolder<Object,Object[]>

This column sink is a wrapper that holds multiple column sinks that are sourced from the same source column.

This by itself will not write anything to disk but acts as the sink that DHC parser attaches to the source column.

The parser for this column will delegate to the parsers of member column sinks.

Field Summary

Fields inherited from class com.illumon.iris.importers.csv.sink.BaseAppendableColumnSink

baseFieldWriter, columnDataTransformer, columnMap, columnName, constantValue, customSetterCol, DEFAULT_PARTITION_COL, evictedPartitions, isColumnInSchema, isColumnInSource, isFromSplitFile, isPartitionCol, schemaHasPartitionCol, sinkDataProcessor, updateObserver, ZERO_LENGTH_WRAPPER_ARRAY

Constructor Summary

Constructors

Constructor	Description
AppendableColumnSinkHolder(com.fishlib.io.logger.Logger log, String columnName)	Constructor.

Method Summary

Modifier and Type	Method	Description
void	add(Object[] values, int startIndex, int length, long destEnd, boolean isSingleValue)	The wrapper method around the BaseAppendableColumn.add(Object[], int, int) The method should make sure the received values are persisted to the right partitions when the table schema supports partitions.
void	addColumnSink(BaseAppendableColumnSink<?,?> columnSink)	Adds the column sink to the holder list of column sinks that are source by SinkHolder's Source column
protected void	addToAppendableColumn(LocalAppendableColumn<Object> appendableColumn, Object[] values, int chunkStartIndex, int chunkSize, boolean isSingleValue)	Writes to disk the contents of values array from chunkStartIndex to chunkSize in the appendable column of a particular partition
List<BaseAppendableColumnSink<?,?>>	getColumnSinks()	Returns the list of ColumnSinks that are part of this SinkHolder.
protected StringBuilder	getStringBuilderIncludingBasicSinkAttributes()
boolean	isColumnInSchema()	Returns true if the column is defined in schema.
boolean	isColumnInSource()	Returns true when the source name attribute in ImporterColumnDefinition is not null and the columns AppendableColumnSink.isConstantColumn() attribute is not true.
boolean	isConstantColumn()	Returns false if column type is not set to ImporterColumnDefinition.IrisImportConstant in schema
boolean	isPartitionCol()	Returns true when the ColumnDefinition.isPartitioning() attribute for the column is set to true.
void	nullFlagsToValues(Object[] values, boolean[] isNull, int size)	The method allows the appropriate null values to be populated in the chunk.
void	onPartitionParserUpdate(PartitionParserUpdate parserUpdate)	Updates are published by the source (CsvPartitionColumnParser or AppendableTableWrapper) by invoking this method.
void	publishToCustomSetter(Object[] values, boolean[] isNull, int size)	The method will publish the values chunk to CustomSetterSinkDataProcessor if it is present and applicable
boolean	supportsTransformations()	Data transformation is considered to be supported when at least one of transform attribute or formula attribute are defined for the column in schema.
void	updateCustomSetterData(int size, long destEnd)	Allows custom setter columns to process updates to next chunk of rows.
void	updateRowChunk(int size, long destEnd)	Allows columns not in csv source file to apply updates to next chunk of rows.
void	writeToLocal(Object[] values, int size, long destEnd)	The method will save the values chunk to disk.

Methods inherited from class com.illumon.iris.importers.csv.sink.BaseAppendableColumnSink

addAppendableColumn, append, awaitAdvance, evict, flush, getColumnDataTransformer, getColumnName, getConstantValue, getCsvSourceColumnName, getCustomSinkDataProcessor, isCustomSetterColumn, isNotConsideredPartSourceFileMapping, processingComplete, processingFailed, publishRowUpdate, read, registerRowUpdateObserver, setColumnIsInSinkHolder, setRowUpdatePhaser, toString, updateLocalAppendableColumn, validateForProcessingErrors

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface com.illumon.iris.importers.csv.sink.AppendableColumnSink

addAppendableColumn, addBooleans, addBytes, addChars, addDoubles, addFloats, addInts, addLongs, addShorts, evict, getColumnDataTransformer, getColumnName, getCsvSourceColumnName, isColumnOnlyInSchema, isNotConsideredPartSourceFileMapping, publishRowUpdate

Methods inherited from interface com.illumon.iris.importers.csv.sink.AppendableSink

getConstantValue, getCustomSinkDataProcessor, getUnderlying, isCustomSetterColumn, updateNonSourceColRowChunk, validateForProcessingErrors, write

Methods inherited from interface com.illumon.iris.importers.csv.RowUpdateObservable

registerRowUpdateObserver

Constructor Details

AppendableColumnSinkHolder

public AppendableColumnSinkHolder(@NotNull
 com.fishlib.io.logger.Logger log,
 @NotNull
 String columnName)

Constructor.

Parameters:

log - The passed in logger needed in Base class

columnName - The source column name

Method Details

addColumnSink

public void addColumnSink(@NotNull
 BaseAppendableColumnSink<?,?> columnSink)

Description copied from interface: ColumnSinkHolder

Adds the column sink to the holder list of column sinks that are source by SinkHolder's Source column

Specified by:

addColumnSink in interface ColumnSinkHolder<Object,Object[]>

Parameters:

columnSink - columnSink that is sourced by SinkHolder's Source column

getColumnSinks

public List<BaseAppendableColumnSink<?,?>> getColumnSinks()

Description copied from interface: ColumnSinkHolder

Returns the list of ColumnSinks that are part of this SinkHolder.

Specified by:

getColumnSinks in interface ColumnSinkHolder<Object,Object[]>

Returns:

list of ColumnSinks that are part of this SinkHolder

isColumnInSource

public boolean isColumnInSource()

Description copied from interface: AppendableColumnSink

Returns true when the source name attribute in ImporterColumnDefinition is not null and the columns AppendableColumnSink.isConstantColumn() attribute is not true.

Specified by:

isColumnInSource in interface AppendableColumnSink<Object,Object[]>

Overrides:

isColumnInSource in class BaseAppendableColumnSink<Object,Object[]>

Returns:

true if the column name or source column name in schema is present in the source csv file.

isColumnInSchema

public boolean isColumnInSchema()

Description copied from interface: AppendableColumnSink

Returns true if the column is defined in schema.

A source csv may have columns that are not defined in the schema. Those columns need to be identified, so they can be handled appropriately to satisfy dhc plumbing requirements. In addition, if such a column is designated to be RowUpdateObserver then row updates need to be published.

Specified by:

isColumnInSchema in interface AppendableColumnSink<Object,Object[]>

Overrides:

isColumnInSchema in class BaseAppendableColumnSink<Object,Object[]>

Returns:

true if the column is defined in schema.

isPartitionCol

public boolean isPartitionCol()

Description copied from interface: AppendableColumnSink

Returns true when the ColumnDefinition.isPartitioning() attribute for the column is set to true.

Specified by:

isPartitionCol in interface AppendableColumnSink<Object,Object[]>

Overrides:

isPartitionCol in class BaseAppendableColumnSink<Object,Object[]>

Returns:

true if the column is defined as a partition column in the schema

supportsTransformations

public boolean supportsTransformations()

Description copied from interface: AppendableColumnSink

Data transformation is considered to be supported when at least one of transform attribute or formula attribute are defined for the column in schema. When data transformation is supported the corresponding parser should use the generated transformer to transform the source file values.

Specified by:

supportsTransformations in interface AppendableColumnSink<Object,Object[]>

Overrides:

supportsTransformations in class BaseAppendableColumnSink<Object,Object[]>

Returns:

true when at least one of formula or transform attribute is defined in schema for the column

isConstantColumn

public boolean isConstantColumn()

Description copied from interface: AppendableColumnSink

Returns false if column type is not set to ImporterColumnDefinition.IrisImportConstant in schema

Specified by:

isConstantColumn in interface AppendableColumnSink<Object,Object[]>

Overrides:

isConstantColumn in class BaseAppendableColumnSink<Object,Object[]>

Returns:

true if the ColumnDataTransformer has the hasConstant attribute as true

add

public void add(@NotNull
 Object[] values,
 int startIndex,
 int length,
 long destEnd,
 boolean isSingleValue)

Description copied from interface: AppendableColumnSink

The wrapper method around the BaseAppendableColumn.add(Object[], int, int) The method should make sure the received values are persisted to the right partitions when the table schema supports partitions. To allow for this, the persisting of values may be blocked until the partition column is processed for the same chunk. In addition, if other columns have registered an interest in receiving current chunk details then that information should be published as the first operation.

Specified by:

add in interface AppendableColumnSink<Object,Object[]>

Parameters:

values - The values to be persisted in the chunk

startIndex - The start index at which to persist

length - The length of the values that should be persisted as part of this chunk

destEnd - The total no of rows processed up to the end of this chunk

isSingleValue - If all the values for the update are the same constant value

addToAppendableColumn

protected void addToAppendableColumn(@NotNull
 LocalAppendableColumn<Object> appendableColumn,
 @NotNull
 Object[] values,
 int chunkStartIndex,
 int chunkSize,
 boolean isSingleValue)

Description copied from class: BaseAppendableColumnSink

Writes to disk the contents of values array from chunkStartIndex to chunkSize in the appendable column of a particular partition

Specified by:

addToAppendableColumn in class BaseAppendableColumnSink<Object,Object[]>

Parameters:

appendableColumn - The LocalAppendableColumn of the current partition

values - The array whose contents need to be written to disk for this column and partition

chunkStartIndex - The start index in the values array

chunkSize - The length of data from the start index that need to be written to disk

isSingleValue - This indicates that entire chunk needs to be updated using a single value which is the constant value

onPartitionParserUpdate

public void onPartitionParserUpdate(@NotNull
 PartitionParserUpdate parserUpdate)

Description copied from interface: PartitionUpdatesObserver

Updates are published by the source (CsvPartitionColumnParser or AppendableTableWrapper) by invoking this method.

Specified by:

onPartitionParserUpdate in interface PartitionUpdatesObserver

Overrides:

onPartitionParserUpdate in class BaseAppendableColumnSink<Object,Object[]>

Parameters:

parserUpdate - Partition for current chunk.

getStringBuilderIncludingBasicSinkAttributes

@NotNull
protected StringBuilder getStringBuilderIncludingBasicSinkAttributes()

Overrides:

getStringBuilderIncludingBasicSinkAttributes in class BaseAppendableColumnSink<Object,Object[]>

nullFlagsToValues

public void nullFlagsToValues(@NotNull
 Object[] values,
 boolean[] isNull,
 int size)

Description copied from interface: AppendableSink

The method allows the appropriate null values to be populated in the chunk.

Specified by:

nullFlagsToValues in interface AppendableSink<Object,Object[]>

Parameters:

values - The chunk to populate null values if 'isNull' param is true for the index

isNull - Indicates if the cell should be null value

size - The size of the values array that should be persisted

writeToLocal

public void writeToLocal(@NotNull
 Object[] values,
 int size,
 long destEnd)

Description copied from interface: AppendableSink

The method will save the values chunk to disk.

Specified by:

writeToLocal in interface AppendableSink<Object,Object[]>

Parameters:

values - The current chunk ready to be persisted

size - The size of the values array that should be persisted

destEnd - The exclusive end index of the destination range.

publishToCustomSetter

public void publishToCustomSetter(@NotNull
 Object[] values,
 boolean[] isNull,
 int size)

Description copied from interface: AppendableSink

The method will publish the values chunk to CustomSetterSinkDataProcessor if it is present and applicable

Specified by:

publishToCustomSetter in interface AppendableSink<Object,Object[]>

Parameters:

values - The current chunk ready to be persisted

isNull - A boolean array, with the same range of valid elements. A "true" value at position i means that src[i] should be ignored and the element should be considered as the "null value".

size - The size of the values array that should be persisted

updateRowChunk

public void updateRowChunk(int size,
 long destEnd)

Description copied from interface: AppendableSink

Allows columns not in csv source file to apply updates to next chunk of rows. For example constant column values are not part of the csv source file, they will receive current row chunk details by registering with a row that is in csv source file.

Specified by:

updateRowChunk in interface AppendableSink<Object,Object[]>

Parameters:

size - The current chunk length

destEnd - The exclusive end index of the destination range.

updateCustomSetterData

public void updateCustomSetterData(int size,
 long destEnd)

Description copied from interface: AppendableSink

Allows custom setter columns to process updates to next chunk of rows. These are the steps that are expected to be done in implementing classes

1. Pull data from CustomSetterSinkDataProcessor.getAllColumnDataMap(String)
2. loop across individual row and invoke BaseCsvFieldWriter.processValues(Map, int, long)
3. Pull and populate columns value array at the index using processed value from BaseCsvFieldWriter.getSetterValue()
4. invoke appropriate type based add call, with singleValue flag set to false

Specified by:

updateCustomSetterData in interface AppendableSink<Object,Object[]>

Parameters:

size - The current chunk length

destEnd - The exclusive end index of the destination range.