Package com.illumon.iris.importers.csv.sink

Class BaseAppendableColumnSink<DATA_TYPE,TARRAY>

java.lang.Object

com.illumon.iris.importers.csv.sink.BaseAppendableColumnSink<DATA_TYPE,TARRAY>

All Implemented Interfaces:

com.fishlib.base.log.LogOutputAppendable, PartitionUpdatesObserver, AppendableColumnSink<DATA_TYPE,TARRAY>, AppendableSink<DATA_TYPE,TARRAY>, io.deephaven.csv.sinks.Sink<TARRAY>, io.deephaven.csv.sinks.Source<TARRAY>

public abstract class BaseAppendableColumnSink<DATA_TYPE,TARRAY>
extends Object
implements AppendableColumnSink<DATA_TYPE,TARRAY>, io.deephaven.csv.sinks.Source<TARRAY>, com.fishlib.base.log.LogOutputAppendable

Field Summary

Fields

Modifier and Type	Field	Description
protected ImportColumnDataTransformer	columnDataTransformer
protected Map<String,LocalAppendableColumn<DATA_TYPE>>	columnMap
protected String	columnName
protected DATA_TYPE	constantValue
static String	DEFAULT_PARTITION_COL
protected Set<String>	evictedPartitions
protected boolean	isColumnInSchema
protected boolean	isColumnInSource
protected boolean	isFromSplitFile
protected boolean	isPartitionCol
protected boolean	schemaHasPartitionCol
protected RowUpdateObserver	updateObserver
static BaseAppendableColumnSink[]	ZERO_LENGTH_WRAPPER_ARRAY

Constructor Summary

Constructors

Modifier	Constructor	Description
protected	BaseAppendableColumnSink(com.fishlib.io.logger.Logger log, String columnName, ImporterColumnDefinition icdColDef, ImportColumnDataTransformer columnDataTransformer, boolean isPartitionCol, boolean isColumnInSchema, boolean isColumnInSource, boolean schemaHasPartitionCol, boolean fromSplitFile)

Method Summary

Modifier and Type	Method	Description
void	addAppendableColumn(String partition, LocalAppendableColumn<DATA_TYPE> appendableColumn)	The method provides the partition and its associated LocalAppendableColumn This information should be cached in the column sink implementation and should be used when updating the column values based on PartitionParserUpdate that will be pushed to the column when the table has a partitioning column and the values of the column should be persisted in the appropriate partition.
protected abstract void	addToAppendableColumn(LocalAppendableColumn<DATA_TYPE> appendableColumn, TARRAY values, int chunkStartIndex, int chunkSize, boolean isSingleValue)	Writes to disk the contents of values array from chunkStartIndex to chunkSize in the appendable column of a particular partition
com.fishlib.base.log.LogOutput	append(com.fishlib.base.log.LogOutput logOutput)
void	evict(String partition)	The provided partition should be evicted from the current partition column cache.
ImportColumnDataTransformer	getColumnDataTransformer()	Getter for ColumnDataTransformer
String	getColumnName()	Getter for column name
DATA_TYPE	getConstantValue()	Returns the defined constant value in case of a constant column value.
boolean	isColumnInSchema()	Returns true if the column is present in schema associated with the csv import
boolean	isColumnInSource()	Returns true if the column is present in the source csv file.
boolean	isConstantColumn()	Returns true if the ColumnDataTransformer has the hasConstant attribute return true
boolean	isPartitionCol()	Returns true if the column is defined as a partition column in the schema
void	onPartitionParserUpdate(PartitionParserUpdate parserUpdate)	Updates are published by the source (CsvPartitionColumnParser or AppendableTableWrapper) by invoking this method.
void	publishRowUpdate(int size, long end)	Publish row updates to interested sinks.
void	read(TARRAY dest, boolean[] isNull, long srcBegin, long srcEnd)
void	registerRowUpdateObserver(RowUpdateObserver rowUpdateObserver)	Used to set the RowUpdateObserver for one of the column sinks that are in csv source file.
boolean	supportsTransformations()	Returns true if the ColumnDataTransformer supports transformation
String	toString()
protected void	updateLocalAppendableColumn(TARRAY values, long destEnd, boolean isSingleValue)	Blocks until the partition column is parsed for the same chunk.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface com.illumon.iris.importers.csv.sink.AppendableColumnSink

add, addBooleans, addBytes, addChars, addDoubles, addFloats, addInts, addLongs, addShorts, isColumnOnlyInSchema

Methods inherited from interface com.illumon.iris.importers.csv.sink.AppendableSink

getUnderlying, nullFlagsToValues, updateRowChunk, write, writeToLocal

Field Details

ZERO_LENGTH_WRAPPER_ARRAY

public static final BaseAppendableColumnSink[] ZERO_LENGTH_WRAPPER_ARRAY

DEFAULT_PARTITION_COL

public static final String DEFAULT_PARTITION_COL

See Also:

Constant Field Values

columnDataTransformer

protected final ImportColumnDataTransformer columnDataTransformer

constantValue

protected final DATA_TYPE constantValue

columnName

protected final String columnName

isPartitionCol

protected final boolean isPartitionCol

isColumnInSchema

protected final boolean isColumnInSchema

isColumnInSource

protected final boolean isColumnInSource

schemaHasPartitionCol

protected final boolean schemaHasPartitionCol

isFromSplitFile

protected final boolean isFromSplitFile

columnMap

protected final Map<String,LocalAppendableColumn<DATA_TYPE>> columnMap

evictedPartitions

protected final Set<String> evictedPartitions

updateObserver

protected RowUpdateObserver updateObserver

Constructor Details

BaseAppendableColumnSink

protected BaseAppendableColumnSink(@NotNull com.fishlib.io.logger.Logger log, @NotNull String columnName, @Nullable ImporterColumnDefinition icdColDef, @Nullable ImportColumnDataTransformer columnDataTransformer, boolean isPartitionCol, boolean isColumnInSchema, boolean isColumnInSource, boolean schemaHasPartitionCol, boolean fromSplitFile)

Method Details

getConstantValue

@Nullable public DATA_TYPE getConstantValue()

Description copied from interface: AppendableSink

Returns the defined constant value in case of a constant column value. Constant value definition is present in the importColumn xml element. The value will be accessible through ImportDataTransformer

Specified by:

getConstantValue in interface AppendableSink<DATA_TYPE,TARRAY>

getColumnName

public String getColumnName()

Description copied from interface: AppendableColumnSink

Getter for column name

Specified by:

getColumnName in interface AppendableColumnSink<DATA_TYPE,TARRAY>

isColumnInSchema

public boolean isColumnInSchema()

Description copied from interface: AppendableColumnSink

Returns true if the column is present in schema associated with the csv import

Specified by:

isColumnInSchema in interface AppendableColumnSink<DATA_TYPE,TARRAY>

isColumnInSource

public boolean isColumnInSource()

Description copied from interface: AppendableColumnSink

Returns true if the column is present in the source csv file. This is determined by whether the column name exists in the columnNamesInFile attribute of CsvImportHelper

Specified by:

isColumnInSource in interface AppendableColumnSink<DATA_TYPE,TARRAY>

isPartitionCol

public boolean isPartitionCol()

Description copied from interface: AppendableColumnSink

Returns true if the column is defined as a partition column in the schema

Specified by:

isPartitionCol in interface AppendableColumnSink<DATA_TYPE,TARRAY>

supportsTransformations

public boolean supportsTransformations()

Description copied from interface: AppendableColumnSink

Returns true if the ColumnDataTransformer supports transformation

Specified by:

supportsTransformations in interface AppendableColumnSink<DATA_TYPE,TARRAY>

getColumnDataTransformer

@NotNull public ImportColumnDataTransformer getColumnDataTransformer()

Description copied from interface: AppendableColumnSink

Getter for ColumnDataTransformer

Specified by:

getColumnDataTransformer in interface AppendableColumnSink<DATA_TYPE,TARRAY>

isConstantColumn

public boolean isConstantColumn()

Description copied from interface: AppendableColumnSink

Returns true if the ColumnDataTransformer has the hasConstant attribute return true

Specified by:

isConstantColumn in interface AppendableColumnSink<DATA_TYPE,TARRAY>

toString

public String toString()

Overrides:

toString in class Object

registerRowUpdateObserver

public void registerRowUpdateObserver(@NotNull RowUpdateObserver rowUpdateObserver)

Used to set the RowUpdateObserver for one of the column sinks that are in csv source file.

Parameters:

rowUpdateObserver - A RowUpdateObserver to which the row chunk attribute (size and destEnd ) can be published

publishRowUpdate

public void publishRowUpdate(int size, long end)

Publish row updates to interested sinks.

Specified by:

publishRowUpdate in interface AppendableColumnSink<DATA_TYPE,TARRAY>

addAppendableColumn

public void addAppendableColumn(@Nullable String partition, @Nullable LocalAppendableColumn<DATA_TYPE> appendableColumn)

Description copied from interface: AppendableColumnSink

The method provides the partition and its associated LocalAppendableColumn This information should be cached in the column sink implementation and should be used when updating the column values based on PartitionParserUpdate that will be pushed to the column when the table has a partitioning column and the values of the column should be persisted in the appropriate partition.

Specified by:

addAppendableColumn in interface AppendableColumnSink<DATA_TYPE,TARRAY>

Parameters:

partition - Partition value as a string

appendableColumn - The LocalAppendableColumn of the column associated for this partition

evict

public void evict(@Nullable String partition)

Description copied from interface: AppendableColumnSink

The provided partition should be evicted from the current partition column cache. This should be registered as one of the evicted partitions.

Specified by:

evict in interface AppendableColumnSink<DATA_TYPE,TARRAY>

Parameters:

partition - Partition value as a string, that should be evicted from the local column cache of partitions

addToAppendableColumn

protected abstract void addToAppendableColumn(@NotNull LocalAppendableColumn<DATA_TYPE> appendableColumn, @NotNull TARRAY values, int chunkStartIndex, int chunkSize, boolean isSingleValue)

Writes to disk the contents of values array from chunkStartIndex to chunkSize in the appendable column of a particular partition

Parameters:

appendableColumn - The LocalAppendableColumn of the current partition

values - The array whose contents need to be written to disk for this column and partition

chunkStartIndex - The start index in the values array

chunkSize - The length of data from the start index that need to be written to disk

isSingleValue - This indicates that entire chunk needs to be updated using a single value which is the constant value

updateLocalAppendableColumn

protected void updateLocalAppendableColumn(@NotNull TARRAY values, long destEnd, boolean isSingleValue)

Blocks until the partition column is parsed for the same chunk.

onPartitionParserUpdate

public void onPartitionParserUpdate(@NotNull PartitionParserUpdate parserUpdate)

Description copied from interface: PartitionUpdatesObserver

Updates are published by the source (CsvPartitionColumnParser or AppendableTableWrapper) by invoking this method.

Specified by:

onPartitionParserUpdate in interface PartitionUpdatesObserver

Parameters:

parserUpdate - Partition for current chunk.

read

public void read(TARRAY dest, boolean[] isNull, long srcBegin, long srcEnd)

Specified by:

read in interface io.deephaven.csv.sinks.Source<DATA_TYPE>

append

public com.fishlib.base.log.LogOutput append(@NotNull com.fishlib.base.log.LogOutput logOutput)

Specified by:

append in interface com.fishlib.base.log.LogOutputAppendable