Package com.illumon.iris.importers.merge

Class ParquetFormatMergeDestination

java.lang.Object

com.illumon.iris.importers.merge.MergeDestination

com.illumon.iris.importers.merge.ParquetFormatMergeDestination

All Implemented Interfaces:

com.fishlib.base.log.LogOutputAppendable

public class ParquetFormatMergeDestination
extends MergeDestination

Field Summary

Fields inherited from class com.illumon.iris.importers.merge.MergeDestination

destinationOperationName, groupingColumns, hadGroupings, indexer, inputSubTable, inputSubTableSize, internalPartitionName, logPrefix, MERGE_TEMP_PREFIX, orderedSubTables, orderingPairs, permanentDirectory, temporaryDirectory

Constructor Summary

Constructors

Constructor	Description
ParquetFormatMergeDestination(File writableInternalPartition, String partitioningColumnValue, String tableName, String operationName, String parquetCodecName)	For internal Deephaven use only.

Method Summary

Modifier and Type	Method	Description
void	computeSortedOrder(com.fishlib.io.logger.Logger log, String sortColumnFormula, com.illumon.dataobjects.ColumnDefinition<?> sortColumnDefinition, List<String[]> indexDataSets)
static MergeDestinationFactory	factory()
void	finish(com.fishlib.io.logger.Logger log)
void	initialize(com.fishlib.io.logger.Logger log, TableDefinition writableDefinition, SyncOperator syncOperator)
static Stream<String>	supportedParquetCodecsStream(com.fishlib.configuration.Configuration configuration)	Determine the supported parquet codecs.
void	writeDataIndex(String[] keyColumns, Table computedDataIndex)
void	writeGrouping(com.fishlib.io.logger.Logger log, com.fishlib.util.process.FatalErrorReporter fatalErrorReporter, TableDefinition definition)
void	writeMergedDataLowHeap(com.fishlib.io.logger.Logger log, com.fishlib.util.process.FatalErrorReporter fatalErrorReporter, ExecutorService executorService, Table inputData, String[] writableColumnNames, StatusCallback destinationCallback, TableDefinition tableDefinition)

Methods inherited from class com.illumon.iris.importers.merge.MergeDestination

append, cleanup, computeDataIndexes, computeGroupOrder, computeSortedOrder, getDestinationOperationName, getInternalPartitionName, getLogPrefix, getPermanentDirectory, getTemporaryDirectory, groupInputSubTable, install, permanentDirectoryExists, setInputSubTable, toString

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Details

ParquetFormatMergeDestination

public ParquetFormatMergeDestination(@NotNull
 File writableInternalPartition,
 @NotNull
 String partitioningColumnValue,
 @NotNull
 String tableName,
 @NotNull
 String operationName,
 @Nullable
 String parquetCodecName)

For internal Deephaven use only.

Parameters:

writableInternalPartition -

partitioningColumnValue -

tableName -

operationName -

Method Details

factory

public static MergeDestinationFactory factory()

initialize

public void initialize(@NotNull
 com.fishlib.io.logger.Logger log,
 @NotNull
 TableDefinition writableDefinition,
 SyncOperator syncOperator)

Specified by:

initialize in class MergeDestination

writeGrouping

public void writeGrouping(@NotNull
 com.fishlib.io.logger.Logger log,
 @NotNull
 com.fishlib.util.process.FatalErrorReporter fatalErrorReporter,
 @NotNull
 TableDefinition definition)

Specified by:

writeGrouping in class MergeDestination

computeSortedOrder

public void computeSortedOrder(@NotNull
 com.fishlib.io.logger.Logger log,
 @Nullable
 String sortColumnFormula,
 @Nullable
 com.illumon.dataobjects.ColumnDefinition<?> sortColumnDefinition,
 @Nullable
 List<String[]> indexDataSets)

Specified by:

computeSortedOrder in class MergeDestination

writeMergedDataLowHeap

public void writeMergedDataLowHeap(com.fishlib.io.logger.Logger log,
 com.fishlib.util.process.FatalErrorReporter fatalErrorReporter,
 ExecutorService executorService,
 Table inputData,
 String[] writableColumnNames,
 StatusCallback destinationCallback,
 TableDefinition tableDefinition)

Specified by:

writeMergedDataLowHeap in class MergeDestination

finish

public void finish(@NotNull
 com.fishlib.io.logger.Logger log)

Specified by:

finish in class MergeDestination

writeDataIndex

public void writeDataIndex(@NotNull
 String[] keyColumns,
 @NotNull
 Table computedDataIndex)

Specified by:

writeDataIndex in class MergeDestination

supportedParquetCodecsStream

public static Stream<String> supportedParquetCodecsStream(com.fishlib.configuration.Configuration configuration)

Determine the supported parquet codecs. Some codecs require additional jar files which aren't part of the base Deephaven installation, so by default the available codecs are limited through the property SUPPORTED_PARQUET_CODECS_PROPERTY.

Parameters:

configuration - the Configuration instance

Returns:

a Stream of available codecs