Package com.illumon.iris.importers.merge

Class ParquetFormatMergeDestination

java.lang.Object

com.illumon.iris.importers.merge.MergeDestination

com.illumon.iris.importers.merge.ParquetFormatMergeDestination

All Implemented Interfaces:

com.fishlib.base.log.LogOutputAppendable

public class ParquetFormatMergeDestination
extends MergeDestination

Field Summary

Fields inherited from class com.illumon.iris.importers.merge.MergeDestination

destinationOperationName, groupingColumns, indexer, inputSubTable, inputSubTableSize, internalPartitionName, logPrefix, MERGE_TEMP_PREFIX, orderedSubTables, orderingPairs, permanentDirectory, temporaryDirectory

Constructor Summary

Constructors

Constructor	Description
ParquetFormatMergeDestination(File writableInternalPartition, String partitioningColumnValue, String tableName, String operationName, String parquetCodecName)	For internal Deephaven use only.

Method Summary

Modifier and Type	Method	Description
void	computeSortedOrder(com.fishlib.io.logger.Logger log, String sortColumnFormula, com.illumon.dataobjects.ColumnDefinition<?> sortColumnDefinition, List<String[]> indexDataSets)
static MergeDestinationFactory	factory()
void	finish(com.fishlib.io.logger.Logger log)
void	initialize(com.fishlib.io.logger.Logger log, TableDefinition writableDefinition, SyncOperator syncOperator)
static Stream<String>	supportedParquetCodecsStream(com.fishlib.configuration.Configuration configuration)	Determine the supported parquet codecs.
void	writeDataIndex(String[] keyColumns, Table computedDataIndex)
void	writeGrouping(com.fishlib.io.logger.Logger log, com.fishlib.util.process.FatalErrorReporter fatalErrorReporter, TableDefinition definition)
void	writeMergedDataLowHeap(com.fishlib.io.logger.Logger log, com.fishlib.util.process.FatalErrorReporter fatalErrorReporter, ExecutorService executorService, Table inputData, String[] writableColumnNames, StatusCallback destinationCallback, TableDefinition tableDefinition)

Methods inherited from class com.illumon.iris.importers.merge.MergeDestination

append, cleanup, computeDataIndexes, computeGroupOrder, computeSortedOrder, getDestinationOperationName, getInternalPartitionName, getLogPrefix, getPermanentDirectory, getTemporaryDirectory, groupInputSubTable, install, permanentDirectoryExists, setInputSubTable, toString

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Details

ParquetFormatMergeDestination

public ParquetFormatMergeDestination(@NotNull File writableInternalPartition, @NotNull String partitioningColumnValue, @NotNull String tableName, @NotNull String operationName, @Nullable String parquetCodecName)

For internal Deephaven use only.

Parameters:

writableInternalPartition -

partitioningColumnValue -

tableName -

operationName -

Method Details

factory

public static MergeDestinationFactory factory()

initialize

public void initialize(@NotNull com.fishlib.io.logger.Logger log, @NotNull TableDefinition writableDefinition, SyncOperator syncOperator)

Specified by:

initialize in class MergeDestination

writeGrouping

public void writeGrouping(@NotNull com.fishlib.io.logger.Logger log, @NotNull com.fishlib.util.process.FatalErrorReporter fatalErrorReporter, @NotNull TableDefinition definition)

Specified by:

writeGrouping in class MergeDestination

computeSortedOrder

public void computeSortedOrder(@NotNull com.fishlib.io.logger.Logger log, @Nullable String sortColumnFormula, @Nullable com.illumon.dataobjects.ColumnDefinition<?> sortColumnDefinition, @Nullable List<String[]> indexDataSets)

Specified by:

computeSortedOrder in class MergeDestination

writeMergedDataLowHeap

public void writeMergedDataLowHeap(com.fishlib.io.logger.Logger log, com.fishlib.util.process.FatalErrorReporter fatalErrorReporter, ExecutorService executorService, Table inputData, String[] writableColumnNames, StatusCallback destinationCallback, TableDefinition tableDefinition)

Specified by:

writeMergedDataLowHeap in class MergeDestination

finish

public void finish(@NotNull com.fishlib.io.logger.Logger log)

Specified by:

finish in class MergeDestination

writeDataIndex

public void writeDataIndex(@NotNull String[] keyColumns, @NotNull Table computedDataIndex)

Specified by:

writeDataIndex in class MergeDestination

supportedParquetCodecsStream

public static Stream<String> supportedParquetCodecsStream(com.fishlib.configuration.Configuration configuration)

Determine the supported parquet codecs. Some codecs require additional jar files which aren't part of the base Deephaven installation, so by default the available codecs are limited through the property SUPPORTED_PARQUET_CODECS_PROPERTY.

Parameters:

configuration - the Configuration instance

Returns:

a Stream of available codecs