Package io.deephaven.parquet.table

Class ParquetInstructions

java.lang.Object

io.deephaven.parquet.table.ParquetInstructions

All Implemented Interfaces:

ColumnToCodecMappings

public abstract class ParquetInstructions
extends Object
implements ColumnToCodecMappings

This class provides instructions intended for read and write parquet operations (which take it as an optional argument) specifying desired transformations. Examples are mapping column names and use of specific codecs during (de)serialization.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	ParquetInstructions.Builder
static interface	ParquetInstructions.OnWriteCompleted
static enum	ParquetInstructions.ParquetFileLayout
static enum	ParquetInstructions.UnsignedLongTarget	The Deephaven type to read a UINT_64 parquet column as; such values have no exact Java primitive.

Field Summary

Fields

Modifier and Type	Field	Description
static final String	DEFAULT_COMPRESSION_CODEC_NAME
static final int	DEFAULT_MAXIMUM_DICTIONARY_KEYS
static final int	DEFAULT_MAXIMUM_DICTIONARY_SIZE
static final int	DEFAULT_TARGET_PAGE_SIZE
static final ParquetInstructions	EMPTY
static final int	MIN_TARGET_PAGE_SIZE

Method Summary

Modifier and Type	Method	Description
abstract String	baseNameForPartitionedParquetData()
static ParquetInstructions.Builder	builder()
abstract boolean	generateMetadataFiles()
abstract String	getCodecArgs(String columnName)
abstract String	getCodecName(String columnName)
abstract String	getColumnNameFromParquetColumnName(String parquetColumnName)
final String	getColumnNameFromParquetColumnNameOrDefault(String parquetColumnName)
abstract Optional<ParquetColumnResolver.Factory>	getColumnResolverFactory()
abstract String	getCompressionCodecName()
abstract OptionalInt	getFieldId(String columnName)	The field ID for the given columnName.
abstract Optional<ParquetInstructions.ParquetFileLayout>	getFileLayout()
abstract Optional<Collection<List<String>>>	getIndexColumns()
abstract int	getMaximumDictionaryKeys()
abstract int	getMaximumDictionarySize()
abstract String	getParquetColumnNameFromColumnNameOrDefault(String columnName)
abstract RowGroupInfo	getRowGroupInfo()
abstract Optional<SeekableChannelsProvider>	getSeekableChannelsProviderForWriting()
abstract Object	getSpecialInstructions()
abstract Optional<TableDefinition>	getTableDefinition()
abstract int	getTargetPageSize()
abstract Optional<ParquetInstructions.UnsignedLongTarget>	getUnsignedLongTarget(String columnName)	The ParquetInstructions.UnsignedLongTarget requested for the given columnName.
abstract boolean	isLegacyParquet()
abstract boolean	isRefreshing()
abstract Optional<ParquetInstructions.OnWriteCompleted>	onWriteCompleted()
static boolean	sameColumnNamesAndCodecMappings(ParquetInstructions i1, ParquetInstructions i2)
abstract boolean	useDictionary(String columnName)
abstract ParquetInstructions	withLayout(ParquetInstructions.ParquetFileLayout fileLayout)	Creates a new ParquetInstructions object with the same properties as the current object but layout set as the provided ParquetInstructions.ParquetFileLayout.
abstract ParquetInstructions	withTableDefinition(TableDefinition tableDefinition)	Creates a new ParquetInstructions object with the same properties as the current object but definition set as the provided TableDefinition.
abstract ParquetInstructions	withTableDefinitionAndLayout(TableDefinition tableDefinition, ParquetInstructions.ParquetFileLayout fileLayout)	Creates a new ParquetInstructions object with the same properties as the current object but definition and layout set as the provided values.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

DEFAULT_COMPRESSION_CODEC_NAME

public static final String DEFAULT_COMPRESSION_CODEC_NAME

DEFAULT_MAXIMUM_DICTIONARY_KEYS

public static final int DEFAULT_MAXIMUM_DICTIONARY_KEYS

See Also:

• Constant Field Values

DEFAULT_MAXIMUM_DICTIONARY_SIZE

public static final int DEFAULT_MAXIMUM_DICTIONARY_SIZE

See Also:

• Constant Field Values

MIN_TARGET_PAGE_SIZE

public static final int MIN_TARGET_PAGE_SIZE

DEFAULT_TARGET_PAGE_SIZE

public static final int DEFAULT_TARGET_PAGE_SIZE

EMPTY

public static final ParquetInstructions EMPTY

Method Details

getColumnNameFromParquetColumnNameOrDefault

public final String getColumnNameFromParquetColumnNameOrDefault(String parquetColumnName)

getParquetColumnNameFromColumnNameOrDefault

public abstract String getParquetColumnNameFromColumnNameOrDefault(String columnName)

getColumnNameFromParquetColumnName

public abstract String getColumnNameFromParquetColumnName(String parquetColumnName)

getCodecName

public abstract String getCodecName(String columnName)

Specified by:

getCodecName in interface ColumnToCodecMappings

getCodecArgs

public abstract String getCodecArgs(String columnName)

Specified by:

getCodecArgs in interface ColumnToCodecMappings

useDictionary

public abstract boolean useDictionary(String columnName)

Returns:

A hint that the writer should use dictionary-based encoding for writing this column; never evaluated for non-String columns, defaults to false

getFieldId

public abstract OptionalInt getFieldId(String columnName)

The field ID for the given columnName.

Parameters:

columnName - the Deephaven column name

Returns:

the field id

getUnsignedLongTarget

public abstract Optional<ParquetInstructions.UnsignedLongTarget> getUnsignedLongTarget(String columnName)

The ParquetInstructions.UnsignedLongTarget requested for the given columnName. Only consulted while inferring a TableDefinition; a supplied definition takes precedence, and one that disagrees is rejected at construction.

Parameters:

columnName - the Deephaven column name

Returns:

the requested target, or Optional.empty() if none was requested

getSpecialInstructions

public abstract Object getSpecialInstructions()

getCompressionCodecName

public abstract String getCompressionCodecName()

getMaximumDictionaryKeys

public abstract int getMaximumDictionaryKeys()

Returns:

The maximum number of unique keys the writer should add to a dictionary page before switching to non-dictionary encoding; never evaluated for non-String columns, ignored if useDictionary(String)

getMaximumDictionarySize

public abstract int getMaximumDictionarySize()

Returns:

The maximum number of bytes the writer should add to a dictionary before switching to non-dictionary encoding; never evaluated for non-String columns, ignored if useDictionary(String)

isLegacyParquet

public abstract boolean isLegacyParquet()

getTargetPageSize

public abstract int getTargetPageSize()

isRefreshing

public abstract boolean isRefreshing()

Returns:

if the data source is refreshing

generateMetadataFiles

public abstract boolean generateMetadataFiles()

Returns:

should we generate "_metadata" and "_common_metadata" files while writing parquet files?

getFileLayout

public abstract Optional<ParquetInstructions.ParquetFileLayout> getFileLayout()

getTableDefinition

public abstract Optional<TableDefinition> getTableDefinition()

getIndexColumns

public abstract Optional<Collection<List<String>>> getIndexColumns()

getRowGroupInfo

public abstract RowGroupInfo getRowGroupInfo()

getColumnResolverFactory

public abstract Optional<ParquetColumnResolver.Factory> getColumnResolverFactory()

withTableDefinition

public abstract ParquetInstructions withTableDefinition(TableDefinition tableDefinition)

Creates a new ParquetInstructions object with the same properties as the current object but definition set as the provided TableDefinition.

withLayout

public abstract ParquetInstructions withLayout(ParquetInstructions.ParquetFileLayout fileLayout)

Creates a new ParquetInstructions object with the same properties as the current object but layout set as the provided ParquetInstructions.ParquetFileLayout.

withTableDefinitionAndLayout

public abstract ParquetInstructions withTableDefinitionAndLayout(TableDefinition tableDefinition,
 ParquetInstructions.ParquetFileLayout fileLayout)

Creates a new ParquetInstructions object with the same properties as the current object but definition and layout set as the provided values.

baseNameForPartitionedParquetData

public abstract String baseNameForPartitionedParquetData()

Returns:

the base name for partitioned parquet data. Check setBaseNameForPartitionedParquetData for more details about different tokens that can be used in the base name.

onWriteCompleted

public abstract Optional<ParquetInstructions.OnWriteCompleted> onWriteCompleted()

Returns:

A callback to be executed when on completing each parquet data file write (excluding the index and metadata files). This callback gets invoked by the writing thread in a linear fashion.

getSeekableChannelsProviderForWriting

public abstract Optional<SeekableChannelsProvider> getSeekableChannelsProviderForWriting()

sameColumnNamesAndCodecMappings

@VisibleForTesting
public static boolean sameColumnNamesAndCodecMappings(ParquetInstructions i1,
 ParquetInstructions i2)

builder

public static ParquetInstructions.Builder builder()