Package com.illumon.iris.db.v2.parquet

Class ParquetTableWriter

java.lang.Object

com.illumon.iris.db.v2.parquet.ParquetTableWriter

public class ParquetTableWriter
extends Object

API for writing DH tables in parquet format

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	ParquetTableWriter.DestinationInfo	Information about a writing destination (e.g.

Field Summary

Fields

Modifier and Type	Field	Description
static String	BEGIN_POS
static String	END_POS
static String	GROUPING_KEY
static String	METADATA_KEY
static String	PARQUET_FILE_EXTENSION

Constructor Summary

Constructors

Constructor	Description
ParquetTableWriter()

Method Summary

Modifier and Type	Method	Description
static Function<String,String>	defaultGroupingFileName(String path)
static void	write(Table table, TableDefinition definition, ParquetInstructions writeInstructions, String path, Map<String,String> tableMeta, TableInfo.Builder tableInfoBuilder)	Writes a table in parquet format under a given path
static void	write(Table t, TableDefinition definition, ParquetInstructions writeInstructions, String path, Map<String,String> incomingMeta, String... groupingColumns)	Writes a table in parquet format under a given path
static void	write(Table t, TableDefinition definition, ParquetInstructions writeInstructions, String destPathName, Map<String,String> incomingMeta, Function<String,String> groupingPathFactory, String... groupingColumns)	Writes a table in parquet format under a given path
static void	write(Table t, String path, Map<String,String> incomingMeta, String... groupingColumns)	Writes a table in parquet format under a given path
static void	write(Table t, String path, Map<String,String> incomingMeta, Function<String,String> groupingPathFactory, String... groupingColumns)	Writes a table in parquet format under a given path
static void	write(ParquetTableWriter.DestinationInfo[] destinationInfos, ParquetInstructions instructions, TableDefinition definition, ExecutorService executorService, int maxConcurrentColumns)	Writes a table in parquet format under a given path

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

METADATA_KEY

public static final String METADATA_KEY

See Also:

Constant Field Values

BEGIN_POS

public static final String BEGIN_POS

See Also:

Constant Field Values

END_POS

public static final String END_POS

See Also:

Constant Field Values

GROUPING_KEY

public static final String GROUPING_KEY

See Also:

Constant Field Values

PARQUET_FILE_EXTENSION

public static final String PARQUET_FILE_EXTENSION

See Also:

Constant Field Values

Constructor Details

ParquetTableWriter

public ParquetTableWriter()

Method Details

defaultGroupingFileName

public static Function<String,String> defaultGroupingFileName(@NotNull String path)

write

public static void write(@NotNull ParquetTableWriter.DestinationInfo[] destinationInfos, @NotNull ParquetInstructions instructions, @NotNull TableDefinition definition, @NotNull ExecutorService executorService, int maxConcurrentColumns) throws SchemaMappingException, IOException

Writes a table in parquet format under a given path

Parameters:

destinationInfos - Destination information coupling input data, output path, and grouping metadata for each desired result table

instructions - The ParquetInstructions containing write directives.

definition - The definition to use for the output tables

Throws:

SchemaMappingException - Error creating a parquet table schema for the given table (likely due to unsupported types)

IOException - For file writing related errors

write

public static void write(@NotNull Table t, @NotNull String path, @NotNull Map<String,String> incomingMeta, @NotNull Function<String,String> groupingPathFactory, String... groupingColumns) throws SchemaMappingException, IOException

Writes a table in parquet format under a given path

Parameters:

t - The table to write

path - The destination path

incomingMeta - A map of metadata values to be stores in the file footer

groupingPathFactory - a factory to construct paths for grouping tables.

groupingColumns - List of columns the tables are grouped by (the write operation will store the grouping info)

Throws:

SchemaMappingException - Error creating a parquet table schema for the given table (likely due to unsupported types)

IOException - For file writing related errors

write

public static void write(@NotNull Table t, @NotNull String path, @NotNull Map<String,String> incomingMeta, String... groupingColumns) throws SchemaMappingException, IOException

Writes a table in parquet format under a given path

Parameters:

t - the table to write

path - the destination path

incomingMeta - any metadata to include in the parquet metadata

groupingColumns - the grouping columns (if any)

Throws:

SchemaMappingException

IOException

write

public static void write(@NotNull Table t, @NotNull TableDefinition definition, @NotNull ParquetInstructions writeInstructions, @NotNull String destPathName, @NotNull Map<String,String> incomingMeta, @NotNull Function<String,String> groupingPathFactory, String... groupingColumns) throws SchemaMappingException, IOException

Writes a table in parquet format under a given path

Parameters:

t - The table to write

definition - Table definition

writeInstructions - Write instructions for customizations while writing

destPathName - The destination path

incomingMeta - A map of metadata values to be stores in the file footer

groupingPathFactory - a factory for constructing paths for grouping tables

groupingColumns - List of columns the tables are grouped by (the write operation will store the grouping info)

Throws:

SchemaMappingException - Error creating a parquet table schema for the given table (likely due to unsupported types)

IOException - For file writing related errors

write

public static void write(@NotNull Table t, @NotNull TableDefinition definition, @NotNull ParquetInstructions writeInstructions, @NotNull String path, @NotNull Map<String,String> incomingMeta, String... groupingColumns) throws SchemaMappingException, IOException

Writes a table in parquet format under a given path

Parameters:

t - the table to write

definition - the writable table definition

writeInstructions - the parquet instructions for writing

path - the path to write to

incomingMeta - any metadata to include ih the parquet metadata

groupingColumns - the grouping columns (if any)

Throws:

SchemaMappingException

IOException

write

public static void write(@NotNull Table table, @NotNull TableDefinition definition, @NotNull ParquetInstructions writeInstructions, @NotNull String path, @NotNull Map<String,String> tableMeta, @NotNull TableInfo.Builder tableInfoBuilder) throws SchemaMappingException, IOException

Writes a table in parquet format under a given path

Parameters:

table - The table to write

definition - The table definition

writeInstructions - Write instructions for customizations while writing

path - The destination path

tableMeta - A map of metadata values to be stores in the file footer

tableInfoBuilder - A partially-constructed builder for the metadata object

Throws:

SchemaMappingException - Error creating a parquet table schema for the given table (likely due to unsupported types)

IOException - For file writing related errors