Package com.illumon.iris.db.v2.parquet

Class ParquetTableWriter

java.lang.Object
com.illumon.iris.db.v2.parquet.ParquetTableWriter
public class ParquetTableWriter extends Object
API for writing DH tables in parquet format

  • Field Details

  • Constructor Details

    • ParquetTableWriter

      public ParquetTableWriter()

  • Method Details

    • write

      public static void write(@NotNull @NotNull Table table, @NotNull @NotNull TableDefinition definition, @NotNull @NotNull ParquetInstructions writeInstructions, @NotNull @NotNull String path, @NotNull @NotNull Map<String,String> tableMeta) throws SchemaMappingException, IOException
      Writes a table in parquet format under a given path
      Parameters:
      table - The table to write
      definition - The table definition
      writeInstructions - Write instructions for customizations while writing
      path - The destination path
      tableMeta - A map of metadata values to be stores in the file footer
      Throws:
      SchemaMappingException - Error creating a parquet table schema for the given table (likely due to unsupported types)
      IOException - For file writing related errors

    • write

      public static void write(@NotNull @NotNull Table table, @NotNull @NotNull TableDefinition definition, @NotNull @NotNull ParquetInstructions writeInstructions, @NotNull @NotNull String path, @NotNull @NotNull Map<String,String> tableMeta, @NotNull TableInfo.Builder tableInfoBuilder) throws SchemaMappingException, IOException
      Writes a table in parquet format under a given path
      Parameters:
      table - The table to write
      definition - The table definition
      writeInstructions - Write instructions for customizations while writing
      path - The destination path
      tableMeta - A map of metadata values to be stores in the file footer
      tableInfoBuilder - A partially-constructed builder for the metadata object
      Throws:
      SchemaMappingException - Error creating a parquet table schema for the given table (likely due to unsupported types)
      IOException - For file writing related errors

    • write

      @VisibleForTesting protected static void write(@NotNull @NotNull Table table, @NotNull @NotNull TableDefinition definition, @NotNull @NotNull ParquetInstructions writeInstructions, @NotNull @NotNull ParquetFileWriter parquetFileWriter, @NotNull @NotNull Map<String,Map<ParquetCacheTags,Object>> computedCache) throws IOException
      Writes a table in parquet format under a given path. This method should only be invoked when wrapped in a try-with-resources using a new LivenessScope to ensure that the various derived tables created are properly cleaned up.
      Parameters:
      table - The table to write
      definition - The table definition
      writeInstructions - Write instructions for customizations while writing
      parquetFileWriter - the writer
      Throws:
      IOException - For file writing related errors

    • pretransformTable

      @NotNull @InternalUseOnly public static @NotNull Table pretransformTable(@NotNull @NotNull Table table, @NotNull @NotNull TableDefinition definition)
      Detect any missing or StringSet columns and convert them to arrays / null values as appropriate to prepare the input table to be written to the parquet file.
      Parameters:
      table - the input table
      definition - the table definition being written
      Returns:
      a transformed view of the input table.

    • getParquetFileWriter

      @NotNull @InternalUseOnly public static @NotNull ParquetFileWriter getParquetFileWriter(@NotNull @NotNull Map<String,Map<ParquetCacheTags,Object>> computedCache, @NotNull @NotNull TableDefinition definition, @NotNull @NotNull ReadOnlyIndex tableRowSet, @NotNull @NotNull Map<String,? extends ColumnSource> columnSourceMap, @NotNull @NotNull String path, @NotNull @NotNull ParquetInstructions writeInstructions, @NotNull @NotNull Map<String,String> tableMeta, @NotNull TableInfo.Builder tableInfoBuilder) throws IOException
      Create a ParquetFileWriter for writing the table to disk.
      Parameters:
      computedCache - Per column cache tags
      definition - the writable definition
      tableRowSet - the index being written
      columnSourceMap - the columns of the table
      path - the destination to write to
      writeInstructions - write instructions for the file
      tableMeta - metadata to include in the parquet metadata
      tableInfoBuilder - a builder for accumulating per-column information to construct the deephaven metadata
      Returns:
      a new file writer
      Throws:
      IOException

    • writeColumnSource

      @InternalUseOnly public static <DATA_TYPE> void writeColumnSource(@NotNull @NotNull Map<String,Map<ParquetCacheTags,Object>> computedCache, @NotNull @NotNull Index tableRowSet, @NotNull @NotNull RowGroupWriter rowGroupWriter, @NotNull @NotNull String name, @NotNull @NotNull ColumnSource<DATA_TYPE> columnSourceIn, @NotNull @NotNull ColumnDefinition<DATA_TYPE> columnDefinition, @NotNull @NotNull ParquetInstructions writeInstructions) throws IllegalAccessException, IOException
      Throws:
      IllegalAccessException
      IOException