Package io.deephaven.enterprise.derivedtablewriter

Class DerivedTableWriter

java.lang.Object

io.deephaven.enterprise.derivedtablewriter.DerivedTableWriter

public class DerivedTableWriter
extends Object

Persist data from a Table to Deephaven storage using an in-worker DataImportServer.

This utility is intended to read one live table from storage, perform query operations, and then write a derived live table to storage.

The source table must be either a blink table or an add-only table. Each new row is persisted by the DIS.

To ensure exactly-once delivery, your input table must have a column of ascending sequence numbers specified using the DerivedTableWriter.Options.sequenceColumn. If you do not have a sequence column, then upon restart rows may be logged twice. If your source table is ephemeral, then even with a sequence number rows may be dropped on restart. As an alternative, you may also read the derived table and extract a sequence number, timestamp, or other information that can be used to resume your source appropriately before starting the DerivedTableWriter.

When processing independent partitions, using the ingestPartitionedTable(String, String, DataImportServer, PartitionedTable, PartitionedOptions, Options) is recommended. Each constituent is independently logged as an output partition.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	DerivedTableWriter.Options	Represents Options for ingesting a table to a DIS partition.
static class	DerivedTableWriter.PartitionedOptions	Represents options for partitioned table ingestion.

Constructor Summary

Constructors

Constructor	Description
DerivedTableWriter()

Method Summary

Modifier and Type	Method	Description
static LivenessArtifact	ingestPartitionedTable(@NotNull String namespace, @NotNull String tableName, @NotNull io.deephaven.shadow.enterprise.com.illumon.iris.db.tables.dataimport.logtailer.DataImportServer dis, @NotNull PartitionedTable sourcePartitionedTable, @NotNull DerivedTableWriter.PartitionedOptions partitionedOptions, @NotNull DerivedTableWriter.Options options)	Ingest a PartitionedTable to an in-worker DIS.
static LivenessArtifact	ingestTable(@NotNull String namespace, @NotNull String tableName, @NotNull String columnPartition, @NotNull String internalPartition, @NotNull io.deephaven.shadow.enterprise.com.illumon.iris.db.tables.dataimport.logtailer.DataImportServer dis, @NotNull Table sourceTable, @NotNull DerivedTableWriter.Options options)	Ingest a Table to an in-worker DIS.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

DerivedTableWriter

public DerivedTableWriter()

Method Details

ingestTable

public static LivenessArtifact ingestTable(@NotNull
 @NotNull String namespace,
 @NotNull
 @NotNull String tableName,
 @NotNull
 @NotNull String columnPartition,
 @NotNull
 @NotNull String internalPartition,
 @NotNull
 @NotNull io.deephaven.shadow.enterprise.com.illumon.iris.db.tables.dataimport.logtailer.DataImportServer dis,
 @NotNull
 @NotNull Table sourceTable,
 @NotNull
 @NotNull DerivedTableWriter.Options options)

Ingest a Table to an in-worker DIS.

Parameters:

namespace - the namespace of the table to ingest

tableName - the name of the table to ingest

columnPartition - the column partition to ingest to

internalPartition - the internal partition to ingest to

dis - a DataImportServer instance, generally created with DataImportServerTools.getDisByName(String) or DataImportServerTools.getDisByNameWithStorage(String, String)

sourceTable - the table to ingest

options - an DerivedTableWriter.Options object that control ingestion behavior

Returns:

a LivenessArtifact that must be retained to ensure liveness

ingestPartitionedTable

public static LivenessArtifact ingestPartitionedTable(@NotNull
 @NotNull String namespace,
 @NotNull
 @NotNull String tableName,
 @NotNull
 @NotNull io.deephaven.shadow.enterprise.com.illumon.iris.db.tables.dataimport.logtailer.DataImportServer dis,
 @NotNull
 @NotNull PartitionedTable sourcePartitionedTable,
 @NotNull
 @NotNull DerivedTableWriter.PartitionedOptions partitionedOptions,
 @NotNull
 @NotNull DerivedTableWriter.Options options)

Ingest a PartitionedTable to an in-worker DIS.

Parameters:

namespace - the namespace of the table to ingest

tableName - the name of the table to ingest

dis - a DataImportServer instance, generally created with DataImportServerTools.getDisByName(String) or DataImportServerTools.getDisByNameWithStorage(String, String)

sourcePartitionedTable - the PartitionedTable to ingest

partitionedOptions - a DerivedTableWriter.PartitionedOptions object that controls ingestion behavior across partitions

options - an DerivedTableWriter.Options object that controls ingestion behavior for individual partitions

Returns:

a LivenessArtifact that must be retained to ensure liveness