Package io.deephaven.enterprise.kafkawriter

Class KafkaTableWriter

java.lang.Object

io.deephaven.enterprise.kafkawriter.KafkaTableWriter

@ScriptApi
public class KafkaTableWriter
extends Object

The KafkaTableWriter creates a Deephaven Core KafkaIngester that persists the data to disk and serves it to other queries using an in-worker instance of the DataImportServer.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	KafkaTableWriter.Options	A builder class containing the options for Kafka ingestion.

Field Summary

Fields

Modifier and Type	Field	Description
static final String	ENABLE_AUTO_COMMIT_PROPERTY

Constructor Summary

Constructors

Constructor	Description
KafkaTableWriter()

Method Summary

Modifier and Type	Method	Description
static void	consumeToDis(@NotNull KafkaTableWriter.Options options)	Consume a Kafka stream to a persistent Deephaven table.
static @NotNull io.deephaven.shadow.enterprise.com.illumon.iris.db.tables.dataimport.logtailer.DataImportServer	getDisByName(@NotNull String disName)	Retrieve a DataImportServer by name.
static @NotNull io.deephaven.shadow.enterprise.com.illumon.iris.db.tables.dataimport.logtailer.DataImportServer	getDisByNameWithStorage(@NotNull String disName, @NotNull String storageRoot)	Retrieve a DataImportServer by name, specifying the storage root.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

ENABLE_AUTO_COMMIT_PROPERTY

public static final String ENABLE_AUTO_COMMIT_PROPERTY

See Also:

• Constant Field Values

Constructor Details

KafkaTableWriter

public KafkaTableWriter()

Method Details

consumeToDis

@ScriptApi
public static void consumeToDis(@NotNull
 @NotNull KafkaTableWriter.Options options)

Consume a Kafka stream to a persistent Deephaven table.

The underlying ingestion is performed by the Deephaven Core KafkaTools class. In particular, the keySpec and valueSpec are Core KeyOrValueSpec objects.

Parameters:

options - an Options object that describes the ingestion process

getDisByName

@ScriptApi
@NotNull
public static @NotNull io.deephaven.shadow.enterprise.com.illumon.iris.db.tables.dataimport.logtailer.DataImportServer getDisByName(@NotNull
 @NotNull String disName)

Retrieve a DataImportServer by name.

The DIS is started as part of this call, and can be passed to the KafkaTableWriter.Options.dataImportServer(DataImportServer) function. The DIS can only be started one time, so if you are ingesting multiple tables each with a consumeToDis(Options) call, then you must call this function first and pass it to each Options structure.

If the DIS uses private storage, then you must call getDisByNameWithStorage(String, String).

Parameters:

disName - the name of the DIS to instantiate and start

Returns:

the dis referenced by the given name

getDisByNameWithStorage

@ScriptApi
@NotNull
public static @NotNull io.deephaven.shadow.enterprise.com.illumon.iris.db.tables.dataimport.logtailer.DataImportServer getDisByNameWithStorage(@NotNull
 @NotNull String disName,
 @NotNull
 @NotNull String storageRoot)

Retrieve a DataImportServer by name, specifying the storage root.

The DIS is started as part of this call, and can be passed to the KafkaTableWriter.Options.dataImportServer(DataImportServer) function. The DIS can only be started one time, so if you are ingesting multiple tables each with a consumeToDis(Options) call, then you must call this function first and pass it to each Options structure.

Parameters:

disName - the name of the DIS to instantiate and start

storageRoot - the storage root for DISes configured with "private storage".

Returns:

the dis referenced by the given name