Package io.deephaven.enterprise.kafkawriter

Class SchemaHelper

java.lang.Object
io.deephaven.enterprise.kafkawriter.SchemaHelper
public class SchemaHelper extends Object
The SchemaHelper assists query-builders with creating and validating Deephaven Enterprise schemas from a Core+ KafkaTableWriter.Options instance.

  • Constructor Details

    • SchemaHelper

      public SchemaHelper(@NotNull KafkaTableWriter.Options opts, @NotNull @NotNull String partitionColumn, int storageType)
      Instantiate a SchemaHelper instance.
      Parameters:
      opts - the KafkaTableWriter.Options to use
      partitionColumn - the partitioningColumn for the schema
      storageType - the StorageType for the schema

    • SchemaHelper

      public SchemaHelper(@NotNull KafkaTableWriter.Options opts, @NotNull @NotNull String partitionColumn)
      Instantiate a SchemaHelper instance.

      Will use a default storageType of STORAGETYPE_NESTEDPARTITIONEDONDISK

      Parameters:
      opts - the KafkaTableWriter.Options to use
      partitionColumn - the partitioningColumn for the schema

    • SchemaHelper

      @ScriptApi public SchemaHelper(@NotNull KafkaTableWriter.Options opts)
      Instantiate a SchemaHelper instance.

      Will use a default storageType of STORAGETYPE_NESTEDPARTITIONEDONDISK and a default partitionColumn of "Date"

      Parameters:
      opts - the KafkaTableWriter.Options to use

  • Method Details

    • withGroupingCols

      @ScriptApi public SchemaHelper withGroupingCols(@NotNull @NotNull Collection<String> groupingCols)
      Define column(s) as GroupingColumns.

      By default, columns are not defined as grouping columns

      Parameters:
      groupingCols - column(s) which should be defined as GroupingColumns
      Returns:
      this SchemaHelper instance

    • withGroupingCols

      @ScriptApi public SchemaHelper withGroupingCols(@NotNull @NotNull String... groupingCols)
      Define column(s) as GroupingColumns.

      By default, columns are not defined as grouping columns

      Parameters:
      groupingCols - column(s) which should be defined as GroupingColumns
      Returns:
      this SchemaHelper instance

    • withSymbolTableNone

      @ScriptApi public SchemaHelper withSymbolTableNone(@NotNull @NotNull Collection<String> symbolTableNone)
      Define column(s) which should be written without a SymbolTable.

      By default, String columns are written with a SymbolTable

      Parameters:
      symbolTableNone - column(s) which should be written without a SymbolTable
      Returns:
      this SchemaHelper instance

    • withSymbolTableNone

      @ScriptApi public SchemaHelper withSymbolTableNone(@NotNull @NotNull String... symbolTableNone)
      Define column(s) which should be written without a SymbolTable.

      By default, String columns are written with a SymbolTable

      Parameters:
      symbolTableNone - column(s) which should be written without a SymbolTable
      Returns:
      this SchemaHelper instance

    • withMergeKeyFormula

      @ScriptApi public SchemaHelper withMergeKeyFormula(@Nullable @Nullable String mergeKeyFormula)
      Define KeyFormula used for merge process.

      Defaults to "${autobalance_single}". To disable merge, set to null.

      Parameters:
      mergeKeyFormula - new merge keyFormula for schema
      Returns:
      this SchemaHelper instance

    • getCoreDefinition

      public TableDefinition getCoreDefinition()
      Constructs a Core TableDefinition instance.

      Uses the KeySpec, ValueSpec, and KafkaProperties values of the pre-defined KafkaTableWriter.Options

      Returns:
      a Core TableDefinition instance

    • getTableDefinition

      public io.deephaven.shadow.enterprise.com.illumon.iris.db.tables.TableDefinition getTableDefinition()
      Constructs a DHE TableDefinition instance.

      Converted from the Core TableDefinition returned by getCoreDefinition()

      Returns:
      a DHE TableDefinition instance

    • getXmlSchema

      public io.deephaven.shadow.enterprise.com.illumon.iris.db.schema.Schema getXmlSchema()
      Constructs a DHE Schema instance.

      Derived from the DHE TableDefinition returned by getTableDefinition()

      Returns:
      a DHE Schema instance

    • validateSchema

      @ScriptApi public void validateSchema() throws io.deephaven.shadow.enterprise.com.illumon.iris.db.schema.SchemaValidationException
      Validates the schema derived from getXmlSchema().

      Validates the schema against the latest schema known by the SchemaService. If the schema does not exist within the SchemaService or if the schemas do not match, an appropriate exception is thrown

      Throws:
      io.deephaven.shadow.enterprise.com.illumon.iris.db.schema.SchemaValidationException - an exception describing the cause of the validation failure

    • addOrValidateSchema

      @ScriptApi public boolean addOrValidateSchema() throws io.deephaven.shadow.enterprise.com.illumon.iris.db.schema.SchemaValidationException
      Validates the schema derived from getXmlSchema().

      Validates the schema against the latest schema known by the SchemaService. If the schema does not exist within the SchemaService, the schema will be added. If the schema does not match the latest schema, then an appropriate exception is thrown

      Returns:
      true if the schema is added by this call, else false if the schema matches the current schema
      Throws:
      io.deephaven.shadow.enterprise.com.illumon.iris.db.schema.SchemaValidationException - an exception describing the cause of the validation failure

    • validateOrUpdateSchema

      @ScriptApi public boolean validateOrUpdateSchema()
      Validates the schema derived from getXmlSchema().

      Validates the schema against the latest schema known by the SchemaService. If the schema does not exist within the SchemaService, the schema will be added. If the schema does not match the current schema, then an attempt is made to update the schema within the SchemaService. The only schema-changes permitted are adding or removing columns, changes to SymbolTable, Grouping, and merge KeyFormula. Any other schema change will throw an exception

      Existing partitions cannot be appended after a schema change takes effect. A schema change should only be done prior to adding data for a given partition (e.g. "Date")

      Returns:
      true if the schema is added or updated by this call, else false if the schema matches the current schema