Package io.deephaven.enterprise.database

Interface WritableDatabase

All Known Implementing Classes:
DatabaseImpl, WritableReplayDatabase
public interface WritableDatabase
Interface for writing tables to persistent Deephaven storage.

The WritableDatabase writes to User tables, this interface does not apply to System tables.

There are two types of historical User tables:

  • Unpartitioned: unpartitioned tables have a single location with data.
  • Partitioned: partitioned tables have one or more locations with data, that can be added and removed independently. Each partitioned User table must have exactly one partitioning column (e.g., Date) which separates the data by partition.

Historical User tables may be read and written from any Deephaven query worker, but users are responsible for ensuring that tables are not concurrently written.

Live User tables are always partitioned and are centrally managed by Deephaven system processes. Live tables provide an eventually consistent view of the data, even when multiple workers write to them.

  • Method Details

    • addUnpartitionedTable

      void addUnpartitionedTable(@NotNull @NotNull String namespace, @NotNull @NotNull String tableName, @NotNull @NotNull Table table)

      Adds an unpartitioned user table.

      Writes an unpartitioned user table to disk. If the namespace does not exist, then it is created. The schema must not already exist.

      If a UserTableIOException is thrown, the state of the table is undefined.

      Parameters:
      namespace - table namespace
      tableName - table name
      table - table that the definition and data will be based on
      Throws:
      UserTablePreexistingSchemaException - when a schema already exists for the given namespace and table name
      UserTablePermissionException - when the caller doesn't have full access to the specified namespace and table name
      UserTableNamespaceException - when a system namespace is specified
      UserTableWriteException - when there's a failure in data writing
      UserTableColumnDefinitionException - when a partitioning column is defined

    • deleteUnpartitionedTable

      boolean deleteUnpartitionedTable(@NotNull @NotNull String namespace, @NotNull @NotNull String tableName)
      Deletes an unpartitioned user table and the schema.

      If the schema does not exist, then data is not deleted. If there is no data but the schema exists, then the schema is deleted.

      The namespace is not removed even if this is the last table in the namespace.

      If a UserTableIOException is thrown, the state of the table is undefined.

      Parameters:
      namespace - table namespace
      tableName - table name
      Returns:
      true if the data was deleted, false if there was no data or preexisting schema
      Throws:
      UserTablePermissionException - when the caller doesn't have full access to the specified namespace and table name
      UserTableNamespaceException - when a system namespace is specified
      UserTableStorageTypeException - when the schema is not an unpartitioned table
      UserTableDeleteException - when there's a failure in data deletion

    • addPartitionedTableSchema

      boolean addPartitionedTableSchema(@NotNull @NotNull String namespace, @NotNull @NotNull String tableName, @NotNull @NotNull String partitionColumnName, @NotNull @NotNull TableDefinition prototype)
      Adds a schema for a partitioned user table.

      The schema is derived from the prototype TableDefinition and the partitionColumnName parameter. The prototype definition must not include a partitioning column.

      If the namespace does not exist, then it is created.

      If the schema already exists and it is identical (this is a stricter check than compatibility; all columns must be present in the same order with the same properties), then the method returns false. If the schema already exists and is not identical, then an error is thrown.

      Parameters:
      namespace - table namespace
      tableName - table name
      partitionColumnName - name of the partitioning column
      prototype - table definition to derive schema from
      Returns:
      true if the partitioned table schema was added, false if it already existed
      Throws:
      UserTableColumnDefinitionException - when the prototype contains a column with the same name as partitionColumnName
      UserTablePermissionException - when the caller doesn't have full access to the specified namespace and table name
      UserTablePreexistingSchemaException - when a schema already exists and is not identical
      UserTableNamespaceException - when a system namespace is specified

    • updatePartitionedTableSchema

      boolean updatePartitionedTableSchema(@NotNull @NotNull String namespace, @NotNull @NotNull String tableName, @NotNull @NotNull TableDefinition prototype)
      Updates a preexisting User table schema.

      If the schema does not exist an error is thrown. Not all schema modifications are permitted. The partitioning column may not be changed. Existing columns may not have their type changed. Columns may be added or deleted.

      Note that no data is modified by this operation. Removed columns remain on persistent storage, and added columns are treated as null on read.

      Although each modification in isolation is verified for safety, a sequence of modifications to the schema may be unsafe. For example, deleting a column and adding it back with a new type results in unreadable data.

      Parameters:
      namespace - table namespace
      tableName - table name
      prototype - table definition to derive schema from
      Returns:
      true if the partitioned table schema was updated, false if there was already an identical definition
      Throws:
      UserTablePermissionException - when the caller doesn't have full access to the specified namespace and table name
      UserTableMissingSchemaException - when there is no table to update
      UserTableStorageTypeException - when the schema is not a partitioned table
      UserTableColumnDefinitionException - when the target user table doesn't have exactly one partitioning column, the prototype contains a column with the same name as the partitioning column, or a column's type is attempted to be updated
      UserTableNamespaceException - when a system namespace is specified

    • addTablePartition

      void addTablePartition(@NotNull @NotNull String namespace, @NotNull @NotNull String tableName, @NotNull @NotNull String partitionColumnValue, @NotNull @NotNull Table table)
      Adds a single column partition of data to a partitioned user table.

      The data table must have a mutually compatible definition with the current schema. The data table must not have a column with the same name as the partitioning column.

      If a UserTableIOException is thrown, the state of the table is undefined.

      Parameters:
      namespace - table namespace
      tableName - table name
      partitionColumnValue - value for the partitioning column, e.g. "2015-09-25" for "Date"
      table - table to write data from
      Throws:
      UserTablePermissionException - when the caller doesn't have full access to the specified namespace and table name
      UserTableMissingSchemaException - when a table schema is absent for the given namespace and table name
      UserTableNamespaceException - when a system namespace is specified
      UserTableStorageTypeException - when the schema is not a partitioned table
      UserTableColumnDefinitionException - when the target user table doesn't have exactly one partitioning column or the data contains a column with the same name as the partitioning column
      TableDefinition.IncompatibleTableDefinitionException - when the table input definition isn't mutually compatible
      UserTableWriteException - when there's a failure in data writing

    • deleteTablePartition

      boolean deleteTablePartition(@NotNull @NotNull String namespace, @NotNull @NotNull String tableName, @NotNull @NotNull String partitionColumnValue)
      Deletes a single column partition of data from a partitioned user table.

      If a UserTableIOException is thrown, the state of the table is undefined.

      Parameters:
      namespace - table namespace
      tableName - table name
      partitionColumnValue - value for the partitioning column, e.g. "2015-09-25" for "Date"
      Returns:
      true if the partition was deleted, false if there was no partition or preexisting schema
      Throws:
      UserTablePermissionException - when the caller doesn't have full access to the specified namespace and table name
      UserTableNamespaceException - when a system namespace is specified
      UserTableStorageTypeException - when the schema is not a partitioned table
      UserTableDeleteException - when there's a failure in data deletion

    • appendLiveTable

      void appendLiveTable(@NotNull @NotNull String namespace, @NotNull @NotNull String tableName, @NotNull @NotNull String partitionColumnValue, @NotNull @NotNull Table table)
      Appends all rows from a given table to a live user table partition.

      The data table must have a mutually compatible definition with the current schema. The data table must not have a column with the same name as the partitioning column.

      The specified column partition may already exist, or may be created by the system.

      This method is asynchronous. After returning, the data may not be immediately available. It is possible for the write to fail after this method has returned. When multiple workers append to a partition, ordering is imposed outside the worker by other system components.

      If a UserTableIOException is thrown, the state of the table is undefined.

      Parameters:
      namespace - table namespace
      tableName - table name
      partitionColumnValue - value for the partitioning column, e.g. "2015-09-25" for "Date"
      table - table to append rows from
      Throws:
      UserTablePermissionException - when the caller doesn't have full access to the specified namespace and table name
      UserTableMissingSchemaException - when a table schema is absent for the given namespace and table name
      UserTableNamespaceException - when a system namespace is specified
      UserTableStorageTypeException - when the schema is not a partitioned table
      UserTableColumnDefinitionException - when the target user table doesn't have exactly one partitioning column or the table contains a column with the same name as the partitioning column
      TableDefinition.IncompatibleTableDefinitionException - when the table input definition isn't mutually compatible
      UserTableWriteException - when there's a failure in appending data

    • appendLiveTableIncremental

      SafeCloseable appendLiveTableIncremental(@NotNull @NotNull String namespace, @NotNull @NotNull String tableName, @NotNull @NotNull String partitionColumnValue, @NotNull @NotNull Table table)
      Appends all rows from a given table to a live user table partition. When rows are added to the table, they are additionally appended to the table.

      The input table updates can only have additions and shifts. No modifications or removals are permitted.

      The data table must have a mutually compatible definition with the current schema. The data table must not have a column with the same name as the partitioning column.

      The specified column partition may already exist, or may be created by the system.

      This method is asynchronous, after returning the data may not be immediately available. It is possible for the write to fail after this method has returned. When multiple workers append to a partition, ordering is imposed outside the worker by other system components.

      A reference must be maintained to the returned SafeCloseable to ensure expected functionality; calling SafeCloseable.close() will stop incremental appends, and clean up related resources.

      If a UserTableIOException is thrown, the state of the table is undefined.

      Parameters:
      namespace - table namespace
      tableName - table name
      partitionColumnValue - value for the partitioning column, e.g. "2015-09-25" for "Date"
      table - table to append updates from
      Returns:
      the Closeable reference used to ensure and stop expected functionality
      Throws:
      UserTablePermissionException - when the caller doesn't have full access to the specified namespace and table name
      UserTableMissingSchemaException - when a table schema is absent for the given namespace and table name
      UserTableNamespaceException - when a system namespace is specified
      UserTableStorageTypeException - when the schema is not a partitioned table
      UserTableColumnDefinitionException - when the target schema does not have exactly one partitioning column or the table contains a column with the same name as the partitioning column
      TableDefinition.IncompatibleTableDefinitionException - when the table input definition isn't mutually compatible
      UserTableWriteException - when there's a failure in appending data

    • deleteLiveTablePartition

      boolean deleteLiveTablePartition(@NotNull @NotNull String namespace, @NotNull @NotNull String tableName, @NotNull @NotNull String partitionColumnValue)
      Delete a partition from a live user table.

      If a UserTableIOException is thrown, the state of the table is undefined.

      Parameters:
      namespace - table namespace
      tableName - table name
      partitionColumnValue - value for the partitioning column, e.g. "2015-09-25" for "Date"
      Returns:
      true if the partition was deleted, false if there was no partition or preexisting schema
      Throws:
      UserTablePermissionException - when the caller doesn't have full access to the specified namespace and table name
      UserTableNamespaceException - when a system namespace is specified
      UserTableStorageTypeException - when the schema is not a partitioned table
      UserTableColumnDefinitionException - when the target user table doesn't have exactly one partitioning column
      UserTableDeleteException - when there's a failure deleting data

    • deletePartitionedTable

      boolean deletePartitionedTable(@NotNull @NotNull String namespace, @NotNull @NotNull String tableName)
      Delete all partitions, whether direct or live, and the schema, from a partitioned user table.

      All partitions from the table are deleted sequentially. If a partition cannot be deleted, then the operation fails but some data may have already been removed. After all partitions are deleted, then the schema is deleted. If the schema does not exist, then data is not deleted. If there is no data, but the schema exists, then the schema is deleted.

      The namespace is not removed, even if this is the last table in the namespace.

      If a UserTableIOException is thrown, the state of the table is undefined.

      Parameters:
      namespace - table namespace
      tableName - table name
      Returns:
      true if data was deleted, false if there was no data or preexisting schema
      Throws:
      UserTablePermissionException - when the caller doesn't have full access to the specified namespace and table name
      UserTableNamespaceException - when a system namespace is specified
      UserTableStorageTypeException - when the schema is not a partitioned table
      UserTableColumnDefinitionException - when the target user table doesn't have exactly one partitioning column
      UserTableDeleteException - when there's a failure deleting data

    • addInputTableSchema

      boolean addInputTableSchema(@NotNull @NotNull String namespace, @NotNull @NotNull String tableName, @NotNull @NotNull InputTableSpec inputTableSpec)
      Add a new input table schema using the InputTableSpec.
      Parameters:
      namespace - the namespace of the input table
      tableName - the name of the input table
      inputTableSpec - the input table specification
      Returns:
      true if the input table schema was added, false if the input table already exists with the same spec
      Throws:
      UserTablePreexistingSchemaException - if the input table exists, but does not match spec

    • addInputTableSchema

      boolean addInputTableSchema(@NotNull @NotNull String namespace, @NotNull @NotNull String tableName, @NotNull @NotNull TableDefinition prototype, @NotNull @NotNull String... keyColNames)
      Add a new input table schema using the TableDefinition and specified key column names.
      Parameters:
      namespace - the namespace of the input table
      tableName - the name of the input table
      prototype - TableDefinition to derive the input table spec from
      keyColNames - columns that should be keyed in the input table
      Returns:
      true if the input table schema was added, false if the input table already exists with the same spec
      Throws:
      UserTablePreexistingSchemaException - if the input table exists, but does not match spec

    • updateInputTableSchema

      boolean updateInputTableSchema(@NotNull @NotNull String namespace, @NotNull @NotNull String tableName, InputTableSpec inputTableSpec)
      Update an existing input table schema using the provided InputTableSpec.

      Retrieve and use a new InputTableUpdater via inputTableUpdater(String, String) after updating an input table's specification to ensure proper behavior.

      Parameters:
      namespace - the namespace of the input table
      tableName - the name of the input table
      inputTableSpec - the new specification for the input table
      Returns:
      true if the input table schema was updated, false if the input table already exists with the same spec
      Throws:
      UserInputTableException - when the requested update is invalid

    • deleteInputTable

      boolean deleteInputTable(@NotNull @NotNull String namespace, @NotNull @NotNull String tableName)
      Delete the input table given the namespace and table name.
      Parameters:
      namespace - the namespace of the input table
      tableName - the name of the input table
      Returns:
      true if the input table was deleted, false if there was no data or preexisting schema for the input table

    • inputTableUpdater

      InputTableUpdater inputTableUpdater(@NotNull @NotNull String namespace, @NotNull @NotNull String tableName)
      Retrieve the specified InputTableUpdater, which can be used to programmatically add or delete rows.

      If the caller needs a consistent view between the returned InputTableUpdater and an input table view, the input table view must be derived from InputTableUpdater.table().

      Parameters:
      namespace - the namespace in which the table exists.
      tableName - the name of the table in the namespace.
      Returns:
      an InputTableUpdater for the specified input table.

    • inputTable

      Table inputTable(@NotNull @NotNull String namespace, @NotNull @NotNull String tableName)
      Retrieve the specified input table view.

      If the caller needs a consistent view between the returned table and an InputTableUpdater, the InputTableUpdater must be derived from the Table.INPUT_TABLE_ATTRIBUTE from the returned table. Callers may also get a consistent view by calling inputTableUpdater(String, String) and deriving the table from InputTableUpdater.table().

      Parameters:
      namespace - the namespace in which the table exists.
      tableName - the name of the table in the namespace.
      Returns:
      a table view for the specified input table.

    • inputTableSpecFor

      InputTableSpec inputTableSpecFor(@NotNull @NotNull String namespace, @NotNull @NotNull String tableName)
      Retrieve the current InputTableSpec for the given namespace and table.
      Parameters:
      namespace - the namespace of the input table
      tableName - the name of the input table
      Returns:
      the current input table specification for the given namespace and table

    • isInputTable

      boolean isInputTable(@NotNull @NotNull String namespace, @NotNull @NotNull String tableName)
      Determines whether a table is a Core+ input table.
      Parameters:
      namespace - the namespace of the input table
      tableName - the name of the input table
      Returns:
      true if the table is an input table, false if not
      Throws:
      SchemaNotFoundException - if there is no schema for the specified table