Package com.illumon.iris.db.tables.databases

Class OnDiskQueryDatabase

java.lang.Object

com.illumon.iris.db.tables.databases.BaseDatabase

com.illumon.iris.db.tables.databases.AbstractDatabase

com.illumon.iris.db.tables.databases.OnDiskDatabase

com.illumon.iris.db.tables.databases.OnDiskQueryDatabase

All Implemented Interfaces:

Database

Direct Known Subclasses:

ImporterDatabase

public class OnDiskQueryDatabase
extends OnDiskDatabase

Primary local database implementation for query usage.

Nested Class Summary

Nested classes/interfaces inherited from class com.illumon.iris.db.tables.databases.OnDiskDatabase

OnDiskDatabase.AuditEventLogDetails

Nested classes/interfaces inherited from interface com.illumon.iris.db.tables.databases.Database

Database.LogHandle, Database.StorageFormat

Field Summary

Fields inherited from class com.illumon.iris.db.tables.databases.OnDiskDatabase

DEFAULT_ROOT_DIRECTORY, locationManager, log, ON_DISK_DATABASE_ROOT_DIRECTORY_PROP, permissionFilterProvider, TABLE_DEFINITIONS_TABLE_DEFINITION, userContext

Constructor Summary

Constructors

Constructor	Description
OnDiskQueryDatabase(com.fishlib.io.logger.Logger log)	Deprecated. The tableDataServiceName should be known.
OnDiskQueryDatabase(com.fishlib.io.logger.Logger log, com.fishlib.configuration.Configuration configuration, File rootDirectory, TableDataService tableDataService, SchemaService schemaService, NamespaceSet newNamespaceSet)	Opens an on-disk database suitable for user queries with the specified root and no support for query parallelization.
OnDiskQueryDatabase(com.fishlib.io.logger.Logger log, File rootDirectory, TableDataService tableDataService)	Delegate to OnDiskQueryDatabase(Logger, File, TableDataService, SchemaService) with the default SchemaService.
OnDiskQueryDatabase(com.fishlib.io.logger.Logger log, File rootDirectory, TableDataService tableDataService, SchemaService schemaService)	Delegate to OnDiskQueryDatabase(Logger, File, TableDataService, SchemaService, NamespaceSet) with NamespaceSet User.
OnDiskQueryDatabase(com.fishlib.io.logger.Logger log, File rootDirectory, TableDataService tableDataService, SchemaService schemaService, NamespaceSet newNamespaceSet)	Opens an on-disk database suitable for user queries with the specified root and no support for query parallelization, using the default Configuration instance.
OnDiskQueryDatabase(com.fishlib.io.logger.Logger log, File rootDirectory, String tableDataServiceName)	Delegate to OnDiskQueryDatabase(Logger, File, TableDataService) with a TableDataService constructed from the given tableDataServiceName.
OnDiskQueryDatabase(com.fishlib.io.logger.Logger log, String tableDataServiceName)	Delegate to OnDiskQueryDatabase(Logger, File, TableDataService) with the default rootDirectory.

Method Summary

Modifier and Type	Method	Description
protected void	appendColumnPartition(TableDefinitionSchema schema, String internalPartitionValue, String columnPartitionValue, Table data)
protected void	appendTable(TableDefinitionSchema schema, Table table)
protected void	checkAllowAddColumnPartition(TableDefinitionSchema schema, Table newTable)
protected void	checkAllowAddNamespace(NamespaceSet namespaceSet, String namespace)
protected void	checkAllowAddSchema(TableDefinitionSchema schema)
protected void	checkAllowCleanupNamespace(String namespace)
protected void	checkAllowDeleteColumnPartition(TableDefinitionSchema schema)
protected void	checkAllowIntradayOperation(TableDefinitionSchema schema)
protected void	checkAllowRemoveNamespace(String namespace)
protected void	checkAllowRemoveTable(TableDefinitionSchema schema)
protected void	checkAllowRemoveTableCentral(TableDefinitionSchema schema)
protected void	checkAllowRemoveTableDefinition(TableDefinitionSchema schema)
protected void	checkAllowRemoveTableDefinition(String namespace, String tableName)
protected void	checkAllowReplaceColumnPartition(TableDefinitionSchema schema, Table newTable)
protected void	checkAllowReplaceTable(TableDefinitionSchema schema, Table newTable)
protected void	checkAllowReplaceTableDefinition(TableDefinitionSchema schema)
void	closeAndDeleteCentral(String namespace, String tableName)	Close all logger connections to the centralized logging for the specified table, and then atomically delete all centrally logged table data for this namespace and table name.
void	closeAndDeleteCentral(String namespace, String tableName, String columnPartitionValue)	Close all logger connections to the centralized logging for the specified table, and then atomically delete all centrally logged table data for this namespace and table name, restricted to the column partition value if provided.
protected void	deleteBackupTable(TableDefinitionSchema schema)
protected void	deleteColumnPartition(TableDefinitionSchema schema, String internalPartitionValue, String columnPartitionValue)
protected void	deleteTable(TableDefinitionSchema schema)
protected List<String>	getIntradayPartitions(TableDefinitionSchema schema)
protected Table	getIntradayTableV2(TableDefinitionSchema schema, boolean live)
protected Table	getIntradayTableV2(TableDefinitionSchema schema, Set<String> partitionNames, boolean live)
Table	getLastByTable(String namespace, String tableName)	Retrieves an intraday last-by table by namespace and name.
TableDataService	getTableDataService()
protected boolean	hasData(NamespaceSet set, TableDefinition tDef, String columnPartitionValue, boolean intraday)
protected void	installShadowTable(TableDefinitionSchema schema)
protected void	moveTableToBackup(TableDefinitionSchema schema)
protected void	overwriteColumnPartition(TableDefinitionSchema schema, String internalPartitionValue, String columnPartitionValue, Table data, Database.StorageFormat storageFormat)
void	reloadData()	Reload any internally-cached data.
void	removeTable(String namespace, String tableName)	Removes a table.
void	removeTablePartition(String namespace, String tableName, String internalPartitionValue, String columnPartitionValue)	Removes a single table location (e.g.
void	replaceTable(String namespace, String tableName, Table table, Database.StorageFormat storageFormat)	Replaces a non-partitioned table with a new table, or adds the table, if it is not already present.
void	replaceTablePartition(String namespace, String tableName, String internalPartitionValue, String columnPartitionValue, Table data, Database.StorageFormat storageFormat)	Replaces a single table location (e.g.
protected Table	restoreTable(TableDefinitionSchema schema, boolean isLive)
protected void	rollbackTable(TableDefinitionSchema schema)
int	setWatchdogMillis(int millis)	Sets monitoring of delays in the LiveTableMonitor refresh loop.
void	setWatchdogTimeoutProcedure(LongConsumer watchdogTimeoutProcedure)	Set the watchdog timeout procedure.
protected void	storeColumnPartition(TableDefinitionSchema schema, String internalPartitionValue, String columnPartitionValue, Table data, Database.StorageFormat storageFormat)
protected void	storeShadowTable(TableDefinitionSchema schema, Table table, Database.StorageFormat storageFormat)
protected void	storeTable(TableDefinitionSchema schema, Table table, Database.StorageFormat storageFormat)
protected void	verifyNamespace(String namespace, String tableName)	Ensure the namespace exists.

Methods inherited from class com.illumon.iris.db.tables.databases.OnDiskDatabase

addTable, addTableDefinition, addTablePartition, appendCentral, appendTable, appendTablePartition, checkAllowAddTable, checkAllowAddTableCentral, deleteMatchingTables, deleteNamespace, deleteTables, getAuditEventLogLogger, getCatalog, getColumnDefinitionsTable, getColumnPartitionTableWriter, getIntradayPartitions, getIntradayPartitionsAsTable, getIntradayTableV2, getIntradayTableV2, getIntradayTableV2, getNamespaces, getPermissionFilterProvider, getRootDirectory, getSchemaService, getSystemNamespaces, getTableDefinition, getTableDefinitionsTable, getTableNames, getTableV2, getTableWriter, getUserContext, getUserNamespaces, hasImportedData, hasIntradayData, hasIntradayTable, hasNamespace, hasTable, logPartitionTableIncremental, logPartitionTableIncremental, logPartitionTableSnapshot, logPartitionTableSnapshot, removeTableDefinition, repairMetadata, repairMetadata, repairMetadata, replaceTableDefinition, setAuditEventLogger, setUserContext, setUserContextAndFilterProvider

Methods inherited from class com.illumon.iris.db.tables.databases.AbstractDatabase

executeQuery, gc, getClientHost, getErrorLog, getProcessInfoId, getReplayer, getReplayer, getServerHost, getWorkerName, importClass, importStatic, logPartitionTableSnapshot, logPartitionTableSnapshot, makeRemote, mr, shutdown, startup

Methods inherited from class com.illumon.iris.db.tables.databases.BaseDatabase

addPartitionedTableDefinition, addPartitionedTableDefinition, addPartitionedTableDefinitionIfAbsent, addPartitionedTableDefinitionIfAbsent, addTablePartition, addTablePartitionAndDefinition, addTablePartitionAndDefinition, appendCentral, appendTablePartition, removeTablePartition, replaceTablePartition

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface com.illumon.iris.db.tables.databases.Database

addColumnPartitionAndDefinition, addColumnPartitionAndDefinition, addTable, addTablePartition, addTablePartition, getIntradayTable, getIntradayTable, getIntradayTable, getIntradayTable, getIntradayTable, getIntradayTable, getIntradayTableV2, getIntradayTableV2, getIntradayTableV2, getIntradayTableV2, getReplayer, getReplayer, getReplayer, getTable, i, i, i, i, i, i, i2, i2, i2, i2, i2, i2, replaceTable, replaceTablePartition, replaceTablePartition, rm, t, t2, timeTable, timeTable, timeTable, timeTable, timeTable, timeTable, timeTable, timeTable, timeTable, timeTable, timeTable, timeTable

Constructor Details

OnDiskQueryDatabase

public OnDiskQueryDatabase(@NotNull com.fishlib.io.logger.Logger log, @Nullable File rootDirectory, @NotNull String tableDataServiceName)

Delegate to OnDiskQueryDatabase(Logger, File, TableDataService) with a TableDataService constructed from the given tableDataServiceName.

Parameters:

log - the logger to use

rootDirectory - the root directory for the database's data and metadata storage

tableDataServiceName - initialize the TableDataService with this configuration name.

OnDiskQueryDatabase

public OnDiskQueryDatabase(@NotNull com.fishlib.io.logger.Logger log, @NotNull File rootDirectory, @NotNull TableDataService tableDataService)

Delegate to OnDiskQueryDatabase(Logger, File, TableDataService, SchemaService) with the default SchemaService.

Parameters:

log - the logger to use

rootDirectory - the root directory for the database's data and metadata storage

tableDataService - the constructed TableDataService to use

OnDiskQueryDatabase

public OnDiskQueryDatabase(@NotNull com.fishlib.io.logger.Logger log, @NotNull File rootDirectory, @NotNull TableDataService tableDataService, @NotNull SchemaService schemaService)

Delegate to OnDiskQueryDatabase(Logger, File, TableDataService, SchemaService, NamespaceSet) with NamespaceSet User.

Parameters:

log - the logger to use

rootDirectory - the root directory for the database's data and metadata storage

tableDataService - the constructed TableDataService to use

schemaService - the SchemaService to use

OnDiskQueryDatabase

@Deprecated public OnDiskQueryDatabase(@NotNull com.fishlib.io.logger.Logger log)

Deprecated.

The tableDataServiceName should be known. Use OnDiskQueryDatabase(Logger, File, String) instead.

Delegate to OnDiskQueryDatabase(Logger, File, String) with the default table data service configuration name and null root directory, signifying the default.

Parameters:

log - the logger to use

OnDiskQueryDatabase

public OnDiskQueryDatabase(@NotNull com.fishlib.io.logger.Logger log, @NotNull String tableDataServiceName)

Delegate to OnDiskQueryDatabase(Logger, File, TableDataService) with the default rootDirectory.

Parameters:

log - the logger to use

tableDataServiceName - initialize the TableDataService with this configuration name.

OnDiskQueryDatabase

public OnDiskQueryDatabase(@NotNull com.fishlib.io.logger.Logger log, @Nullable File rootDirectory, @NotNull TableDataService tableDataService, @NotNull SchemaService schemaService, @NotNull NamespaceSet newNamespaceSet)

Opens an on-disk database suitable for user queries with the specified root and no support for query parallelization, using the default Configuration instance.

Parameters:

log - the logger to use

rootDirectory - the root directory for the database's data and metadata storage

tableDataService - the constructed TableDataService to use

schemaService - the SchemaService to use

newNamespaceSet - the NamespaceSet to use for newly created namespaces

OnDiskQueryDatabase

public OnDiskQueryDatabase(@NotNull com.fishlib.io.logger.Logger log, @NotNull com.fishlib.configuration.Configuration configuration, @Nullable File rootDirectory, @NotNull TableDataService tableDataService, @NotNull SchemaService schemaService, @NotNull NamespaceSet newNamespaceSet)

Opens an on-disk database suitable for user queries with the specified root and no support for query parallelization.

Parameters:

log - the logger to use

configuration - the Configuration instance to use

rootDirectory - the root directory for the database's data and metadata storage

tableDataService - the constructed TableDataService to use

schemaService - the SchemaService to use

newNamespaceSet - the NamespaceSet to use for newly created namespaces

Method Details

getTableDataService

public TableDataService getTableDataService()

reloadData

public void reloadData()

Reload any internally-cached data.

ImplNote:

OnDiskQueryDatabase resets its TableDataService.

checkAllowAddSchema

protected void checkAllowAddSchema(TableDefinitionSchema schema)

Overrides:

checkAllowAddSchema in class OnDiskDatabase

checkAllowAddNamespace

protected void checkAllowAddNamespace(NamespaceSet namespaceSet, String namespace)

Overrides:

checkAllowAddNamespace in class OnDiskDatabase

checkAllowRemoveNamespace

protected void checkAllowRemoveNamespace(String namespace)

Overrides:

checkAllowRemoveNamespace in class OnDiskDatabase

storeTable

protected void storeTable(@NotNull TableDefinitionSchema schema, Table table, Database.StorageFormat storageFormat)

Specified by:

storeTable in class OnDiskDatabase

appendTable

protected void appendTable(TableDefinitionSchema schema, Table table)

Specified by:

appendTable in class OnDiskDatabase

checkAllowRemoveTableDefinition

protected void checkAllowRemoveTableDefinition(TableDefinitionSchema schema)

Overrides:

checkAllowRemoveTableDefinition in class OnDiskDatabase

checkAllowRemoveTableDefinition

protected void checkAllowRemoveTableDefinition(String namespace, String tableName)

Overrides:

checkAllowRemoveTableDefinition in class OnDiskDatabase

checkAllowRemoveTable

protected void checkAllowRemoveTable(TableDefinitionSchema schema)

Overrides:

checkAllowRemoveTable in class OnDiskDatabase

checkAllowRemoveTableCentral

protected void checkAllowRemoveTableCentral(TableDefinitionSchema schema)

Overrides:

checkAllowRemoveTableCentral in class OnDiskDatabase

checkAllowCleanupNamespace

protected void checkAllowCleanupNamespace(String namespace)

Overrides:

checkAllowCleanupNamespace in class OnDiskDatabase

deleteTable

protected void deleteTable(TableDefinitionSchema schema)

Specified by:

deleteTable in class OnDiskDatabase

checkAllowReplaceTableDefinition

protected void checkAllowReplaceTableDefinition(TableDefinitionSchema schema)

Overrides:

checkAllowReplaceTableDefinition in class OnDiskDatabase

checkAllowReplaceTable

protected void checkAllowReplaceTable(TableDefinitionSchema schema, Table newTable)

Overrides:

checkAllowReplaceTable in class OnDiskDatabase

verifyNamespace

protected void verifyNamespace(String namespace, @Nullable String tableName)

Description copied from class: OnDiskDatabase

Ensure the namespace exists.

Specified by:

verifyNamespace in class OnDiskDatabase

Parameters:

namespace - name of the namespace to check

tableName - optional table name for logging context

replaceTablePartition

public void replaceTablePartition(String namespace, String tableName, String internalPartitionValue, String columnPartitionValue, Table data, Database.StorageFormat storageFormat)

Replaces a single table location (e.g. one date) from/to a partitioned table.

Requires that a suitable TableDefinition has already been added.

Not guaranteed to be atomic in the case of failures.

Specified by:

replaceTablePartition in interface Database

Overrides:

replaceTablePartition in class OnDiskDatabase

Parameters:

namespace - table namespace

tableName - table name

internalPartitionValue - internal (not user-visible) partition value.

columnPartitionValue - column partition value (e.g. "2015-09-25" for a Date partitioning column).

data - data to replace this partition. Must *not* contain the partitioning column.

storageFormat - the storage format to use (ex DeephavenV1 or Parquet)

ImplNote:

OnDiskQueryDatabase resets its TableDataService for the table.

replaceTable

public void replaceTable(String namespace, String tableName, Table table, Database.StorageFormat storageFormat)

Replaces a non-partitioned table with a new table, or adds the table, if it is not already present. Not guaranteed to be atomic in the case of failures.

Specified by:

replaceTable in interface Database

Overrides:

replaceTable in class OnDiskDatabase

Parameters:

namespace - table namespace

tableName - table name

table - data for the replacement table

storageFormat - the storage format to use (ex DeephavenV1 or Parquet)

ImplNote:

OnDiskQueryDatabase resets its TableDataService for the table.

removeTablePartition

public void removeTablePartition(String namespace, String tableName, String internalPartitionValue, String columnPartitionValue)

Removes a single table location (e.g. one date) from a partitioned table. Requires that a suitable TableDefinition still exists.

Specified by:

removeTablePartition in interface Database

Overrides:

removeTablePartition in class OnDiskDatabase

Parameters:

namespace - table namespace

tableName - table name

internalPartitionValue - internal (not user-visible) partition value.

columnPartitionValue - column partition value (e.g. "2015-09-25" for a Date partitioning column).

ImplNote:

OnDiskQueryDatabase resets its TableDataService for the table.

removeTable

public void removeTable(String namespace, String tableName)

Removes a table. This method is a better choice than Database.deleteTables(String, String...), unless the underlying table no longer has consistent metadata. Throws an exception for tables that don't exist.

Specified by:

removeTable in interface Database

Overrides:

removeTable in class OnDiskDatabase

Parameters:

namespace - table namespace

tableName - table name

ImplNote:

OnDiskQueryDatabase resets its TableDataService for the table.

storeShadowTable

protected void storeShadowTable(TableDefinitionSchema schema, Table table, Database.StorageFormat storageFormat)

Specified by:

storeShadowTable in class OnDiskDatabase

moveTableToBackup

protected void moveTableToBackup(TableDefinitionSchema schema)

Specified by:

moveTableToBackup in class OnDiskDatabase

installShadowTable

protected void installShadowTable(TableDefinitionSchema schema)

Specified by:

installShadowTable in class OnDiskDatabase

rollbackTable

protected void rollbackTable(TableDefinitionSchema schema)

Specified by:

rollbackTable in class OnDiskDatabase

deleteBackupTable

protected void deleteBackupTable(TableDefinitionSchema schema)

Specified by:

deleteBackupTable in class OnDiskDatabase

restoreTable

protected Table restoreTable(TableDefinitionSchema schema, boolean isLive)

Specified by:

restoreTable in class OnDiskDatabase

checkAllowAddColumnPartition

protected void checkAllowAddColumnPartition(TableDefinitionSchema schema, Table newTable)

Overrides:

checkAllowAddColumnPartition in class OnDiskDatabase

checkAllowDeleteColumnPartition

protected void checkAllowDeleteColumnPartition(TableDefinitionSchema schema)

Overrides:

checkAllowDeleteColumnPartition in class OnDiskDatabase

checkAllowReplaceColumnPartition

protected void checkAllowReplaceColumnPartition(TableDefinitionSchema schema, Table newTable)

Overrides:

checkAllowReplaceColumnPartition in class OnDiskDatabase

checkAllowIntradayOperation

protected void checkAllowIntradayOperation(TableDefinitionSchema schema)

Overrides:

checkAllowIntradayOperation in class OnDiskDatabase

storeColumnPartition

protected void storeColumnPartition(TableDefinitionSchema schema, String internalPartitionValue, String columnPartitionValue, Table data, Database.StorageFormat storageFormat)

Specified by:

storeColumnPartition in class OnDiskDatabase

appendColumnPartition

protected void appendColumnPartition(TableDefinitionSchema schema, String internalPartitionValue, String columnPartitionValue, Table data)

Specified by:

appendColumnPartition in class OnDiskDatabase

deleteColumnPartition

protected void deleteColumnPartition(TableDefinitionSchema schema, String internalPartitionValue, String columnPartitionValue)

Specified by:

deleteColumnPartition in class OnDiskDatabase

overwriteColumnPartition

protected void overwriteColumnPartition(TableDefinitionSchema schema, String internalPartitionValue, String columnPartitionValue, Table data, Database.StorageFormat storageFormat)

Specified by:

overwriteColumnPartition in class OnDiskDatabase

getIntradayPartitions

protected List<String> getIntradayPartitions(TableDefinitionSchema schema)

Specified by:

getIntradayPartitions in class OnDiskDatabase

getIntradayTableV2

protected Table getIntradayTableV2(TableDefinitionSchema schema, boolean live)

Specified by:

getIntradayTableV2 in class OnDiskDatabase

getIntradayTableV2

protected Table getIntradayTableV2(TableDefinitionSchema schema, Set<String> partitionNames, boolean live)

Specified by:

getIntradayTableV2 in class OnDiskDatabase

hasData

protected boolean hasData(NamespaceSet set, TableDefinition tDef, @Nullable String columnPartitionValue, boolean intraday)

Specified by:

hasData in class OnDiskDatabase

tDef - the definition of the table

columnPartitionValue - partitioning column value to check. Null indicates "any" value.

intraday - if false, only historical partitions are checked.

Returns:

True if the specified table location has data.

getLastByTable

public Table getLastByTable(String namespace, String tableName)

Description copied from interface: Database

Retrieves an intraday last-by table by namespace and name.

Throws exceptions for unknown tables.

Always provides a live result.

This may not be configured on your system.

Parameters:

namespace - namespace of the table

tableName - name of the table

Returns:

live intraday last-by table

setWatchdogMillis

public int setWatchdogMillis(int millis)

Description copied from interface: Database

Sets monitoring of delays in the LiveTableMonitor refresh loop. If the LiveTableMonitor refresh loop takes longer than the specified amount of time, the watchdog timeout procedure is executed. The watchdog timeout procedure is specified by calling Database.setWatchdogTimeoutProcedure(LongConsumer).

Parameters:

millis - if LiveTableMonitor refresh loop takes longer than this amount of time, the watchdog timeout procedure will be called.

Returns:

the new value of the watchdog timer, in ms

setWatchdogTimeoutProcedure

public void setWatchdogTimeoutProcedure(LongConsumer watchdogTimeoutProcedure)

Description copied from interface: Database

Set the watchdog timeout procedure. If LiveTableMonitor refresh loop takes longer than the amount of time specified via Database.setWatchdogMillis(int), the timeout procedure is called.

Parameters:

watchdogTimeoutProcedure - the procedure called when the LiveTableMonitor refresh loop takes longer than watchdogMillis.

closeAndDeleteCentral

public void closeAndDeleteCentral(@NotNull String namespace, @NotNull String tableName, @Nullable String columnPartitionValue) throws IOException

Description copied from interface: Database

Close all logger connections to the centralized logging for the specified table, and then atomically delete all centrally logged table data for this namespace and table name, restricted to the column partition value if provided.

USE WITH EXTREME CAUTION!!!

Parameters:

namespace - table namespace

tableName - table name

columnPartitionValue - optional column partition value

Throws:

IOException - problems encountered

closeAndDeleteCentral

public void closeAndDeleteCentral(@NotNull String namespace, @NotNull String tableName) throws IOException

Description copied from interface: Database

Close all logger connections to the centralized logging for the specified table, and then atomically delete all centrally logged table data for this namespace and table name. Remove all local table data also, as with Database.removeTable(String, String).

USE WITH EXTREME CAUTION!!!

Parameters:

namespace - table namespace

tableName - table name

Throws:

IOException - problems encountered