Class OnDiskQueryDatabase
java.lang.Object
com.illumon.iris.db.tables.databases.BaseDatabase
com.illumon.iris.db.tables.databases.AbstractDatabase
com.illumon.iris.db.tables.databases.OnDiskDatabase
com.illumon.iris.db.tables.databases.OnDiskQueryDatabase
- All Implemented Interfaces:
Database
- Direct Known Subclasses:
ImporterDatabase
Primary local database implementation for query usage.
-
Nested Class Summary
Nested classes/interfaces inherited from class com.illumon.iris.db.tables.databases.OnDiskDatabase
OnDiskDatabase.AuditEventLogDetailsNested classes/interfaces inherited from interface com.illumon.iris.db.tables.databases.Database
Database.LogHandle, Database.StorageFormat -
Field Summary
Fields inherited from class com.illumon.iris.db.tables.databases.OnDiskDatabase
AEL_ADD_TABLE_DEFINITION_STRING, AEL_APPEND_CENTRAL_STRING, AEL_CLOSE_AND_DELETE_CENTRAL_STRING, AEL_CREATE_LOG_HANDLE_STRING, AEL_CREATE_TABLE_WRITER_STRING, AEL_DELETE_NAMESPACE_STRING, AEL_DELETE_TABLE_PARTITION_STRING, AEL_DELETE_TABLE_STRING, AEL_REPLACE_TABLE_STRING, AEL_WRITE_TABLE_STRING, AUDIT_TABLE_ACCESS_EVENT_STRING, auditEventLogDetails, DEFAULT_ROOT_DIRECTORY, locationManager, log, ON_DISK_DATABASE_ROOT_DIRECTORY_PROP, permissionFilterProvider, TABLE_DEFINITIONS_TABLE_DEFINITION, userContext -
Constructor Summary
ConstructorsConstructorDescriptionOnDiskQueryDatabase(com.fishlib.io.logger.Logger log) Deprecated.The tableDataServiceName should be known.OnDiskQueryDatabase(com.fishlib.io.logger.Logger log, com.fishlib.configuration.Configuration configuration, File rootDirectory, TableDataService tableDataService, SchemaService schemaService, NamespaceSet newNamespaceSet) Opens an on-disk database suitable for user queries with the specified root and no support for query parallelization.OnDiskQueryDatabase(com.fishlib.io.logger.Logger log, File rootDirectory, TableDataService tableDataService) Delegate toOnDiskQueryDatabase(Logger, File, TableDataService, SchemaService)with the default SchemaService.OnDiskQueryDatabase(com.fishlib.io.logger.Logger log, File rootDirectory, TableDataService tableDataService, SchemaService schemaService) Delegate toOnDiskQueryDatabase(Logger, File, TableDataService, SchemaService, NamespaceSet)with NamespaceSet User.OnDiskQueryDatabase(com.fishlib.io.logger.Logger log, File rootDirectory, TableDataService tableDataService, SchemaService schemaService, NamespaceSet newNamespaceSet) Opens an on-disk database suitable for user queries with the specified root and no support for query parallelization, using the default Configuration instance.OnDiskQueryDatabase(com.fishlib.io.logger.Logger log, File rootDirectory, String tableDataServiceName) Delegate toOnDiskQueryDatabase(Logger, File, TableDataService)with a non-dynamic TableDataService constructed from the given tableDataServiceName.OnDiskQueryDatabase(com.fishlib.io.logger.Logger log, String tableDataServiceName) Delegate toOnDiskQueryDatabase(Logger, File, TableDataService)with the default rootDirectory. -
Method Summary
Modifier and TypeMethodDescriptionprotected voidappendColumnPartition(TableDefinitionSchema schema, String internalPartitionValue, String columnPartitionValue, Table data) protected voidappendTable(TableDefinitionSchema schema, Table table) protected voidcheckAllowAddColumnPartition(TableDefinitionSchema schema, Table newTable) protected voidcheckAllowAddNamespace(NamespaceSet namespaceSet, String namespace) protected voidprotected voidcheckAllowCleanupNamespace(String namespace) protected voidprotected voidprotected voidcheckAllowRemoveNamespace(String namespace) protected voidprotected voidprotected voidprotected voidcheckAllowRemoveTableDefinition(String namespace, String tableName) protected voidcheckAllowReplaceColumnPartition(TableDefinitionSchema schema, Table newTable) protected voidcheckAllowReplaceTable(TableDefinitionSchema schema, Table newTable) protected voidvoidcloseAndDeleteCentral(String namespace, String tableName) Close all logger connections to the centralized logging for the specified table, and then atomically delete all centrally logged table data for this namespace and table name.voidcloseAndDeleteCentral(String namespace, String tableName, String columnPartitionValue) Close all logger connections to the centralized logging for the specified table, and then atomically delete all centrally logged table data for this namespace and table name, restricted to the column partition value if provided.protected voiddeleteColumnPartition(TableDefinitionSchema schema, String internalPartitionValue, String columnPartitionValue) protected voiddeleteTable(TableDefinitionSchema schema) protected voiddeleteTable(TableDefinitionSchema schema, File path) Delete the table at the given file path.protected TablegetIntradayTableV2(TableDefinitionSchema schema, boolean live) protected TablegetIntradayTableV2(TableDefinitionSchema schema, Set<String> partitionNames, boolean live) getLastByTable(String namespace, String tableName) Retrieves an intraday last-by table by namespace and name.protected booleanhasData(NamespaceSet set, TableDefinition tDef, String columnPartitionValue, boolean intraday) protected voidinstallShadowTable(TableDefinitionSchema schema, File shadowPath) Install the shadow table for the schema at the shadow location.protected FileMove the table to a backup location determined by the schema and time of execution.protected voidoverwriteColumnPartition(TableDefinitionSchema schema, String internalPartitionValue, String columnPartitionValue, Table data, Database.StorageFormat storageFormat) voidReload any internally-cached data.voidremoveTable(String namespace, String tableName) Removes a table.voidremoveTablePartition(String namespace, String tableName, String internalPartitionValue, String columnPartitionValue) Removes a single table location (e.g.voidreplaceTable(String namespace, String tableName, Table table, Database.StorageFormat storageFormat) Replaces a non-partitioned table with a new table, or adds the table, if it is not already present.voidreplaceTablePartition(String namespace, String tableName, String internalPartitionValue, String columnPartitionValue, Table data, Database.StorageFormat storageFormat) Replaces a single table location (e.g.protected TablerestoreTable(TableDefinitionSchema schema, boolean isLive) protected voidrollbackTable(TableDefinitionSchema schema, File backupPath, File shadowPath) Restore the table from the backup location.intsetWatchdogMillis(int millis) Sets monitoring of delays in the LiveTableMonitor refresh loop.voidsetWatchdogTimeoutProcedure(LongConsumer watchdogTimeoutProcedure) Set the watchdog timeout procedure.protected voidstoreColumnPartition(TableDefinitionSchema schema, String internalPartitionValue, String columnPartitionValue, Table data, Database.StorageFormat storageFormat) protected FilestoreShadowTable(TableDefinitionSchema schema, Table table, Database.StorageFormat storageFormat) Stores a copy of a table at a location determined by the schema and time of execution.protected voidstoreTable(TableDefinitionSchema schema, Table table, Database.StorageFormat storageFormat) protected voidverifyNamespace(String namespace, String tableName) Ensure the namespace exists.Methods inherited from class com.illumon.iris.db.tables.databases.OnDiskDatabase
addTable, addTableDefinition, addTablePartition, appendCentral, appendTable, appendTablePartition, checkAllowAddTable, checkAllowAddTableCentral, checkAllowCentralByGroup, deleteBackupTable, deleteMatchingTables, deleteNamespace, deleteTables, getAuditEventLogLogger, getCatalog, getColumnDefinitionsTable, getColumnPartitionTableWriter, getIntradayPartitions, getIntradayPartitionsAsTable, getIntradayTableV2, getIntradayTableV2, getIntradayTableV2, getNamespaces, getPermissionFilterProvider, getRequiredTableDefinitionSchema, getRootDirectory, getSchemaService, getSystemNamespaces, getTableDefinition, getTableDefinitionsTable, getTableNames, getTableV2, getTableWriter, getUserContext, getUserNamespaces, hasImportedData, hasIntradayData, hasIntradayTable, hasNamespace, hasTable, logAuditEvent, logPartitionTableIncremental, logPartitionTableIncremental, logPartitionTableSnapshot, logPartitionTableSnapshot, logTableAccessAuditEvent, removeTableDefinition, repairMetadata, repairMetadata, repairMetadata, replaceTableDefinition, setAuditEventLogger, setUserContext, setUserContextAndFilterProviderMethods inherited from class com.illumon.iris.db.tables.databases.AbstractDatabase
executeQuery, gc, getClientHost, getErrorLog, getProcessInfoId, getReplayer, getReplayer, getServerHost, getWorkerName, importClass, importStatic, logPartitionTableSnapshot, logPartitionTableSnapshot, makeRemote, mr, shutdown, startupMethods inherited from class com.illumon.iris.db.tables.databases.BaseDatabase
addPartitionedTableDefinition, addPartitionedTableDefinition, addPartitionedTableDefinitionIfAbsent, addPartitionedTableDefinitionIfAbsent, addTablePartition, addTablePartitionAndDefinition, addTablePartitionAndDefinition, appendCentral, appendTablePartition, removeTablePartition, replaceTablePartitionMethods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, waitMethods inherited from interface com.illumon.iris.db.tables.databases.Database
addColumnPartitionAndDefinition, addColumnPartitionAndDefinition, addTable, addTablePartition, addTablePartition, getIntradayTable, getIntradayTable, getIntradayTable, getIntradayTable, getIntradayTable, getIntradayTable, getIntradayTableV2, getIntradayTableV2, getIntradayTableV2, getIntradayTableV2, getReplayer, getReplayer, getReplayer, getTable, i, i, i, i, i, i, i2, i2, i2, i2, i2, i2, replaceTable, replaceTablePartition, replaceTablePartition, rm, t, t2, timeTable, timeTable, timeTable, timeTable, timeTable, timeTable, timeTable, timeTable, timeTable, timeTable, timeTable, timeTable
-
Constructor Details
-
OnDiskQueryDatabase
public OnDiskQueryDatabase(@NotNull com.fishlib.io.logger.Logger log, @Nullable File rootDirectory, @NotNull String tableDataServiceName) Delegate toOnDiskQueryDatabase(Logger, File, TableDataService)with a non-dynamic TableDataService constructed from the given tableDataServiceName.- Parameters:
log- the logger to userootDirectory- the root directory for the database's data and metadata storagetableDataServiceName- initialize the TableDataService with this configuration name.
-
OnDiskQueryDatabase
public OnDiskQueryDatabase(@NotNull com.fishlib.io.logger.Logger log, @NotNull File rootDirectory, @NotNull TableDataService tableDataService) Delegate toOnDiskQueryDatabase(Logger, File, TableDataService, SchemaService)with the default SchemaService.- Parameters:
log- the logger to userootDirectory- the root directory for the database's data and metadata storagetableDataService- the constructed TableDataService to use
-
OnDiskQueryDatabase
public OnDiskQueryDatabase(@NotNull com.fishlib.io.logger.Logger log, @NotNull File rootDirectory, @NotNull TableDataService tableDataService, @NotNull SchemaService schemaService) Delegate toOnDiskQueryDatabase(Logger, File, TableDataService, SchemaService, NamespaceSet)with NamespaceSet User.- Parameters:
log- the logger to userootDirectory- the root directory for the database's data and metadata storagetableDataService- the constructed TableDataService to useschemaService- the SchemaService to use
-
OnDiskQueryDatabase
Deprecated.The tableDataServiceName should be known. UseOnDiskQueryDatabase(Logger, File, String)instead.Delegate toOnDiskQueryDatabase(Logger, File, String)with the default table data service configuration name and null root directory, signifying the default.- Parameters:
log- the logger to use
-
OnDiskQueryDatabase
public OnDiskQueryDatabase(@NotNull com.fishlib.io.logger.Logger log, @NotNull String tableDataServiceName) Delegate toOnDiskQueryDatabase(Logger, File, TableDataService)with the default rootDirectory.- Parameters:
log- the logger to usetableDataServiceName- initialize the TableDataService with this configuration name.
-
OnDiskQueryDatabase
public OnDiskQueryDatabase(@NotNull com.fishlib.io.logger.Logger log, @Nullable File rootDirectory, @NotNull TableDataService tableDataService, @NotNull SchemaService schemaService, @NotNull NamespaceSet newNamespaceSet) Opens an on-disk database suitable for user queries with the specified root and no support for query parallelization, using the default Configuration instance.- Parameters:
log- the logger to userootDirectory- the root directory for the database's data and metadata storagetableDataService- the constructed TableDataService to useschemaService- the SchemaService to usenewNamespaceSet- the NamespaceSet to use for newly created namespaces
-
OnDiskQueryDatabase
public OnDiskQueryDatabase(@NotNull com.fishlib.io.logger.Logger log, @NotNull com.fishlib.configuration.Configuration configuration, @Nullable File rootDirectory, @NotNull TableDataService tableDataService, @NotNull SchemaService schemaService, @NotNull NamespaceSet newNamespaceSet) Opens an on-disk database suitable for user queries with the specified root and no support for query parallelization.- Parameters:
log- the logger to useconfiguration- the Configuration instance to userootDirectory- the root directory for the database's data and metadata storagetableDataService- the constructed TableDataService to useschemaService- the SchemaService to usenewNamespaceSet- the NamespaceSet to use for newly created namespaces
-
-
Method Details
-
getTableDataService
-
reloadData
public void reloadData()Reload any internally-cached data.- ImplNote:
- OnDiskQueryDatabase resets its TableDataService.
-
checkAllowAddSchema
- Overrides:
checkAllowAddSchemain classOnDiskDatabase
-
checkAllowAddNamespace
- Overrides:
checkAllowAddNamespacein classOnDiskDatabase
-
checkAllowRemoveNamespace
- Overrides:
checkAllowRemoveNamespacein classOnDiskDatabase
-
storeTable
protected void storeTable(@NotNull TableDefinitionSchema schema, Table table, Database.StorageFormat storageFormat) - Specified by:
storeTablein classOnDiskDatabase
-
appendTable
- Specified by:
appendTablein classOnDiskDatabase
-
checkAllowRemoveTableDefinition
- Overrides:
checkAllowRemoveTableDefinitionin classOnDiskDatabase
-
checkAllowRemoveTableDefinition
- Overrides:
checkAllowRemoveTableDefinitionin classOnDiskDatabase
-
checkAllowRemoveTable
- Overrides:
checkAllowRemoveTablein classOnDiskDatabase
-
checkAllowRemoveTableCentral
- Overrides:
checkAllowRemoveTableCentralin classOnDiskDatabase
-
checkAllowCleanupNamespace
- Overrides:
checkAllowCleanupNamespacein classOnDiskDatabase
-
deleteTable
- Specified by:
deleteTablein classOnDiskDatabase
-
checkAllowReplaceTableDefinition
- Overrides:
checkAllowReplaceTableDefinitionin classOnDiskDatabase
-
checkAllowReplaceTable
- Overrides:
checkAllowReplaceTablein classOnDiskDatabase
-
verifyNamespace
Description copied from class:OnDiskDatabaseEnsure the namespace exists.- Specified by:
verifyNamespacein classOnDiskDatabase- Parameters:
namespace- name of the namespace to checktableName- optional table name for logging context
-
replaceTablePartition
public void replaceTablePartition(String namespace, String tableName, String internalPartitionValue, String columnPartitionValue, Table data, Database.StorageFormat storageFormat) Replaces a single table location (e.g. one date) from/to a partitioned table.Requires that a suitable TableDefinition has already been added.
Not guaranteed to be atomic in the case of failures.
- Specified by:
replaceTablePartitionin interfaceDatabase- Overrides:
replaceTablePartitionin classOnDiskDatabase- Parameters:
namespace- table namespacetableName- table nameinternalPartitionValue- internal (not user-visible) partition value.columnPartitionValue- column partition value (e.g. "2015-09-25" for a Date partitioning column).data- data to replace this partition. Must *not* contain the partitioning column.storageFormat- the storage format to use (ex DeephavenV1 or Parquet)- ImplNote:
- OnDiskQueryDatabase resets its TableDataService for the table.
-
replaceTable
public void replaceTable(String namespace, String tableName, Table table, Database.StorageFormat storageFormat) Replaces a non-partitioned table with a new table, or adds the table, if it is not already present. Not guaranteed to be atomic in the case of failures.- Specified by:
replaceTablein interfaceDatabase- Overrides:
replaceTablein classOnDiskDatabase- Parameters:
namespace- table namespacetableName- table nametable- data for the replacement tablestorageFormat- the storage format to use (ex DeephavenV1 or Parquet)- ImplNote:
- OnDiskQueryDatabase resets its TableDataService for the table.
-
removeTablePartition
public void removeTablePartition(String namespace, String tableName, String internalPartitionValue, String columnPartitionValue) Removes a single table location (e.g. one date) from a partitioned table. Requires that a suitable TableDefinition still exists.- Specified by:
removeTablePartitionin interfaceDatabase- Overrides:
removeTablePartitionin classOnDiskDatabase- Parameters:
namespace- table namespacetableName- table nameinternalPartitionValue- internal (not user-visible) partition value.columnPartitionValue- column partition value (e.g. "2015-09-25" for a Date partitioning column).- ImplNote:
- OnDiskQueryDatabase resets its TableDataService for the table.
-
removeTable
Removes a table. This method is a better choice thanDatabase.deleteTables(String, String...), unless the underlying table no longer has consistent metadata. Throws an exception for tables that don't exist.- Specified by:
removeTablein interfaceDatabase- Overrides:
removeTablein classOnDiskDatabase- Parameters:
namespace- table namespacetableName- table name- ImplNote:
- OnDiskQueryDatabase resets its TableDataService for the table.
-
storeShadowTable
protected File storeShadowTable(@NotNull TableDefinitionSchema schema, @NotNull Table table, @NotNull Database.StorageFormat storageFormat) Description copied from class:OnDiskDatabaseStores a copy of a table at a location determined by the schema and time of execution. If an exception is thrown while writing the copy, the shadow location is deleted.- Specified by:
storeShadowTablein classOnDiskDatabase- Parameters:
schema- The schema for the tabletable- The tablestorageFormat- The storage format- Returns:
- The directory of the table's copy if successfully stored, else an UncheckedDeephavenException is thrown
-
moveTableToBackup
Description copied from class:OnDiskDatabaseMove the table to a backup location determined by the schema and time of execution.- Specified by:
moveTableToBackupin classOnDiskDatabase- Parameters:
schema- The schema for the table- Returns:
- The directory to where the table was successfully moved, else an UncheckedDeephavenException is thrown
-
installShadowTable
Description copied from class:OnDiskDatabaseInstall the shadow table for the schema at the shadow location.- Specified by:
installShadowTablein classOnDiskDatabase- Parameters:
schema- The schema for the tableshadowPath- The directory where the shadow table is stored
-
rollbackTable
protected void rollbackTable(@NotNull TableDefinitionSchema schema, @NotNull File backupPath, @NotNull File shadowPath) Description copied from class:OnDiskDatabaseRestore the table from the backup location. The table will be moved to the shadow location before the backup is restored. Nothing is deleted.- Specified by:
rollbackTablein classOnDiskDatabase- Parameters:
schema- The schema for the tablebackupPath- The directory where the backup table was storedshadowPath- The directory where the table will be moved to
-
deleteTable
Description copied from class:OnDiskDatabaseDelete the table at the given file path.- Specified by:
deleteTablein classOnDiskDatabase- Parameters:
schema- The schema for the table. Used only for logging.path- The directory where the table is stored
-
restoreTable
- Specified by:
restoreTablein classOnDiskDatabase
-
checkAllowAddColumnPartition
- Overrides:
checkAllowAddColumnPartitionin classOnDiskDatabase
-
checkAllowDeleteColumnPartition
- Overrides:
checkAllowDeleteColumnPartitionin classOnDiskDatabase
-
checkAllowReplaceColumnPartition
- Overrides:
checkAllowReplaceColumnPartitionin classOnDiskDatabase
-
checkAllowIntradayOperation
- Overrides:
checkAllowIntradayOperationin classOnDiskDatabase
-
storeColumnPartition
protected void storeColumnPartition(TableDefinitionSchema schema, String internalPartitionValue, String columnPartitionValue, Table data, Database.StorageFormat storageFormat) - Specified by:
storeColumnPartitionin classOnDiskDatabase
-
appendColumnPartition
protected void appendColumnPartition(TableDefinitionSchema schema, String internalPartitionValue, String columnPartitionValue, Table data) - Specified by:
appendColumnPartitionin classOnDiskDatabase
-
deleteColumnPartition
protected void deleteColumnPartition(TableDefinitionSchema schema, String internalPartitionValue, String columnPartitionValue) - Specified by:
deleteColumnPartitionin classOnDiskDatabase
-
overwriteColumnPartition
protected void overwriteColumnPartition(TableDefinitionSchema schema, String internalPartitionValue, String columnPartitionValue, Table data, Database.StorageFormat storageFormat) - Specified by:
overwriteColumnPartitionin classOnDiskDatabase
-
getIntradayPartitions
- Specified by:
getIntradayPartitionsin classOnDiskDatabase
-
getIntradayTableV2
- Specified by:
getIntradayTableV2in classOnDiskDatabase
-
getIntradayTableV2
protected Table getIntradayTableV2(TableDefinitionSchema schema, Set<String> partitionNames, boolean live) - Specified by:
getIntradayTableV2in classOnDiskDatabase
-
hasData
protected boolean hasData(NamespaceSet set, TableDefinition tDef, @Nullable String columnPartitionValue, boolean intraday) - Specified by:
hasDatain classOnDiskDatabasetDef- the definition of the tablecolumnPartitionValue- partitioning column value to check. Null indicates "any" value.intraday- if false, only historical partitions are checked.- Returns:
- True if the specified table location has data.
-
getLastByTable
Description copied from interface:DatabaseRetrieves an intraday last-by table by namespace and name.Throws exceptions for unknown tables.
Always provides a live result.
This may not be configured on your system.
- Parameters:
namespace- namespace of the tabletableName- name of the table- Returns:
- live intraday last-by table
-
setWatchdogMillis
public int setWatchdogMillis(int millis) Description copied from interface:DatabaseSets monitoring of delays in the LiveTableMonitor refresh loop. If the LiveTableMonitor refresh loop takes longer than the specified amount of time, the watchdog timeout procedure is executed. The watchdog timeout procedure is specified by callingDatabase.setWatchdogTimeoutProcedure(LongConsumer).- Parameters:
millis- if LiveTableMonitor refresh loop takes longer than this amount of time, the watchdog timeout procedure will be called.- Returns:
- the new value of the watchdog timer, in ms
-
setWatchdogTimeoutProcedure
Description copied from interface:DatabaseSet the watchdog timeout procedure. If LiveTableMonitor refresh loop takes longer than the amount of time specified viaDatabase.setWatchdogMillis(int), the timeout procedure is called.- Parameters:
watchdogTimeoutProcedure- the procedure called when the LiveTableMonitor refresh loop takes longer than watchdogMillis.
-
closeAndDeleteCentral
public void closeAndDeleteCentral(@NotNull String namespace, @NotNull String tableName, @Nullable String columnPartitionValue) throws IOException Description copied from interface:DatabaseClose all logger connections to the centralized logging for the specified table, and then atomically delete all centrally logged table data for this namespace and table name, restricted to the column partition value if provided. Remove all local table data also, as withDatabase.removeTable(String, String).USE WITH EXTREME CAUTION!!!
- Parameters:
namespace- table namespacetableName- table namecolumnPartitionValue- optional column partition value- Throws:
IOException- problems encountered
-
closeAndDeleteCentral
public void closeAndDeleteCentral(@NotNull String namespace, @NotNull String tableName) throws IOException Description copied from interface:DatabaseClose all logger connections to the centralized logging for the specified table, and then atomically delete all centrally logged table data for this namespace and table name. Remove all local table data also, as withDatabase.removeTable(String, String).USE WITH EXTREME CAUTION!!!
- Parameters:
namespace- table namespacetableName- table name- Throws:
IOException- problems encountered
-