Package com.illumon.iris.db.v2.locations.local

Class CheckpointRecord

java.lang.Object
com.illumon.iris.db.v2.locations.TableLocationStateHolder
com.illumon.iris.db.v2.locations.local.CheckpointRecord
All Implemented Interfaces:
com.fishlib.base.log.LogOutputAppendable, TableLocationState, NamedImplementation
Direct Known Subclasses:
CheckpointRecord.Reloadable
public class CheckpointRecord extends TableLocationStateHolder implements com.fishlib.base.log.LogOutputAppendable, NamedImplementation

Checkpoint record to group information for a table writer to safely resume after a process or system crash. Concurrent accesses should be externally synchronized.

Currently stores:

  1. The table location's state (see TableLocationState)
  2. The corresponding size (in bytes) of the data region of all data files, e.g. column .dat, .sym, .bytes, and .sym.bytes files
  3. Current source file name and position (optional)
  4. Import state (optional)

  • Field Details

    • NEW_COLUMN_BYTE_ORDER

      public static final ByteOrder NEW_COLUMN_BYTE_ORDER

    • NULL_VERSION

      public static final int NULL_VERSION
      Null version number/
      See Also:

    • VERSION_0_TABLE_LOCATION_SIZE_ONLY

      public static final int VERSION_0_TABLE_LOCATION_SIZE_ONLY
      Version "0" only had an int table location size, and no other metadata.
      See Also:

    • VERSION_1_ATOMIC_STATE

      public static final int VERSION_1_ATOMIC_STATE
      Version "1" has modification time, data file sizes, source file state, and import state, all updated atomically.
      See Also:

    • VERSION_2_NULL_SIZE_ON_TRUNCATE

      public static final int VERSION_2_NULL_SIZE_ON_TRUNCATE
      Version "2" uses a null size when truncating a location via data-driven command.
      See Also:

    • VERSION_3_TRUNCATE_MARKER

      public static final int VERSION_3_TRUNCATE_MARKER
      Version "3" adds a "truncated location" marker.
      See Also:

    • VERSION_4_GROUPING_TABLES

      public static final int VERSION_4_GROUPING_TABLES
      Version "4" eliminates the 128k header bytes in column files and writes Grouping / Indexes as tables adjacent to the historical tables.
      See Also:

    • VERSION_5_DATA_INDEX

      public static final int VERSION_5_DATA_INDEX
      Version "5" adds the concept of a Data Index, which can be thought of as traditional database indexes. Unlike Groupings they are not necessarily contiguous.
      See Also:

    • CURRENT_VERSION

      public static final int CURRENT_VERSION
      The current version that will be output.
      See Also:

    • COLUMN_FORMAT_1_HEADERS

      public static final int COLUMN_FORMAT_1_HEADERS
      Version 1 columns contain a 128KB header on .dat and .sym files, and stored grouping as serialized hashmaps in the metadata of .dat files.
      See Also:

    • COLUMN_FORMAT_2_HEADERLESS

      public static final int COLUMN_FORMAT_2_HEADERLESS
      Version 2 eliminates the column file headers and stores groupings as adjacent tables per column
      See Also:

    • CURRENT_COLUMN_FORMAT

      public static final int CURRENT_COLUMN_FORMAT
      The current column format version for newly written tables
      See Also:

    • dataFileSizeRecords

      protected final io.deephaven.hash.KeyedObjectHashMap<String,CheckpointRecord.DataFileSizeRecord> dataFileSizeRecords
      See getDataFileSizeRecord(String).

  • Constructor Details

    • CheckpointRecord

      public CheckpointRecord()
      Construct a new, cleared checkpoint record.

  • Method Details

    • getVersion

      public int getVersion()
      Get the version this checkpoint record was written with, or NULL_VERSION if not read since the checkpoint record was last cleared.
      Returns:
      The version

    • setVersion

      public void setVersion(int newVersion)
      Set the version field in this checkpoint record. The checkpoint record is always persisted with the current version, so this is useful only to track that a checkpoint record has been upgraded after reading.

    • hasCurrentVersion

      public boolean hasCurrentVersion()

      Determine whether the version this checkpoint record was written is the current version. Non-current records may require updating before data can be read or appended to.

      Should only be called on a record that was just re-read from persistent storage.

      Returns:
      Whether this record was written with the current version

    • hasNullVersion

      public boolean hasNullVersion()

      Determine whether this checkpoint record has a null version. Records with a null version are generally empty and should be treated as non-existent for purposes of appending or reading.

      Should only be called on a record that was just re-read from persistent storage.

      Returns:
      Whether this record has the null version

    • getFlags

      public EnumSet<CheckpointRecord.Flag> getFlags()
      return the flags set on this checkpoint record as a set.
      Returns:
      the set of flags set.

    • setFlags

      public void setFlags(EnumSet<CheckpointRecord.Flag> flags)
      Set the flags on this checkpoint record. The new set replaces any existing flags.

    • clearFlags

      public void clearFlags()
      Clear all flags on this checkpoint record.

    • getTableLocationState

      @NotNull public TableLocationState getTableLocationState()
      Get the table location state, or null if not set or read since the checkpoint record was last cleared.
      Returns:
      The table location state

    • setTableLocationState

      public void setTableLocationState(@NotNull TableLocationState tableLocationState)
      Set the table location state.
      Parameters:
      tableLocationState - The table location state

    • getDataFileSizeRecord

      public CheckpointRecord.DataFileSizeRecord getDataFileSizeRecord(@NotNull String name)
      Get the file size record for a given file name, or null if not put or read since the checkpoint record was last cleared.
      Parameters:
      name - The file name
      Returns:
      The data file size record

    • putDataFileSizeRecord

      public void putDataFileSizeRecord(@NotNull CheckpointRecord.DataFileSizeRecord dataFileSizeRecord)
      Put a file size record.
      Parameters:
      dataFileSizeRecord - The data file size record

    • hasColumnData

      public boolean hasColumnData()
      Test whether this record contains any column data.
      Returns:
      True if this record contains any column data, i.e., if it has at least one column data file with non-zero size

    • getSourceFileSizeRecord

      public CheckpointRecord.SourceFileSizeRecord getSourceFileSizeRecord()
      Get the file size record for the source file, or null if not set or read since the checkpoint record was last cleared.
      Returns:
      The source file size record

    • setSourceFileSizeRecord

      public void setSourceFileSizeRecord(@NotNull CheckpointRecord.SourceFileSizeRecord sourceFileSizeRecord)
      Set the source file size record.
      Parameters:
      sourceFileSizeRecord - The source file size record

    • getImportState

      @Nullable public ImportState getImportState()
      Get the import state, or null if not set or read since the checkpoint record was last cleared.
      Returns:
      The import state

    • setImportState

      public void setImportState(ImportState importState)
      Set the import state.
      Parameters:
      importState - The import state

    • getImportDetails

      public Properties getImportDetails()
      Get the import details derived from the import state, or null if the import state is null.
      Returns:
      The import details

    • getByteOrder

      public ByteOrder getByteOrder()
      Get the byte order of the data stored in this location.
      Returns:
      the byte order

    • setByteOrder

      public void setByteOrder(@NotNull ByteOrder order)
      Set the byte order for data that will be stored in this location.
      Parameters:
      order - the new byte order

    • getColumnVersion

      public int getColumnVersion()
      Get the column version for the table at this location.
      Returns:
      the column format version for this location

    • setColumnVersion

      public void setColumnVersion(int columnVersion)
      Set the column version.
      Parameters:
      columnVersion - the column version
      Throws:
      TableDataException - if the column version is not valid for the current checkpoint version

    • setSortedColumns

      public void setSortedColumns(@Nullable List<SortPair> sortedColumns)
      Set the sorted columns for this table location. The input is copied internally and is expected to be in the same order in which the columns are sorted.
      Parameters:
      sortedColumns - the set of sorted columns

    • getSortedColumns

      @NotNull public List<SortPair> getSortedColumns()
      Get the ordered list of sorted columns for this location.
      Returns:
      An unmodifiable list of the sorted columns.

    • setDataIndexes

      public void setDataIndexes(List<String[]> dataIndexes)

    • getDataIndexes

      public List<String[]> getDataIndexes()
      Get the data indexes for this location.
      Returns:
      An unmodifiable list of data indexes, each represented as an array of column names.

    • clear

      public void clear()
      Clear the checkpoint record, forgetting any externally set/put data providers.

    • clearDataRecords

      protected void clearDataRecords()
      Clear the dataFileSizeRecords, allowing override by subclasses.

    • getImplementationName

      public String getImplementationName()
      Description copied from interface: NamedImplementation

      Get a name for the implementing class. Useful for abstract override Object.toString().

      The default implementation is correct, but not suitable for high-frequency usage.

      Specified by:
      getImplementationName in interface NamedImplementation
      Returns:
      A name for the implementing class

    • append

      public com.fishlib.base.log.LogOutput append(@NotNull com.fishlib.base.log.LogOutput logOutput)
      Specified by:
      append in interface com.fishlib.base.log.LogOutputAppendable

    • toString

      public String toString()
      Overrides:
      toString in class Object

    • writeRecord

      public void writeRecord(@NotNull DataOutputStream out) throws IOException
      Write all fields out from this record to a DataOutputStream. This is only called by importer applications.
      Parameters:
      out - The output stream
      Throws:
      IOException

    • readDataFileState

      protected void readDataFileState(@NotNull DataInputStream in) throws IOException
      Read data file state fields in to this record from a DataInputStream.
      Parameters:
      in - The input stream
      Throws:
      IOException

    • readRecordForSerialConsumers

      public void readRecordForSerialConsumers(@NotNull DataInputStream in) throws IOException

      Read all fields of interest to serial consumer applications (i.e., those that don't concurrently import data) into this record from a DataInputStream.

      It's expected that this will be called on a cleared record.

      Parameters:
      in - The input stream
      Throws:
      IOException

    • readRecordForSerialConsumersWithDataRecords

      public void readRecordForSerialConsumersWithDataRecords(@NotNull DataInputStream in) throws IOException

      Read all fields of interest to serial consumer applications (i.e., those that don't concurrently import data) into this record from a DataInputStream, including data file states.

      It's NOT expected that this will be called on a cleared record.

      Parameters:
      in - The input stream
      Throws:
      IOException

    • readRecordForConcurrentConsumers

      public void readRecordForConcurrentConsumers(@NotNull DataInputStream in) throws IOException

      Read all fields of interest to concurrent consumer applications into this record from a DataInputStream.

      It's expected that this will be called on a cleared record.

      Parameters:
      in - The input stream
      Throws:
      IOException

    • readRecordForImporters

      public void readRecordForImporters(@NotNull DataInputStream in) throws IOException, ClassNotFoundException

      Read all fields of interest to importers (i.e., all fields) into this record from a DataInputStream.

      It's expected that this will be called on a cleared record.

      Parameters:
      in - The input stream
      Throws:
      IOException
      ClassNotFoundException

    • main

      public static void main(@NotNull String... args)
      Main - print the specified checkpoint record files to standard out.
      Parameters:
      args - Array of checkpoint record directories or file names