Package com.illumon.iris.db.v2.locations.local

Class LocalColumnLocation

java.lang.Object
com.illumon.iris.db.v2.locations.AbstractColumnLocation<LocalTableLocation>
com.illumon.iris.db.v2.locations.local.LocalColumnLocation
All Implemented Interfaces:
com.fishlib.base.log.LogOutputAppendable, StringUtils.StringKeyedObject, ColumnLocation, DeephavenFormatColumnLocation, NamedImplementation
public class LocalColumnLocation extends AbstractColumnLocation<LocalTableLocation> implements DeephavenFormatColumnLocation

Implements filesystem layout for Iris DB filesystem-backed columns, relative to a LocalTableLocation.

Header format explanations: (all sizes/offsets are in bytes)

Variables:

  • m = serialized metadata object size
  • d = data start position
  • m_h = min(d - 8, m) = size of the serialized metadata object prefix stored in the primary file's header region
  • m_o = m - m_h = size of the serialized metadata object suffix stored in an overflow file (usually 0)
  • e = end of file

Format for data and symbol offset files (by byte range):

  • [0 - 3]: serialized metadata object size (m)
  • [4 - 7]: data start position (d)
  • [8 - m_h): serialized metadata object prefix
  • [m_h - d): unused space
  • [d - e): binary column data

Note that m and d are written in big-endian byte order. The high bit of m denotes the byte order of all subsequent data in the file, with 0 denoting big-endian byte order and 1 denoting little-endian byte order. The higher 14 bits of d should also be considered reserved - the current maximum value used is 128K (1 << 17).

Typically, the metadata object for a data file is just the partitionKeysToRange map (a Map<TYPE, int[]>) or null if the column location is not grouped. For a symbol offset file, the metadata object is typically null.

Legacy column locations may have data and symbol offset files that instead contain an instance of a DiskMappedDataColumn subclass, which in turn may contain a valid partitionKeysToRange map. ColumnFileHeaderObjectInputStream handles translation in this case.

  • Constructor Details

    • LocalColumnLocation

      public LocalColumnLocation(@NotNull LocalTableLocation tableLocation, @NotNull String name)
      Construct a new LocalColumnLocation for the specified LocalTableLocation and column name.
      Parameters:
      tableLocation - The table location enclosing this column location
      name - The name of the column

  • Method Details

    • hasHeader

      public static boolean hasHeader(@Nullable ColumnFileType type, int columnVersion)

    • detailsFor

      public static ColumnFileTypeDetails detailsFor(@NotNull ColumnFileType type)

    • encodeHeaderPrefix

      @InternalUseOnly public static long encodeHeaderPrefix(@NotNull ByteOrder byteOrder, int metadataSize, int dataStartPosition)
      Encode the two fields in the header prefix into a single long.
      Parameters:
      byteOrder - The byte order
      metadataSize - The metadata size
      dataStartPosition - The data start position
      Returns:
      The header prefix as a long

    • decodeByteOrder

      @InternalUseOnly @NotNull public static ByteOrder decodeByteOrder(long encodedHeaderPrefix)
      Decode the byte order from a header prefix (represented as a long).
      Parameters:
      encodedHeaderPrefix - The header prefix
      Returns:
      The byte order

    • decodeMetadataSize

      @InternalUseOnly public static int decodeMetadataSize(long encodedHeaderPrefix)
      Decode the metadata size from a header prefix (represented as a long).
      Parameters:
      encodedHeaderPrefix - The header prefix
      Returns:
      The metadata size

    • decodeDataStartPosition

      @InternalUseOnly public static int decodeDataStartPosition(long encodedHeaderPrefix)
      Decode the data start position from a header prefix (represented as a long).
      Parameters:
      encodedHeaderPrefix - The header prefix
      Returns:
      The data start position

    • exists

      public boolean exists()
      Description copied from interface: ColumnLocation
      Check for existence of this ColumnLocation.
      Specified by:
      exists in interface ColumnLocation
      Returns:
      True iff the ColumnLocation actually exists

    • getByteOrder

      public final ByteOrder getByteOrder()
      Description copied from interface: DeephavenFormatColumnLocation
      Get the ByteOrder used by all FileDataBufferStores associated with this column location. Invoking this method is an error for any column location if ColumnLocation.exists() has not been invoked and returned true.
      Specified by:
      getByteOrder in interface DeephavenFormatColumnLocation
      Returns:
      The ByteOrder

    • determineHeaderByteOrder

      public final ByteOrder determineHeaderByteOrder()
      Read the ByteOrder for this location from the file header. This is only valid for V1 column formats
      Returns:
      the byte order

    • getMetadata

      @Nullable public <METADATA_TYPE> METADATA_TYPE getMetadata()

    • getMetadata

      public static <METADATA_TYPE> METADATA_TYPE getMetadata(@NotNull String description, int columnVersion, @NotNull Function<ColumnFileTypeDetails,ColumnFileState> stateLookup)

    • getColumnFileDataStartPosition

      public int getColumnFileDataStartPosition(@NotNull ColumnFileType columnFileType)

    • initializeColumnFile

      public void initializeColumnFile(@NotNull ColumnFileType columnFileType)

    • getColumnFileName

      @NotNull public String getColumnFileName(@NotNull ColumnFileType columnFileType)

    • getColumnFile

      @NotNull public File getColumnFile(@NotNull ColumnFileType columnFileType)

    • getColumnFileDataAccessor

      @NotNull public FileAccessor getColumnFileDataAccessor(@NotNull ColumnFileType columnFileType)

    • getColumnFileBufferStore

      @NotNull public FileDataBufferStore getColumnFileBufferStore(@NotNull ColumnFileType columnFileType)
      Description copied from interface: DeephavenFormatColumnLocation
      Get the FileDataBufferStore for buffers from this column location for the specified ColumnFileType.
      Specified by:
      getColumnFileBufferStore in interface DeephavenFormatColumnLocation
      Parameters:
      columnFileType - The ColumnFileType
      Returns:
      The buffer store for the specified column file type for this column location

    • symbolFileExists

      public boolean symbolFileExists()
      Description copied from interface: DeephavenFormatColumnLocation
      Check if the symbol table file exists.
      Specified by:
      symbolFileExists in interface DeephavenFormatColumnLocation
      Returns:
      true if the symbol data file exists.