Package io.deephaven.enterprise.ingester.impl

Class OffsetUtils

java.lang.Object
io.deephaven.enterprise.ingester.impl.OffsetUtils
public class OffsetUtils extends Object
Tools for reading offsets from existing partitions for use with in an Ingester.

Note: This class is not part of the public Deephaven Core+ API.

  • Constructor Summary

    Constructors
    Constructor
    Description
    OffsetUtils()
     

  • Method Summary

    Modifier and Type
    Method
    Description
    static long
    getLastOffsetFromExistingColumnPartitions(@NotNull io.deephaven.shadow.enterprise.com.illumon.iris.db.v2.tabledataservice.TableDataService tableDataService, @NotNull String namespace, @NotNull String tableName, @NotNull String internalPartition, @NotNull io.deephaven.shadow.enterprise.io.deephaven.csv.util.MutableObject<String> columnPartitionOut)
    For dynamic partitions, we read all existing partitions for this namespace, table and column partition.
    static @NotNull OptionalLong
    readOffsetFromSourceRecord(@NotNull Logger logger, @NotNull io.deephaven.shadow.enterprise.com.illumon.iris.db.v2.tabledataservice.TableDataService tableDataService, @NotNull io.deephaven.shadow.enterprise.com.illumon.iris.db.v2.locations.FullTableLocationKey streamKey, @NotNull String priorColumnPartition, @NotNull String expectedName)
    Given the key of a stream, and the prior column partition for this stream, read the expected offset that should be used to resume the stream.

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

  • Constructor Details

    • OffsetUtils

      public OffsetUtils()

  • Method Details

    • getLastOffsetFromExistingColumnPartitions

      public static long getLastOffsetFromExistingColumnPartitions(@NotNull @NotNull io.deephaven.shadow.enterprise.com.illumon.iris.db.v2.tabledataservice.TableDataService tableDataService, @NotNull @NotNull String namespace, @NotNull @NotNull String tableName, @NotNull @NotNull String internalPartition, @NotNull @NotNull io.deephaven.shadow.enterprise.io.deephaven.csv.util.MutableObject<String> columnPartitionOut)
      For dynamic partitions, we read all existing partitions for this namespace, table and column partition.

      We sort them lexicographically, and then read the final checkpoint record to determine where we must resume reading from the broker. If no partitions are found, we return -1 and the fallback methods are used.

      Parameters:
      tableDataService - the table data service for reading the offset
      internalPartition - the internal partition to ingest
      columnPartitionOut - the column partition that we used for resuming (output only)
      Returns:
      the offset to resume from, or -1 if no offset was found

    • readOffsetFromSourceRecord

      @NotNull public static @NotNull OptionalLong readOffsetFromSourceRecord(@NotNull @NotNull Logger logger, @NotNull @NotNull io.deephaven.shadow.enterprise.com.illumon.iris.db.v2.tabledataservice.TableDataService tableDataService, @NotNull @NotNull io.deephaven.shadow.enterprise.com.illumon.iris.db.v2.locations.FullTableLocationKey streamKey, @NotNull @NotNull String priorColumnPartition, @NotNull @NotNull String expectedName)
      Given the key of a stream, and the prior column partition for this stream, read the expected offset that should be used to resume the stream.
      Parameters:
      logger - the logger for info messages describing where the offset was computed from
      tableDataService - the service for reading the prior offset
      streamKey - the key of the new stream
      priorColumnPartition - the column partition value we are reading from
      expectedName - the expected name in the io.deephaven.shadow.enterprise.com.illumon.iris.db.v2.locations.local.CheckpointRecord.SourceFileSizeRecord