Package io.deephaven.engine.table.impl.util

Class TailInitializationFilter

java.lang.Object
io.deephaven.engine.table.impl.util.TailInitializationFilter
public class TailInitializationFilter extends Object
When a query processing a persistent append-only table is restarted mid-day, we often know that all data of interest must take place within a fixed period of time. Rather than processing all the data, we can binary search in each partition to find the relevant rows based on a Timestamp.

This is only designed to operate against a source table, which must be add-only. If the Timestamp source is a RegionedColumnSource, then each region is assumed to be a partition. Otherwise, each contiguous range of row keys is assumed to be a partition.

If you alter the source table before calling TailInitializationFilter, these assumptions may be violated and the resulting table will not be filtered as desired.

Once initialized, the filter returns all new rows, rows that have already been passed are not removed or modified.

Each partition in the input must be sorted by Timestamp, or the resulting table is undefined. Null timestamps are not permitted.

For consistency, the last value of each partition is used to determine the threshold for that partition. As the partitions must be sorted, the last row in each partition must be the most recent timestamp within that partition. The provided period is subtracted from the timestamp; and a binary search is performed to identify rows that are within the window defined by period.

  • Constructor Details

    • TailInitializationFilter

      public TailInitializationFilter()

  • Method Details

    • mostRecent

      public static Table mostRecent(Table table, String timestampName, String period)
      Get the most recent rows from each partition in source table.
      Parameters:
      table - the source table to filter
      timestampName - the name of the timestamp column
      period - interval between the last row in a partition (as converted by DateTimeUtils.parseDurationNanos(String)) and rows that match the filter
      Returns:
      a table with only the most recent values in each partition

    • mostRecent

      public static Table mostRecent(Table table, String timestampName, long nanos)
      Get the most recent rows from each partition in source table.
      Parameters:
      table - the source table to filter
      timestampName - the name of the timestamp column
      nanos - interval between the last row in a partition, in nanoseconds and rows that match the filter
      Returns:
      a table with only the most recent values in each partition

    • mostRecentRows

      public static Table mostRecentRows(Table table, long rowCount)
      Get the most recent rows from each partition in source table.
      Parameters:
      table - the source table to filter
      rowCount - the number of rows to include per partition
      Returns:
      a table with only the most recent values in each partition