Package io.deephaven.engine.table.impl

Interface PushdownFilterMatcher

All Known Subinterfaces:
ColumnRegion<ATTR>, ColumnRegionByte<ATTR>, ColumnRegionChar<ATTR>, ColumnRegionDouble<ATTR>, ColumnRegionFloat<ATTR>, ColumnRegionInt<ATTR>, ColumnRegionLong<ATTR>, ColumnRegionObject<DATA_TYPE,ATTR>, ColumnRegionObject.SelfDictionaryRegion<DATA_TYPE,ATTR>, ColumnRegionReferencing<ATTR,REFERENCED_COLUMN_REGION>, ColumnRegionShort<ATTR>, PushdownPredicateManager, RegionedPushdownFilterMatcher, TableLocation
All Known Implementing Classes:
AbstractArrowColumnSource, AbstractColumnSource, AbstractColumnSource.DefaultedImmutable, AbstractColumnSource.DefaultedMutable, AbstractTableLocation, AlternatingColumnSource, AppendOnlyFixedSizePageRegionByte, AppendOnlyFixedSizePageRegionChar, AppendOnlyFixedSizePageRegionDouble, AppendOnlyFixedSizePageRegionFloat, AppendOnlyFixedSizePageRegionInt, AppendOnlyFixedSizePageRegionLong, AppendOnlyFixedSizePageRegionObject, AppendOnlyFixedSizePageRegionShort, ArrayBackedColumnSource, ArrowBooleanColumnSource, ArrowByteColumnSource, ArrowCharColumnSource, ArrowDoubleColumnSource, ArrowFloatColumnSource, ArrowInstantColumnSource, ArrowIntColumnSource, ArrowLocalTimeColumnSource, ArrowLongColumnSource, ArrowObjectColumnSource, ArrowShortColumnSource, ArrowStringColumnSource, ArrowUInt1ColumnSource, ArrowUInt4ColumnSource, ArrowUInt8ColumnSource, BaseAggregateSlicedColumnSource, BitMaskingColumnSource, BitShiftingColumnSource, BooleanArraySource, BooleanAsByteColumnSource, BooleanSingleValueSource, BooleanSparseArraySource, BooleanSparseArraySource.ReinterpretedAsByte, ByteAggregateColumnSource, ByteArraySource, ByteAsBooleanColumnSource, ByteChunkColumnSource, ByteSingleValueSource, ByteSparseArraySource, ByteSsmBackedSource, CharacterArraySource, CharacterSingleValueSource, CharacterSparseArraySource, CharAggregateColumnSource, CharChunkColumnSource, CharSsmBackedSource, ColumnRegion.Null, ColumnRegionByte.Constant, ColumnRegionByte.Null, ColumnRegionByte.StaticPageStore, ColumnRegionChar.Constant, ColumnRegionChar.Null, ColumnRegionChar.StaticPageStore, ColumnRegionChunkDictionary, ColumnRegionDouble.Constant, ColumnRegionDouble.Null, ColumnRegionDouble.StaticPageStore, ColumnRegionFloat.Constant, ColumnRegionFloat.Null, ColumnRegionFloat.StaticPageStore, ColumnRegionInt.Constant, ColumnRegionInt.Null, ColumnRegionInt.StaticPageStore, ColumnRegionLong.Constant, ColumnRegionLong.Null, ColumnRegionLong.StaticPageStore, ColumnRegionObject.Constant, ColumnRegionObject.DictionaryKeysWrapper, ColumnRegionObject.Null, ColumnRegionObject.StaticPageStore, ColumnRegionReferencing.Null, ColumnRegionReferencingImpl, ColumnRegionShort.Constant, ColumnRegionShort.Null, ColumnRegionShort.StaticPageStore, CrossJoinRightColumnSource, DataIndexPushdownManager, DeferredColumnRegionBase, DeferredColumnRegionChar, DeferredColumnRegionDouble, DeferredColumnRegionFloat, DeferredColumnRegionInt, DeferredColumnRegionLong, DeferredColumnRegionObject, DeferredColumnRegionReferencing, DeferredColumnRegionShort, DelegatingColumnSource, DeltaAwareColumnSource, DoubleAggregateColumnSource, DoubleArraySource, DoubleChunkColumnSource, DoubleNullToZeroColumnSource, DoubleSingleValueSource, DoubleSparseArraySource, DoubleSsmBackedSource, FloatAggregateColumnSource, FloatArraySource, FloatChunkColumnSource, FloatSingleValueSource, FloatSparseArraySource, FloatSsmBackedSource, GenericColumnRegionBase, HashTableColumnSource, Immutable2DByteArraySource, Immutable2DCharArraySource, Immutable2DDoubleArraySource, Immutable2DFloatArraySource, Immutable2DInstantArraySource, Immutable2DIntArraySource, Immutable2DLongArraySource, Immutable2DNanosBasedTimeArraySource, Immutable2DObjectArraySource, Immutable2DShortArraySource, Immutable2DZonedDateTimeArraySource, ImmutableByteArraySource, ImmutableCharArraySource, ImmutableConstantByteSource, ImmutableConstantCharSource, ImmutableConstantDoubleSource, ImmutableConstantFloatSource, ImmutableConstantInstantSource, ImmutableConstantIntSource, ImmutableConstantLongSource, ImmutableConstantNanosBasedTimeSource, ImmutableConstantObjectSource, ImmutableConstantShortSource, ImmutableConstantZonedDateTimeSource, ImmutableDoubleArraySource, ImmutableFloatArraySource, ImmutableInstantArraySource, ImmutableIntArraySource, ImmutableLongArraySource, ImmutableNanosBasedTimeArraySource, ImmutableObjectArraySource, ImmutableShortArraySource, ImmutableZonedDateTimeArraySource, InstantArraySource, InstantAsLongColumnSource, InstantSparseArraySource, InstantSsmSourceWrapper, IntAggregateColumnSource, IntChunkColumnSource, IntegerArraySource, IntegerSingleValueSource, IntegerSparseArraySource, IntSsmBackedSource, LocalDateWrapperSource, LocalTimeWrapperSource, LongAggregateColumnSource, LongArraySource, LongAsInstantColumnSource, LongAsLocalDateColumnSource, LongAsLocalTimeColumnSource, LongAsTimeSource, LongAsZonedDateTimeColumnSource, LongChunkColumnSource, LongSingleValueSource, LongSparseArraySource, LongSsmBackedSource, MergeSortedHelper.SortedMergeColumnSource, NanosBasedTimeArraySource, NanosBasedTimeSparseArraySource, NonexistentTableLocation, NullValueColumnSource, ObjectAggregateColumnSource, ObjectArraySource, ObjectChunkColumnSource, ObjectSingleValueSource, ObjectSparseArraySource, ObjectSsmBackedSource, ParquetColumnRegionBase, ParquetColumnRegionByte, ParquetColumnRegionChar, ParquetColumnRegionDouble, ParquetColumnRegionFloat, ParquetColumnRegionInt, ParquetColumnRegionLong, ParquetColumnRegionObject, ParquetColumnRegionShort, ParquetTableLocation, PrevColumnSource, PythonTableDataService.TableLocationImpl, RangeAggregateColumnSource, RangeAggregateColumnSourceByte, RangeAggregateColumnSourceChar, RangeAggregateColumnSourceDouble, RangeAggregateColumnSourceFloat, RangeAggregateColumnSourceInt, RangeAggregateColumnSourceLong, RangeAggregateColumnSourceObject, RangeAggregateColumnSourceShort, RedirectedColumnSource, RegionedColumnSourceManager, ReversedColumnSource, RowKeyColumnSource, RowPositionColumnSource, RowSetColumnSourceWrapper, ShiftedColumnSource, ShortAggregateColumnSource, ShortArraySource, ShortChunkColumnSource, ShortSingleValueSource, ShortSparseArraySource, ShortSsmBackedSource, SingleValueColumnSource, SingleValueObjectColumnSource, SlicedByteAggregateColumnSource, SlicedCharAggregateColumnSource, SlicedDoubleAggregateColumnSource, SlicedFloatAggregateColumnSource, SlicedIntAggregateColumnSource, SlicedLongAggregateColumnSource, SlicedObjectAggregateColumnSource, SlicedShortAggregateColumnSource, SortedColumnPushdownManager, SparseArrayColumnSource, SwitchColumnSource, SymbolTableToUniqueIdSource, UnboxedLongBackedColumnSource, UnboxedLongBackedColumnSource, UnboxedTimeBackedColumnSource, UngroupedArrayColumnSource, UngroupedBooleanArrayColumnSource, UngroupedBoxedBooleanArrayColumnSource, UngroupedBoxedByteArrayColumnSource, UngroupedBoxedByteObjectVectorColumnSource, UngroupedBoxedCharArrayColumnSource, UngroupedBoxedCharObjectVectorColumnSource, UngroupedBoxedDoubleArrayColumnSource, UngroupedBoxedDoubleObjectVectorColumnSource, UngroupedBoxedFloatArrayColumnSource, UngroupedBoxedFloatObjectVectorColumnSource, UngroupedBoxedIntArrayColumnSource, UngroupedBoxedIntObjectVectorColumnSource, UngroupedBoxedLongArrayColumnSource, UngroupedBoxedLongObjectVectorColumnSource, UngroupedBoxedShortArrayColumnSource, UngroupedBoxedShortObjectVectorColumnSource, UngroupedByteArrayColumnSource, UngroupedByteVectorColumnSource, UngroupedCharArrayColumnSource, UngroupedCharVectorColumnSource, UngroupedColumnSource, UngroupedDoubleArrayColumnSource, UngroupedDoubleVectorColumnSource, UngroupedFloatArrayColumnSource, UngroupedFloatVectorColumnSource, UngroupedIntArrayColumnSource, UngroupedIntVectorColumnSource, UngroupedLongArrayColumnSource, UngroupedLongVectorColumnSource, UngroupedObjectVectorColumnSource, UngroupedShortArrayColumnSource, UngroupedShortVectorColumnSource, UnionColumnSource, UnionSourceManager, ViewColumnSource, WritableByteAsBooleanColumnSource, WritableRedirectedColumnSource, ZonedDateTimeArraySource, ZonedDateTimeAsLongSource, ZonedDateTimeSparseArraySource
public interface PushdownFilterMatcher
Interface for entities that support pushdown filtering. Must implement a filter cost estimation function that allows comparison of filter pushdown execution costs. These costs determine the order in which filters are executed.

NOTE: There may be multiple pushdown filter operations available for a single filter and the pushdown cost is dynamic based on the input rowset and on previously executed pushdown filter steps. For example, parquet table locations may leverage low cost metadata operations (row group min/max) as a first step, followed by an index table operation or a binary search on a sorted column. The PushdownFilterContext is used to track the state of the pushdown filter to return accurate cost estimates for the next step.

  • Method Details

    • estimatePushdownFilterCost

      default void estimatePushdownFilterCost(WhereFilter filter, RowSet selection, boolean usePrev, PushdownFilterContext context, JobScheduler jobScheduler, LongConsumer onComplete, Consumer<Exception> onError)
      Estimate the cost of pushing down the next pushdown filter. This returns a unitless value to compare the cost of executing different filters. Common costs are listed in PushdownResult (such as PushdownResult.REGION_METADATA_STATS_COST) and should be used as a baseline for estimating the cost of newly implemented pushdown operations.

      A no-op implementation should simply complete with Long.MAX_VALUE.

      Parameters:
      filter - The filter to test.
      selection - The set of rows to tests.
      usePrev - Whether to use the previous result
      context - The PushdownFilterContext to use for the pushdown operation.
      jobScheduler - The job scheduler to use for scheduling child jobs
      onComplete - Consumer of the estimated cost of the pushdown operation. 9223372036854775807L indicates that the filter cannot be pushed down.
      onError - Consumer of any exceptions that occur during the estimate operation

    • pushdownFilter

      default void pushdownFilter(WhereFilter filter, RowSet selection, boolean usePrev, PushdownFilterContext context, long costCeiling, JobScheduler jobScheduler, Consumer<PushdownResult> onComplete, Consumer<Exception> onError)
      Push down the given filter to the underlying table and pass the result to the consumer. This method is expected to execute all pushdown filter steps that are greater than PushdownFilterContext.executedFilterCost() and less than or equal to costCeiling.

      The resulting PushdownResult (to onComplete) must only contain rows from selection. The match row set are rows that are guaranteed to match. The implicitly "missing" rows selection - match - maybeMatch are rows that are guaranteed to not match. The remaining maybe match row set are rows that may, or may not, match. The pushdown result ownership passes to onComplete.

      A no-op implementation should simply complete with PushdownResult.maybeMatch(selection).

      Parameters:
      filter - The filter to apply.
      selection - The set of rows to test.
      usePrev - Whether to use the previous result
      context - The PushdownFilterContext to use for the pushdown operation.
      costCeiling - Execute all possible filters with a cost less than or equal this value.
      jobScheduler - The job scheduler to use for scheduling child jobs
      onComplete - Consumer of the output rowsets for added and modified rows that pass the filter
      onError - Consumer of any exceptions that occur during the pushdown operation

    • makePushdownFilterContext

      default PushdownFilterContext makePushdownFilterContext(WhereFilter filter, List<ColumnSource<?>> filterSources)
      Create a pushdown filter context for this entity.
      Parameters:
      filter - the filter to use while making the context
      filterSources - the column sources that match the filter column names
      Returns:
      the created filter context

    • getPushdownFilterMatcher

      static PushdownFilterMatcher getPushdownFilterMatcher(WhereFilter filter, List<ColumnSource<?>> filterSources)
      Given a filter and a list of column sources, return the appropriate PushdownFilterMatcher to use for pushing down the filter (or null when pushdown is not supported).
      Parameters:
      filter - The filter to match.
      filterSources - The list of column sources for the filter columns; must be parallel to filter.getColumns().
      Returns:
      The PushdownFilterMatcher to use for pushing down the filter, or null if pushdown is not supported.

    • canPushdownFilter

      static boolean canPushdownFilter(WhereFilter filter)
      Check if the given filter can be pushed down.
      Parameters:
      filter - The filter to check.
      Returns:
      true if the filter can be pushed down, false otherwise.