Package io.deephaven.engine.table.impl

Class AbstractColumnSource<T>

java.lang.Object
io.deephaven.engine.table.impl.AbstractColumnSource<T>
All Implemented Interfaces:
ChunkSource<Values>, ChunkSource.WithPrev<Values>, ColumnSource<T>, ElementSource<T>, FillContextMaker, GetContextMaker, DefaultChunkSource<Values>, DefaultChunkSource.WithPrev<Values>, PushdownFilterMatcher, Releasable, TupleExporter<T>, TupleSource<T>
Direct Known Subclasses:
AbstractArrowColumnSource, AbstractColumnSource.DefaultedImmutable, AbstractColumnSource.DefaultedMutable, AlternatingColumnSource, ArrayBackedColumnSource, BaseAggregateSlicedColumnSource, BitMaskingColumnSource, BitShiftingColumnSource, BooleanAsByteColumnSource, BooleanSparseArraySource.ReinterpretedAsByte, ByteAggregateColumnSource, ByteAsBooleanColumnSource, ByteChunkColumnSource, ByteSsmBackedSource, CharAggregateColumnSource, CharChunkColumnSource, CharSsmBackedSource, CrossJoinRightColumnSource, DelegatingColumnSource, DeltaAwareColumnSource, DoubleAggregateColumnSource, DoubleChunkColumnSource, DoubleNullToZeroColumnSource, DoubleSsmBackedSource, FloatAggregateColumnSource, FloatChunkColumnSource, FloatSsmBackedSource, HashTableColumnSource, Immutable2DByteArraySource, Immutable2DCharArraySource, Immutable2DDoubleArraySource, Immutable2DFloatArraySource, Immutable2DIntArraySource, Immutable2DLongArraySource, Immutable2DNanosBasedTimeArraySource, Immutable2DObjectArraySource, Immutable2DShortArraySource, ImmutableByteArraySource, ImmutableCharArraySource, ImmutableConstantByteSource, ImmutableConstantCharSource, ImmutableConstantDoubleSource, ImmutableConstantFloatSource, ImmutableConstantIntSource, ImmutableConstantLongSource, ImmutableConstantNanosBasedTimeSource, ImmutableConstantObjectSource, ImmutableConstantShortSource, ImmutableDoubleArraySource, ImmutableFloatArraySource, ImmutableIntArraySource, ImmutableLongArraySource, ImmutableNanosBasedTimeArraySource, ImmutableObjectArraySource, ImmutableShortArraySource, InstantSsmSourceWrapper, IntAggregateColumnSource, IntChunkColumnSource, IntSsmBackedSource, LocalDateWrapperSource, LocalTimeWrapperSource, LongAggregateColumnSource, LongAsTimeSource, LongChunkColumnSource, LongSsmBackedSource, MergeSortedHelper.SortedMergeColumnSource, NanosBasedTimeArraySource, NanosBasedTimeSparseArraySource, NullValueColumnSource, ObjectAggregateColumnSource, ObjectChunkColumnSource, ObjectSsmBackedSource, PrevColumnSource, RangeAggregateColumnSource, RedirectedColumnSource, ReversedColumnSource, RowKeyColumnSource, RowPositionColumnSource, RowSetColumnSourceWrapper, ShiftedColumnSource, ShortAggregateColumnSource, ShortChunkColumnSource, ShortSsmBackedSource, SingleValueColumnSource, SingleValueObjectColumnSource, SparseArrayColumnSource, SwitchColumnSource, SymbolTableToUniqueIdSource, UnboxedLongBackedColumnSource, UnboxedLongBackedColumnSource, UnboxedTimeBackedColumnSource, UngroupedColumnSource, UnionColumnSource, ViewColumnSource
public abstract class AbstractColumnSource<T> extends Object implements ColumnSource<T>, DefaultChunkSource.WithPrev<Values>, PushdownFilterMatcher

  • Field Details

    • USE_PARTIAL_TABLE_DATA_INDEX

      public static boolean USE_PARTIAL_TABLE_DATA_INDEX
      For a match(boolean, boolean, boolean, DataIndex, RowSet, Object...) call that uses a DataIndex, by default we do not force the entire DataIndex to be loaded into memory. This is because many MatchFilters are highly selective and only need to instantiate a single RowSet value rather than the complete DataIndex for the entire table. When the Configuration property "AbstractColumnSource.usePartialDataIndex" is set to false, the query engine materializes the entire DataIndex table for the match call.

    • USE_PARALLEL_ROWSET_BUILD

      public static boolean USE_PARALLEL_ROWSET_BUILD
      After generating a DataIndex table and identifying which row keys are responsive to the filter, the result RowSet can be built in serial or in parallel. By default, the index is built in parallel which may take advantage of using more threads for I/O of the index data structure. Parallel builds do require more setup and thread synchronization, so they can be disabled by setting the Configuration property "AbstractColumnSource.useParallelIndexBuild" to false.

    • USE_RANGES_AVERAGE_RUN_LENGTH

      public static final long USE_RANGES_AVERAGE_RUN_LENGTH
      Minimum average run length in an RowSequence that should trigger Chunk-filling by key ranges instead of individual keys.
      See Also:

    • type

      protected final Class<T> type

    • componentType

      protected final Class<?> componentType

    • updateGraph

      protected final UpdateGraph updateGraph

  • Constructor Details

    • AbstractColumnSource

      protected AbstractColumnSource(@NotNull @NotNull Class<T> type)

    • AbstractColumnSource

      public AbstractColumnSource(@NotNull @NotNull Class<T> type, @Nullable @Nullable Class<?> elementType)

  • Method Details

    • getType

      public Class<T> getType()
      Specified by:
      getType in interface ColumnSource<T>

    • getComponentType

      public Class<?> getComponentType()
      Specified by:
      getComponentType in interface ColumnSource<T>

    • getPrevSource

      public ColumnSource<T> getPrevSource()
      Specified by:
      getPrevSource in interface ChunkSource.WithPrev<T>
      Specified by:
      getPrevSource in interface ColumnSource<T>
      Specified by:
      getPrevSource in interface DefaultChunkSource.WithPrev<T>
      Returns:
      a chunk source which accesses the previous values.

    • match

      public WritableRowSet match(boolean invertMatch, boolean usePrev, boolean caseInsensitive, @Nullable @Nullable DataIndex dataIndex, @NotNull @NotNull RowSet rowsetToFilter, Object... keys)
      Description copied from interface: ColumnSource
      Return a row set where the values in the column source match the given keys.
      Specified by:
      match in interface ColumnSource<T>
      Parameters:
      invertMatch - Whether to invert the match, i.e. return the rows where the values do not match the given keys
      usePrev - Whether to use the previous values for the ColumnSource
      caseInsensitive - Whether to perform a case insensitive match
      dataIndex - An optional data index that can be used to accelerate the match (the index table must be included in snapshot controls or otherwise guaranteed to be current)
      rowsetToFilter - Restrict results to this row set
      keys - The keys to match in the column
      Returns:
      The rows that match the given keys

    • fillChunk

      public void fillChunk(@NotNull @NotNull ChunkSource.FillContext context, @NotNull @NotNull WritableChunk<? super Values> destination, @NotNull @NotNull RowSequence rowSequence)
      Description copied from interface: ChunkSource
      Populates the given destination chunk with data corresponding to the keys from the given RowSequence.
      Specified by:
      fillChunk in interface ChunkSource<T>
      Parameters:
      context - A context containing all mutable/state related data used in retrieving the Chunk.
      destination - The chunk to be populated according to rowSequence. No assumptions shall be made about the size of the chunk shall be made. The chunk will be populated from position [0,rowSequence.size()).
      rowSequence - An RowSequence representing the keys to be fetched

    • defaultFillChunk

      @VisibleForTesting public final void defaultFillChunk(@NotNull @NotNull ChunkSource.FillContext context, @NotNull @NotNull WritableChunk<? super Values> destination, @NotNull @NotNull RowSequence rowSequence)

    • fillPrevChunk

      public void fillPrevChunk(@NotNull @NotNull ChunkSource.FillContext context, @NotNull @NotNull WritableChunk<? super Values> destination, @NotNull @NotNull RowSequence rowSequence)
      Description copied from interface: ChunkSource.WithPrev
      Populates the given destination chunk with data corresponding to the keys from the given RowSequence.
      Specified by:
      fillPrevChunk in interface ChunkSource.WithPrev<T>
      Parameters:
      context - A context containing all mutable/state related data used in retrieving the Chunk.
      destination - The chunk to be populated according to rowSequence. No assumptions shall be made about the size of the chunk shall be made. The chunk will be populated from position [0,rowSequence.size()).
      rowSequence - An RowSequence representing the keys to be fetched

    • defaultFillPrevChunk

      protected final void defaultFillPrevChunk(@NotNull @NotNull ChunkSource.FillContext context, @NotNull @NotNull WritableChunk<? super Values> destination, @NotNull @NotNull RowSequence rowSequence)

    • pushdownManager

      public PushdownPredicateManager pushdownManager()
      Get the pushdown predicate manager for this column source; returns null if there is no pushdown manager.

    • estimatePushdownFilterCost

      public void estimatePushdownFilterCost(WhereFilter filter, RowSet selection, boolean usePrev, PushdownFilterContext context, JobScheduler jobScheduler, LongConsumer onComplete, Consumer<Exception> onError)
      Description copied from interface: PushdownFilterMatcher
      Estimate the cost of pushing down the next pushdown filter. This returns a unitless value to compare the cost of executing different filters. Common costs are listed in PushdownResult (such as PushdownResult.METADATA_STATS_COST) and should be used as a baseline for estimating the cost of newly implemented pushdown operations.

      A no-op implementation should simply complete with Long.MAX_VALUE.

      Specified by:
      estimatePushdownFilterCost in interface PushdownFilterMatcher
      Parameters:
      filter - The filter to test.
      selection - The set of rows to tests.
      usePrev - Whether to use the previous result
      context - The PushdownFilterContext to use for the pushdown operation.
      jobScheduler - The job scheduler to use for scheduling child jobs
      onComplete - Consumer of the estimated cost of the pushdown operation. 9223372036854775807L indicates that the filter cannot be pushed down.
      onError - Consumer of any exceptions that occur during the estimate operation

    • pushdownFilter

      public void pushdownFilter(WhereFilter filter, RowSet selection, boolean usePrev, PushdownFilterContext context, long costCeiling, JobScheduler jobScheduler, Consumer<PushdownResult> onComplete, Consumer<Exception> onError)
      Description copied from interface: PushdownFilterMatcher
      Push down the given filter to the underlying table and pass the result to the consumer. This method is expected to execute all pushdown filter steps that are greater than PushdownFilterContext.executedFilterCost() and less than or equal to costCeiling.

      The resulting PushdownResult (to onComplete) must only contain rows from selection. The match row set are rows that are guaranteed to match. The implicitly "missing" rows selection - match - maybeMatch are rows that are guaranteed to not match. The remaining maybe match row set are rows that may, or may not, match. The pushdown result ownership passes to onComplete.

      A no-op implementation should simply complete with PushdownResult.maybeMatch(selection).

      Specified by:
      pushdownFilter in interface PushdownFilterMatcher
      Parameters:
      filter - The filter to apply.
      selection - The set of rows to test.
      usePrev - Whether to use the previous result
      context - The PushdownFilterContext to use for the pushdown operation.
      costCeiling - Execute all possible filters with a cost less than or equal this value.
      jobScheduler - The job scheduler to use for scheduling child jobs
      onComplete - Consumer of the output rowsets for added and modified rows that pass the filter
      onError - Consumer of any exceptions that occur during the pushdown operation

    • makePushdownFilterContext

      public PushdownFilterContext makePushdownFilterContext(WhereFilter filter, List<ColumnSource<?>> filterSources)
      Description copied from interface: PushdownFilterMatcher
      Create a pushdown filter context for this entity.
      Specified by:
      makePushdownFilterContext in interface PushdownFilterMatcher
      Parameters:
      filter - the filter to use while making the context
      filterSources - the column sources that match the filter column names
      Returns:
      the created filter context

    • allowsReinterpret

      public <ALTERNATE_DATA_TYPE> boolean allowsReinterpret(@NotNull @NotNull Class<ALTERNATE_DATA_TYPE> alternateDataType)
      Description copied from interface: ColumnSource
      Test if a reinterpret call will succeed.
      Specified by:
      allowsReinterpret in interface ColumnSource<T>
      Parameters:
      alternateDataType - The alternative type to consider
      Returns:
      If a reinterpret on this column source with the supplied alternateDataType will succeed.

    • reinterpret

      public final <ALTERNATE_DATA_TYPE> ColumnSource<ALTERNATE_DATA_TYPE> reinterpret(@NotNull @NotNull Class<ALTERNATE_DATA_TYPE> alternateDataType) throws IllegalArgumentException
      Description copied from interface: ColumnSource
      Provide an alternative view into the data underlying this column source.
      Specified by:
      reinterpret in interface ColumnSource<T>
      Parameters:
      alternateDataType - The alternative type to expose
      Returns:
      A column source of the alternate data type, backed by the same underlying data.
      Throws:
      IllegalArgumentException - If the alternativeDataType supplied is not supported

    • doReinterpret

      protected <ALTERNATE_DATA_TYPE> ColumnSource<ALTERNATE_DATA_TYPE> doReinterpret(@NotNull @NotNull Class<ALTERNATE_DATA_TYPE> alternateDataType)
      Supply allowed reinterpret results. The default implementation handles the most common case to avoid code duplication.
      Parameters:
      alternateDataType - The alternate data type
      Returns:
      The resulting ColumnSource