Package io.deephaven.engine.table.impl.by

Interface IterativeChunkedAggregationOperator

All Known Subinterfaces:

GroupByOperator

All Known Implementing Classes:

AddOnlyFirstOrLastChunkedOperator, BaseBlinkFirstOrLastChunkedOperator, BigDecimalChunkedSumOperator, BigIntegerChunkedSumOperator, BlinkFirstChunkedOperator, BlinkLastChunkedOperator, BooleanChunkedSumOperator, ByteAddOnlySortedFirstOrLastChunkedOperator, ByteBlinkSortedFirstOrLastChunkedOperator, ByteChunkedAddOnlyUniqueOperator, ByteChunkedCountDistinctOperator, ByteChunkedDistinctOperator, ByteChunkedSumOperator, ByteChunkedUniqueOperator, ByteRollupCountDistinctOperator, ByteRollupDistinctOperator, ByteRollupUniqueOperator, ByteStaticPercentileOperator, CharAddOnlySortedFirstOrLastChunkedOperator, CharBlinkSortedFirstOrLastChunkedOperator, CharChunkedAddOnlyUniqueOperator, CharChunkedCountDistinctOperator, CharChunkedDistinctOperator, CharChunkedSumOperator, CharChunkedUniqueOperator, CharRollupCountDistinctOperator, CharRollupDistinctOperator, CharRollupUniqueOperator, CharStaticPercentileOperator, CopyingPermutedBlinkFirstOrLastChunkedOperator, CountWhereOperator, DoubleAddOnlySortedFirstOrLastChunkedOperator, DoubleBlinkSortedFirstOrLastChunkedOperator, DoubleChunkedAddOnlyUniqueOperator, DoubleChunkedCountDistinctOperator, DoubleChunkedDistinctOperator, DoubleChunkedUniqueOperator, DoubleRollupCountDistinctOperator, DoubleRollupDistinctOperator, DoubleRollupUniqueOperator, DoubleStaticPercentileOperator, FirstOrLastChunkedOperator, FloatAddOnlySortedFirstOrLastChunkedOperator, FloatBlinkSortedFirstOrLastChunkedOperator, FloatChunkedAddOnlyUniqueOperator, FloatChunkedCountDistinctOperator, FloatChunkedDistinctOperator, FloatChunkedUniqueOperator, FloatRollupCountDistinctOperator, FloatRollupDistinctOperator, FloatRollupUniqueOperator, FloatStaticPercentileOperator, FreezeByCountOperator, FreezeByOperator, GroupByChunkedOperator, GroupByReaggregateOperator, IntAddOnlySortedFirstOrLastChunkedOperator, IntBlinkSortedFirstOrLastChunkedOperator, IntChunkedAddOnlyUniqueOperator, IntChunkedCountDistinctOperator, IntChunkedDistinctOperator, IntChunkedSumOperator, IntChunkedUniqueOperator, IntRollupCountDistinctOperator, IntRollupDistinctOperator, IntRollupUniqueOperator, IntStaticPercentileOperator, LongAddOnlySortedFirstOrLastChunkedOperator, LongBlinkSortedFirstOrLastChunkedOperator, LongChunkedAddOnlyUniqueOperator, LongChunkedCountDistinctOperator, LongChunkedDistinctOperator, LongChunkedSumOperator, LongChunkedUniqueOperator, LongRollupCountDistinctOperator, LongRollupDistinctOperator, LongRollupUniqueOperator, LongStaticPercentileOperator, ObjectAddOnlySortedFirstOrLastChunkedOperator, ObjectBlinkSortedFirstOrLastChunkedOperator, ObjectChunkedAddOnlyUniqueOperator, ObjectChunkedCountDistinctOperator, ObjectChunkedDistinctOperator, ObjectChunkedUniqueOperator, ObjectRollupCountDistinctOperator, ObjectRollupDistinctOperator, ObjectRollupUniqueOperator, PartitionByChunkedOperator, ShortAddOnlySortedFirstOrLastChunkedOperator, ShortBlinkSortedFirstOrLastChunkedOperator, ShortChunkedAddOnlyUniqueOperator, ShortChunkedCountDistinctOperator, ShortChunkedDistinctOperator, ShortChunkedSumOperator, ShortChunkedUniqueOperator, ShortRollupCountDistinctOperator, ShortRollupDistinctOperator, ShortRollupUniqueOperator, ShortStaticPercentileOperator, SortedFirstOrLastChunkedOperator, SsmChunkedMinMaxOperator, SsmChunkedPercentileOperator, StaticFirstOrLastChunkedOperator, StaticPercentileOperator, TDigestPercentileOperator, UniqueRowKeyChunkedOperator

public interface IterativeChunkedAggregationOperator

A chunked, iterative operator that processes row keys and/or data from one input column to produce one or more output columns.

Nested Class Summary

Nested Classes

Modifier and Type	Interface	Description
static interface	IterativeChunkedAggregationOperator.BucketedContext	Context interface for bucketed operator updates.
static interface	IterativeChunkedAggregationOperator.SingletonContext	Context interface for singleton (that is, one aggregation state) operator updates.

Field Summary

Fields

Modifier and Type	Field	Description
static final IterativeChunkedAggregationOperator[]	ZERO_LENGTH_ITERATIVE_CHUNKED_AGGREGATION_OPERATOR_ARRAY

Method Summary

Modifier and Type	Method	Description
void	addChunk(IterativeChunkedAggregationOperator.BucketedContext context, Chunk<? extends Values> values, LongChunk<? extends RowKeys> inputRowKeys, IntChunk<RowKeys> destinations, IntChunk<ChunkPositions> startPositions, IntChunk<ChunkLengths> length, WritableBooleanChunk<Values> stateModified)	Aggregate a chunk of data into the result columns.
boolean	addChunk(IterativeChunkedAggregationOperator.SingletonContext context, int chunkSize, Chunk<? extends Values> values, LongChunk<? extends RowKeys> inputRowKeys, long destination)	Aggregate a chunk of data into the result columns.
default boolean	addRowSet(IterativeChunkedAggregationOperator.SingletonContext context, RowSet rowSet, long destination)
void	ensureCapacity(long tableSize)	Ensure that this operator can handle destinations up to tableSize - 1.
Map<String,? extends ColumnSource<?>>	getResultColumns()	Return a map of result columns produced by this operator.
default UnaryOperator<ModifiedColumnSet>	initializeRefreshing(@NotNull QueryTable resultTable, @NotNull LivenessReferent aggregationUpdateListener)	Initialize refreshing result support for this operator.
default IterativeChunkedAggregationOperator.BucketedContext	makeBucketedContext(int size)	Make a IterativeChunkedAggregationOperator.BucketedContext suitable for this operator if necessary.
default IterativeChunkedAggregationOperator.SingletonContext	makeSingletonContext(int size)	Make a IterativeChunkedAggregationOperator.SingletonContext suitable for this operator if necessary.
default void	modifyChunk(IterativeChunkedAggregationOperator.BucketedContext context, Chunk<? extends Values> previousValues, Chunk<? extends Values> newValues, LongChunk<? extends RowKeys> postShiftRowKeys, IntChunk<RowKeys> destinations, IntChunk<ChunkPositions> startPositions, IntChunk<ChunkLengths> length, WritableBooleanChunk<Values> stateModified)	Modify a chunk of data previously aggregated into the result columns using a parallel chunk of new values.
default boolean	modifyChunk(IterativeChunkedAggregationOperator.SingletonContext context, int chunkSize, Chunk<? extends Values> previousValues, Chunk<? extends Values> newValues, LongChunk<? extends RowKeys> postShiftRowKeys, long destination)	Modify a chunk of data previously aggregated into the result columns using a parallel chunk of new values.
default void	modifyRowKeys(IterativeChunkedAggregationOperator.BucketedContext context, LongChunk<? extends RowKeys> inputRowKeys, IntChunk<RowKeys> destinations, IntChunk<ChunkPositions> startPositions, IntChunk<ChunkLengths> length, WritableBooleanChunk<Values> stateModified)	Called with the modified row keys when requiresRowKeys() returns true if our input columns have not changed (or we have none).
default boolean	modifyRowKeys(IterativeChunkedAggregationOperator.SingletonContext context, LongChunk<? extends RowKeys> rowKeys, long destination)	Called with the modified row keys when requiresRowKeys() returns true if our input columns have not changed (or we have none).
default void	propagateFailure(@NotNull Throwable originalException, TableListener.Entry sourceEntry)	Called on error to propagate listener failure to this operator.
default void	propagateInitialState(@NotNull QueryTable resultTable, int startingDestinationsCount)	Perform any internal state keeping needed for destinations that were added during initialization.
default void	propagateUpdates(@NotNull TableUpdate downstream, @NotNull RowSet newDestinations)	Perform any internal state keeping needed for destinations that were added (went from 0 keys to > 0), removed (went from > 0 keys to 0), or modified (keys added or removed, or keys modified) by this iteration.
void	removeChunk(IterativeChunkedAggregationOperator.BucketedContext context, Chunk<? extends Values> values, LongChunk<? extends RowKeys> inputRowKeys, IntChunk<RowKeys> destinations, IntChunk<ChunkPositions> startPositions, IntChunk<ChunkLengths> length, WritableBooleanChunk<Values> stateModified)	Remove a chunk of data previously aggregated into the result columns.
boolean	removeChunk(IterativeChunkedAggregationOperator.SingletonContext context, int chunkSize, Chunk<? extends Values> values, LongChunk<? extends RowKeys> inputRowKeys, long destination)	Remove a chunk of data previously aggregated into the result columns.
default boolean	requiresRowKeys()	Whether the operator requires row keys.
default boolean	requiresRunFinds()	Some operators require that all values for a destination within a given chunk are adjacent.
default boolean	resetForStep(@NotNull TableUpdate upstream, int startingDestinationsCount)	Reset any per-step internal state.
default void	shiftChunk(IterativeChunkedAggregationOperator.BucketedContext context, Chunk<? extends Values> previousValues, Chunk<? extends Values> newValues, LongChunk<? extends RowKeys> preShiftRowKeys, LongChunk<? extends RowKeys> postShiftRowKeys, IntChunk<RowKeys> destinations, IntChunk<ChunkPositions> startPositions, IntChunk<ChunkLengths> length, WritableBooleanChunk<Values> stateModified)	Called with shifted row keys when requiresRowKeys() returns true, including shifted same-slot modifies.
default boolean	shiftChunk(IterativeChunkedAggregationOperator.SingletonContext context, Chunk<? extends Values> previousValues, Chunk<? extends Values> newValues, LongChunk<? extends RowKeys> preShiftRowKeys, LongChunk<? extends RowKeys> postShiftRowKeys, long destination)	Shift a chunk of data previously aggregated into the result columns, including shifted same-slot modifies.
void	startTrackingPrevValues()	Called after initialization; when the operator's result columns must have previous tracking enabled.
default boolean	unchunkedRowSet()	Whether the operator can deal with an unchunked RowSet more efficiently than a chunked RowSet.

Field Details

ZERO_LENGTH_ITERATIVE_CHUNKED_AGGREGATION_OPERATOR_ARRAY

static final IterativeChunkedAggregationOperator[] ZERO_LENGTH_ITERATIVE_CHUNKED_AGGREGATION_OPERATOR_ARRAY

Method Details

addChunk

void addChunk(IterativeChunkedAggregationOperator.BucketedContext context,
 Chunk<? extends Values> values,
 LongChunk<? extends RowKeys> inputRowKeys,
 IntChunk<RowKeys> destinations,
 IntChunk<ChunkPositions> startPositions,
 IntChunk<ChunkLengths> length,
 WritableBooleanChunk<Values> stateModified)

Aggregate a chunk of data into the result columns.

Parameters:

context - the operator-specific context

values - a chunk of values to aggregate

inputRowKeys - the input row keys, in post-shift space

destinations - the destinations in resultColumn to aggregate into, parallel with startPositions and length

startPositions - the starting positions in the chunk for each destination

length - the number of values in the chunk for each destination

stateModified - a boolean output array, parallel to destinations, which is set to true if the corresponding destination has been modified

removeChunk

void removeChunk(IterativeChunkedAggregationOperator.BucketedContext context,
 Chunk<? extends Values> values,
 LongChunk<? extends RowKeys> inputRowKeys,
 IntChunk<RowKeys> destinations,
 IntChunk<ChunkPositions> startPositions,
 IntChunk<ChunkLengths> length,
 WritableBooleanChunk<Values> stateModified)

Remove a chunk of data previously aggregated into the result columns.

Parameters:

context - the operator-specific context

values - a chunk of values that have been previously aggregated.

inputRowKeys - the input row keys, in pre-shift space

destinations - the destinations in resultColumn to remove the values from, parallel with startPositions and length

startPositions - the starting positions in the chunk for each destination

length - the number of values in the chunk for each destination

stateModified - a boolean output array, parallel to destinations, which is set to true if the corresponding destination has been modified

modifyChunk

default void modifyChunk(IterativeChunkedAggregationOperator.BucketedContext context,
 Chunk<? extends Values> previousValues,
 Chunk<? extends Values> newValues,
 LongChunk<? extends RowKeys> postShiftRowKeys,
 IntChunk<RowKeys> destinations,
 IntChunk<ChunkPositions> startPositions,
 IntChunk<ChunkLengths> length,
 WritableBooleanChunk<Values> stateModified)

Modify a chunk of data previously aggregated into the result columns using a parallel chunk of new values. Never includes modifies that have been shifted if requiresRowKeys() returns true - those are handled in shiftChunk(BucketedContext, Chunk, Chunk, LongChunk, LongChunk, IntChunk, IntChunk, IntChunk, WritableBooleanChunk).

Parameters:

context - the operator-specific context

previousValues - a chunk of values that have been previously aggregated

newValues - a chunk of values to aggregate

postShiftRowKeys - the input row keys, in post-shift space

destinations - the destinations in resultColumn to remove the values from, parallel with startPositions and length

startPositions - the starting positions in the chunk for each destination

length - the number of values in the chunk for each destination

stateModified - a boolean output array, parallel to destinations, which is set to true if the corresponding destination has been modified

shiftChunk

default void shiftChunk(IterativeChunkedAggregationOperator.BucketedContext context,
 Chunk<? extends Values> previousValues,
 Chunk<? extends Values> newValues,
 LongChunk<? extends RowKeys> preShiftRowKeys,
 LongChunk<? extends RowKeys> postShiftRowKeys,
 IntChunk<RowKeys> destinations,
 IntChunk<ChunkPositions> startPositions,
 IntChunk<ChunkLengths> length,
 WritableBooleanChunk<Values> stateModified)

Called with shifted row keys when requiresRowKeys() returns true, including shifted same-slot modifies.

Parameters:

context - the operator-specific context

previousValues - a chunk of values that have been previously aggregated.

newValues - a chunk of values to aggregate

preShiftRowKeys - the input row keys, in pre-shift space

postShiftRowKeys - the input row keys, in post-shift space

destinations - the destinations in resultColumn to aggregate into, parallel with startPositions and length

startPositions - the starting positions in the chunk for each destination

length - the number of values in the chunk for each destination

stateModified - a boolean output array, parallel to destinations, which is set to true if the corresponding destination has been modified

modifyRowKeys

default void modifyRowKeys(IterativeChunkedAggregationOperator.BucketedContext context,
 LongChunk<? extends RowKeys> inputRowKeys,
 IntChunk<RowKeys> destinations,
 IntChunk<ChunkPositions> startPositions,
 IntChunk<ChunkLengths> length,
 WritableBooleanChunk<Values> stateModified)

Called with the modified row keys when requiresRowKeys() returns true if our input columns have not changed (or we have none).

Parameters:

context - the operator-specific context

inputRowKeys - the input row keys, in post-shift space

destinations - the destinations in resultColumn to aggregate into, parallel with startPositions and length

startPositions - the starting positions in the chunk for each destination

length - the number of values in the chunk for each destination

stateModified - a boolean output array, parallel to destinations, which is set to true if the corresponding destination has been modified

addChunk

boolean addChunk(IterativeChunkedAggregationOperator.SingletonContext context,
 int chunkSize,
 Chunk<? extends Values> values,
 LongChunk<? extends RowKeys> inputRowKeys,
 long destination)

Aggregate a chunk of data into the result columns.

Parameters:

context - the operator-specific context

chunkSize - the size of the addition

values - the values to aggregate

inputRowKeys - the input row keys, in post-shift space

destination - the destination in the result columns

Returns:

true if the state was modified, false otherwise

removeChunk

boolean removeChunk(IterativeChunkedAggregationOperator.SingletonContext context,
 int chunkSize,
 Chunk<? extends Values> values,
 LongChunk<? extends RowKeys> inputRowKeys,
 long destination)

Remove a chunk of data previously aggregated into the result columns.

Parameters:

context - the operator-specific context

chunkSize - the size of the removal

values - the values to remove from the aggregation

inputRowKeys - the input row keys, in pre-shift space

destination - the destination in the result columns

Returns:

true if the state was modified, false otherwise

modifyChunk

default boolean modifyChunk(IterativeChunkedAggregationOperator.SingletonContext context,
 int chunkSize,
 Chunk<? extends Values> previousValues,
 Chunk<? extends Values> newValues,
 LongChunk<? extends RowKeys> postShiftRowKeys,
 long destination)

Modify a chunk of data previously aggregated into the result columns using a parallel chunk of new values. Never includes modifies that have been shifted if requiresRowKeys() returns true - those are handled in shiftChunk(SingletonContext, Chunk, Chunk, LongChunk, LongChunk, long).

Parameters:

context - the operator-specific context

chunkSize - the size of the modification

previousValues - a chunk of values that have been previously aggregated.

newValues - a chunk of values to aggregate

postShiftRowKeys - the input row keys, in post-shift space

Returns:

true if the state was modified, false otherwise

shiftChunk

default boolean shiftChunk(IterativeChunkedAggregationOperator.SingletonContext context,
 Chunk<? extends Values> previousValues,
 Chunk<? extends Values> newValues,
 LongChunk<? extends RowKeys> preShiftRowKeys,
 LongChunk<? extends RowKeys> postShiftRowKeys,
 long destination)

Shift a chunk of data previously aggregated into the result columns, including shifted same-slot modifies.

Parameters:

context - the operator-specific context

previousValues - a chunk of values that have been previously aggregated.

newValues - a chunk of values to aggregate

preShiftRowKeys - the input row keys, in pre-shift space

postShiftRowKeys - the input row keys, in post-shift space

destination - the destination in the result columns

Returns:

true if the result should be considered modified

modifyRowKeys

default boolean modifyRowKeys(IterativeChunkedAggregationOperator.SingletonContext context,
 LongChunk<? extends RowKeys> rowKeys,
 long destination)

Called with the modified row keys when requiresRowKeys() returns true if our input columns have not changed (or we have none).

Parameters:

context - the operator-specific context

rowKeys - the modified row keys for a given destination, in post-shift space

destination - the destination that was modified

Returns:

true if the result should be considered modified

requiresRowKeys

default boolean requiresRowKeys()

Whether the operator requires row keys. This implies that the operator must process shifts (i.e. shiftChunk(io.deephaven.engine.table.impl.by.IterativeChunkedAggregationOperator.BucketedContext, io.deephaven.chunk.Chunk<? extends io.deephaven.chunk.attributes.Values>, io.deephaven.chunk.Chunk<? extends io.deephaven.chunk.attributes.Values>, io.deephaven.chunk.LongChunk<? extends io.deephaven.engine.rowset.chunkattributes.RowKeys>, io.deephaven.chunk.LongChunk<? extends io.deephaven.engine.rowset.chunkattributes.RowKeys>, io.deephaven.chunk.IntChunk<io.deephaven.engine.rowset.chunkattributes.RowKeys>, io.deephaven.chunk.IntChunk<io.deephaven.chunk.attributes.ChunkPositions>, io.deephaven.chunk.IntChunk<io.deephaven.chunk.attributes.ChunkLengths>, io.deephaven.chunk.WritableBooleanChunk<io.deephaven.chunk.attributes.Values>)), and must observe modifications even when its input columns (if any) are not modified (i.e. modifyRowKeys(io.deephaven.engine.table.impl.by.IterativeChunkedAggregationOperator.BucketedContext, io.deephaven.chunk.LongChunk<? extends io.deephaven.engine.rowset.chunkattributes.RowKeys>, io.deephaven.chunk.IntChunk<io.deephaven.engine.rowset.chunkattributes.RowKeys>, io.deephaven.chunk.IntChunk<io.deephaven.chunk.attributes.ChunkPositions>, io.deephaven.chunk.IntChunk<io.deephaven.chunk.attributes.ChunkLengths>, io.deephaven.chunk.WritableBooleanChunk<io.deephaven.chunk.attributes.Values>)).

Returns:

true if the operator requires row keys, false otherwise

requiresRunFinds

default boolean requiresRunFinds()

Some operators require that all values for a destination within a given chunk are adjacent. These operators must return true.

Returns:

true if the operator requires runs of the same destination to be adjacent

unchunkedRowSet

default boolean unchunkedRowSet()

Whether the operator can deal with an unchunked RowSet more efficiently than a chunked RowSet.

Returns:

true if the operator can deal with unchunked RowSets, false otherwise

addRowSet

default boolean addRowSet(IterativeChunkedAggregationOperator.SingletonContext context,
 RowSet rowSet,
 long destination)

ensureCapacity

void ensureCapacity(long tableSize)

Ensure that this operator can handle destinations up to tableSize - 1.

Parameters:

tableSize - the new size of the table

getResultColumns

Map<String,? extends ColumnSource<?>> getResultColumns()

Return a map of result columns produced by this operator.

Returns:

a map of name to columns for the result table

propagateInitialState

default void propagateInitialState(@NotNull
 @NotNull QueryTable resultTable,
 int startingDestinationsCount)

Perform any internal state keeping needed for destinations that were added during initialization.

Parameters:

resultTable - The result QueryTable after initialization

startingDestinationsCount - The number of used destinations at the beginning of this step

startTrackingPrevValues

void startTrackingPrevValues()

Called after initialization; when the operator's result columns must have previous tracking enabled.

initializeRefreshing

default UnaryOperator<ModifiedColumnSet> initializeRefreshing(@NotNull
 @NotNull QueryTable resultTable,
 @NotNull
 @NotNull LivenessReferent aggregationUpdateListener)

Initialize refreshing result support for this operator. As a side effect, make a factory method for converting upstream modified column sets to result modified column sets, to be invoked whenever this operator reports a modification in order to determine the operator's contribution to the final result modified column set.

Parameters:

resultTable - The result QueryTable after initialization

aggregationUpdateListener - The aggregation update listener, which may be needed for referential integrity

Returns:

A factory that produces a result modified column set from the upstream modified column set

resetForStep

default boolean resetForStep(@NotNull
 @NotNull TableUpdate upstream,
 int startingDestinationsCount)

Reset any per-step internal state. Note that the arguments to this method should not be mutated in any way.

Parameters:

upstream - The upstream ShiftAwareListener.Update

startingDestinationsCount - The number of used destinations at the beginning of this step

Returns:

true if this operator must generate modifications on this cycle; typically an operator returns false and depends on the actual add/remove/modify/shift calls to determine modifications

propagateUpdates

default void propagateUpdates(@NotNull
 @NotNull TableUpdate downstream,
 @NotNull
 @NotNull RowSet newDestinations)

Perform any internal state keeping needed for destinations that were added (went from 0 keys to > 0), removed (went from > 0 keys to 0), or modified (keys added or removed, or keys modified) by this iteration. Note that the arguments to this method should not be mutated in any way.

Parameters:

downstream - The downstream TableUpdate (which does not have its ModifiedColumnSet finalized yet)

newDestinations - New destinations added on this update

propagateFailure

default void propagateFailure(@NotNull
 @NotNull Throwable originalException,
 @NotNull
 TableListener.Entry sourceEntry)

Called on error to propagate listener failure to this operator.

Parameters:

originalException - The error Throwable

sourceEntry - The UpdatePerformanceTracker.Entry for the failed listener

makeBucketedContext

default IterativeChunkedAggregationOperator.BucketedContext makeBucketedContext(int size)

Make a IterativeChunkedAggregationOperator.BucketedContext suitable for this operator if necessary.

Parameters:

size - The maximum size of input chunks that will be used with the result context

Returns:

A new IterativeChunkedAggregationOperator.BucketedContext, or null if none is necessary

makeSingletonContext

default IterativeChunkedAggregationOperator.SingletonContext makeSingletonContext(int size)

Make a IterativeChunkedAggregationOperator.SingletonContext suitable for this operator if necessary.

Parameters:

size - The maximum size of input chunks that will be used with the result context

Returns:

A new IterativeChunkedAggregationOperator.SingletonContext, or null if none is necessary