Package io.deephaven.engine.table.impl.sources

Class SparseArrayColumnSource<T>

java.lang.Object
io.deephaven.engine.table.impl.AbstractColumnSource<T>
io.deephaven.engine.table.impl.sources.SparseArrayColumnSource<T>
All Implemented Interfaces:
RowSetShiftCallback, ChunkSink<Values>, ChunkSource<Values>, ChunkSource.WithPrev<Values>, ColumnSource<T>, ElementSource<T>, FillContextMaker, GetContextMaker, DefaultChunkSource<Values>, DefaultChunkSource.WithPrev<Values>, PushdownFilterMatcher, FillUnordered<Values>, InMemoryColumnSource, PossiblyImmutableColumnSource, Releasable, TupleExporter<T>, TupleSource<T>, WritableColumnSource<T>, WritableSourceWithPrepareForParallelPopulation
Direct Known Subclasses:
BooleanSparseArraySource, ByteSparseArraySource, CharacterSparseArraySource, DoubleSparseArraySource, FloatSparseArraySource, IntegerSparseArraySource, LongSparseArraySource, ObjectSparseArraySource, ShortSparseArraySource
public abstract class SparseArrayColumnSource<T> extends AbstractColumnSource<T> implements FillUnordered<Values>, WritableColumnSource<T>, InMemoryColumnSource, PossiblyImmutableColumnSource, WritableSourceWithPrepareForParallelPopulation, RowSetShiftCallback
A column source backed by arrays that may not be filled in all blocks.

To store the blocks, we use a multi-level page table like structure. Each entry that exists is complete, i.e. we never reallocate partial blocks, we always allocate the complete block. The row key is divided as follows:

Description Size Bits
Block 0 19 62-44
Block 1 18 43-26
Block 2 18 25-8
Index Within Block 8 7-0

Bit 63, the sign bit, is used to indicate null (that is, all negative numbers are defined to be null)

Parallel structures are used for previous values and prevInUse. We recycle all levels of the previous blocks, so that the previous structure takes up memory only while it is in use.

  • Method Details

    • set

      public void set(long key, byte value)
      Specified by:
      set in interface WritableColumnSource<T>

    • set

      public void set(long key, char value)
      Specified by:
      set in interface WritableColumnSource<T>

    • set

      public void set(long key, double value)
      Specified by:
      set in interface WritableColumnSource<T>

    • set

      public void set(long key, float value)
      Specified by:
      set in interface WritableColumnSource<T>

    • set

      public void set(long key, int value)
      Specified by:
      set in interface WritableColumnSource<T>

    • set

      public void set(long key, long value)
      Specified by:
      set in interface WritableColumnSource<T>

    • set

      public void set(long key, short value)
      Specified by:
      set in interface WritableColumnSource<T>

    • remove

      public void remove(RowSet toRemove)

    • clearBlocks

      public void clearBlocks(RowSet blocksToClear, RowSet removeBlocks2, RowSet removeBlocks1, boolean empty)
      At the end of this cycle, clear the provided rowsets of blocks by releasing the blocks back to the recyclers.
      Parameters:
      blocksToClear - the lowest level blocks to clear
      removeBlocks2 - Block2 structures to clear from the Block1 structures
      removeBlocks1 - Block1 structures to clear from the Block0 structure
      empty - if the resulting table is empty

    • getSparseMemoryColumnSource

      public static <T> WritableColumnSource<T> getSparseMemoryColumnSource(Collection<T> data, Class<T> type)

    • getSparseMemoryColumnSource

      public static SparseArrayColumnSource<Byte> getSparseMemoryColumnSource(byte[] data)

    • getSparseMemoryColumnSource

      public static SparseArrayColumnSource<Character> getSparseMemoryColumnSource(char[] data)

    • getSparseMemoryColumnSource

      public static SparseArrayColumnSource<Double> getSparseMemoryColumnSource(double[] data)

    • getSparseMemoryColumnSource

      public static SparseArrayColumnSource<Float> getSparseMemoryColumnSource(float[] data)

    • getSparseMemoryColumnSource

      public static SparseArrayColumnSource<Integer> getSparseMemoryColumnSource(int[] data)

    • getSparseMemoryColumnSource

      public static SparseArrayColumnSource<Long> getSparseMemoryColumnSource(long[] data)

    • getInstantMemoryColumnSource

      public static WritableColumnSource<Instant> getInstantMemoryColumnSource(long[] data)

    • getSparseMemoryColumnSource

      public static SparseArrayColumnSource<Short> getSparseMemoryColumnSource(short[] data)

    • getSparseMemoryColumnSource

      public static <T> WritableColumnSource<T> getSparseMemoryColumnSource(Class<T> type)

    • getSparseMemoryColumnSource

      public static <T> WritableColumnSource<T> getSparseMemoryColumnSource(Class<T> type, Class<?> componentType)

    • getSparseMemoryColumnSource

      public static <T> WritableColumnSource<T> getSparseMemoryColumnSource(long size, Class<T> type)

    • getSparseMemoryColumnSource

      public static <T> WritableColumnSource<T> getSparseMemoryColumnSource(long size, Class<T> type, @Nullable @Nullable Class<?> componentType)

    • getSparseMemoryColumnSource

      public static ColumnSource<?> getSparseMemoryColumnSource(Object dataArray)

    • getPreferredChunkSize

      public int getPreferredChunkSize()
      Using a preferred chunk size of BLOCK_SIZE gives us the opportunity to directly return chunks from our data structure rather than copying data.

    • fillChunk

      public void fillChunk(@NotNull @NotNull ChunkSource.FillContext context, @NotNull @NotNull WritableChunk<? super Values> dest, @NotNull @NotNull RowSequence rowSequence)
      Description copied from interface: ChunkSource
      Populates the given destination chunk with data corresponding to the keys from the given RowSequence.
      Specified by:
      fillChunk in interface ChunkSource<T>
      Overrides:
      fillChunk in class AbstractColumnSource<T>
      Parameters:
      context - A context containing all mutable/state related data used in retrieving the Chunk.
      dest - The chunk to be populated according to rowSequence. No assumptions shall be made about the size of the chunk shall be made. The chunk will be populated from position [0,rowSequence.size()).
      rowSequence - An RowSequence representing the keys to be fetched

    • setNull

      public void setNull(RowSequence rowSequence)
      Specified by:
      setNull in interface WritableColumnSource<T>

    • fillChunkUnordered

      public void fillChunkUnordered(@NotNull @NotNull ChunkSource.FillContext context, @NotNull @NotNull WritableChunk<? super Values> dest, @NotNull @NotNull LongChunk<? extends RowKeys> keys)
      Description copied from interface: FillUnordered
      Populates a contiguous portion of the given destination chunk with data corresponding to the keys from the given LongChunk.

      It behaves as if the following code were executed: 

       destination.setSize(keys.size());
 for (int ii = 0; ii < keys.size(); ++ii) {
     destination.set(ii, get(keys.get(ii)));
 }
      Specified by:
      fillChunkUnordered in interface FillUnordered<T>
      Parameters:
      context - A context containing all mutable/state related data used in retrieving the Chunk.
      dest - The chunk to be populated according to keys
      keys - A chunk of individual, not assumed to be ordered keys to be fetched

    • fillPrevChunkUnordered

      public void fillPrevChunkUnordered(@NotNull @NotNull ChunkSource.FillContext context, @NotNull @NotNull WritableChunk<? super Values> dest, @NotNull @NotNull LongChunk<? extends RowKeys> keys)
      Description copied from interface: FillUnordered
      Populates a contiguous portion of the given destination chunk with prev data corresponding to the keys from the given LongChunk.

      It behaves as if the following code were executed: 

       destination.setSize(keys.size());
 for (int ii = 0; ii < keys.size(); ++ii) {
     destination.set(ii, getPrev(keys.get(ii)));
 }
      Specified by:
      fillPrevChunkUnordered in interface FillUnordered<T>
      Parameters:
      context - A context containing all mutable/state related data used in retrieving the Chunk.
      dest - The chunk to be populated according to keys
      keys - A chunk of individual, not assumed to be ordered keys to be fetched

    • makeFillFromContext

      public ChunkSink.FillFromContext makeFillFromContext(int chunkCapacity)
      Description copied from interface: WritableColumnSource
      Provide a default, empty ChunkSink.FillFromContext for use with our default WritableColumnSource.fillFromChunk(io.deephaven.engine.table.ChunkSink.FillFromContext, io.deephaven.chunk.Chunk<? extends io.deephaven.chunk.attributes.Values>, io.deephaven.engine.rowset.RowSequence).
      Specified by:
      makeFillFromContext in interface ChunkSink<T>
      Specified by:
      makeFillFromContext in interface WritableColumnSource<T>

    • fillFromChunk

      public void fillFromChunk(@NotNull @NotNull ChunkSink.FillFromContext context, @NotNull @NotNull Chunk<? extends Values> src, @NotNull @NotNull RowSequence rowSequence)
      Description copied from interface: WritableColumnSource
      Our default, inefficient, implementation. Inheritors who care should provide a better implementation.
      Specified by:
      fillFromChunk in interface ChunkSink<T>
      Specified by:
      fillFromChunk in interface WritableColumnSource<T>
      Parameters:
      context - A context containing all mutable/state related data used in writing the Chunk.
      src - The source of the data rowSequence
      rowSequence - An RowSequence representing the keys to be written

    • isImmutable

      public boolean isImmutable()
      Description copied from interface: ColumnSource
      Determine if this column source is immutable, meaning that the values at a given row key never change.
      Specified by:
      isImmutable in interface ColumnSource<T>
      Returns:
      true if the values at a given row key of the column source never change, false otherwise

    • setImmutable

      public void setImmutable()
      Description copied from interface: PossiblyImmutableColumnSource
      Set this column source as having an immutable result.
      Specified by:
      setImmutable in interface PossiblyImmutableColumnSource

    • providesFillUnordered

      public boolean providesFillUnordered()
      Description copied from interface: FillUnordered
      Returns true if this column source can efficiently provide an unordered fill. If this method returns false, then fillChunkUnordered and fillPrevChunkUnordered may throw an UnsupportedOperationException.
      Specified by:
      providesFillUnordered in interface FillUnordered<T>
      Returns:
      if this column source can provide an unordered fill

    • estimateSize

      @TestUseOnly public abstract long estimateSize()
      Return an estimate of the heap size taken by the current values within this sparse array source.

      Only leaf nodes and the size arrays of references are included in this estimate. Intermediate objects are ignored, and an array of references is assumed to take 8 bytes per element with no overhead.

      Returns:
      an estimate of the size of this column source's current data