Package io.deephaven.extensions.barrage.util

Class BarrageUtil

java.lang.Object
io.deephaven.extensions.barrage.util.BarrageUtil
public class BarrageUtil extends Object

  • Field Details

    • DEFAULT_SUBSCRIPTION_OPTIONS

      public static final BarrageSubscriptionOptions DEFAULT_SUBSCRIPTION_OPTIONS

    • DEFAULT_SNAPSHOT_OPTIONS

      public static final BarrageSnapshotOptions DEFAULT_SNAPSHOT_OPTIONS

    • FLATBUFFER_MAGIC

      public static final long FLATBUFFER_MAGIC
      See Also:

    • TARGET_SNAPSHOT_PERCENTAGE

      public static final double TARGET_SNAPSHOT_PERCENTAGE

    • MIN_SNAPSHOT_CELL_COUNT

      public static final long MIN_SNAPSHOT_CELL_COUNT

    • MAX_SNAPSHOT_CELL_COUNT

      public static final long MAX_SNAPSHOT_CELL_COUNT

    • NANO_SINCE_EPOCH_TYPE

      public static final org.apache.arrow.vector.types.pojo.ArrowType.Timestamp NANO_SINCE_EPOCH_TYPE
      Note that arrow's wire format states that Timestamps without timezones are not UTC -- that they are no timezone at all. It's very important that we mark these times as UTC.

    • NANO_DURATION_TYPE

      public static final org.apache.arrow.vector.types.pojo.ArrowType.Duration NANO_DURATION_TYPE
      By default we'll use nanosecond resolution for Duration columns.

    • TABLE_ATTRIBUTE_IS_FLAT

      public static final String TABLE_ATTRIBUTE_IS_FLAT
      The name of the attribute that indicates that a table is flat.
      See Also:

    • ATTR_DH_PREFIX

      public static final String ATTR_DH_PREFIX
      The Apache Arrow metadata prefix for Deephaven attributes.
      See Also:

    • ATTR_TYPE_TAG

      public static final String ATTR_TYPE_TAG
      The deephaven metadata tag to indicate the deephaven column type.
      See Also:

    • ATTR_COMPONENT_TYPE_TAG

      public static final String ATTR_COMPONENT_TYPE_TAG
      The deephaven metadata tag to indicate the deephaven column component type.
      See Also:

    • ATTR_PROTO_METADATA_TAG

      public static final String ATTR_PROTO_METADATA_TAG
      The deephaven metadata tag to indicate the input table information.
      See Also:

  • Constructor Details

    • BarrageUtil

      public BarrageUtil()

  • Method Details

    • createSubscriptionRequestMetadataBytes

      public static byte[] createSubscriptionRequestMetadataBytes(@NotNull @org.jetbrains.annotations.NotNull byte[] ticketId, @Nullable @Nullable BarrageSubscriptionOptions options)
      Create a subscription request payload to be sent via DoExchange.
      Parameters:
      ticketId - the ticket id of the table to subscribe to
      options - the barrage options
      Returns:
      the subscription request payload

    • createSubscriptionRequestMetadataBytes

      public static byte[] createSubscriptionRequestMetadataBytes(@NotNull @org.jetbrains.annotations.NotNull byte[] ticketId, @Nullable @Nullable BarrageSubscriptionOptions options, @Nullable @Nullable RowSet viewport, @Nullable @Nullable BitSet columns, boolean reverseViewport)
      Create a subscription request payload to be sent via DoExchange.
      Parameters:
      ticketId - the ticket id of the table to subscribe to
      options - the barrage options
      viewport - the viewport to subscribe to
      columns - the columns to subscribe to
      reverseViewport - whether to reverse the viewport
      Returns:
      the subscription request payload

    • createSubscriptionRequestMetadataBytes

      public static byte[] createSubscriptionRequestMetadataBytes(@NotNull @org.jetbrains.annotations.NotNull byte[] ticketId, @Nullable @Nullable BarrageSubscriptionOptions options, @Nullable @Nullable ByteBuffer viewportBuffer, @Nullable @org.jetbrains.annotations.Nullable byte[] columns, boolean reverseViewport, byte requestType)
      Create a subscription request payload to be sent via DoExchange.
      Parameters:
      ticketId - the ticket id of the table to subscribe to
      options - the barrage options
      viewportBuffer - the viewport to subscribe to, already converted to a ByteBuffer
      columns - the columns to subscribe to
      reverseViewport - whether to reverse the viewport
      requestType - the type of the request
      Returns:
      the subscription request payload

    • createSnapshotRequestMetadataBytes

      public static byte[] createSnapshotRequestMetadataBytes(@NotNull @org.jetbrains.annotations.NotNull byte[] ticketId, @Nullable @Nullable BarrageSnapshotOptions options)
      Create a snapshot request payload to be sent via DoExchange.
      Parameters:
      ticketId - the ticket id of the table to subscribe to
      options - the barrage options
      Returns:
      the subscription request payload

    • createSnapshotRequestMetadataBytes

      public static byte[] createSnapshotRequestMetadataBytes(@NotNull @org.jetbrains.annotations.NotNull byte[] ticketId, @Nullable @Nullable BarrageSnapshotOptions options, @Nullable @Nullable RowSet viewport, @Nullable @Nullable BitSet columns, boolean reverseViewport)
      Create a subscription request payload to be sent via DoExchange.
      Parameters:
      ticketId - the ticket id of the table to subscribe to
      options - the barrage options
      viewport - the viewport to subscribe to
      columns - the columns to subscribe to
      reverseViewport - whether to reverse the viewport
      Returns:
      the subscription request payload

    • createSerializationOptionsMetadataBytes

      public static byte[] createSerializationOptionsMetadataBytes(@NotNull @org.jetbrains.annotations.NotNull byte[] ticketId, @Nullable @Nullable BarrageSubscriptionOptions options)
      Create a snapshot request payload to be sent via DoExchange.
      Parameters:
      ticketId - the ticket id of the table to subscribe to
      options - the barrage options
      Returns:
      the subscription request payload

    • schemaBytesFromTable

      public static com.google.protobuf.ByteString schemaBytesFromTable(@NotNull @NotNull Table table)

    • schemaBytesFromTableDefinition

      public static com.google.protobuf.ByteString schemaBytesFromTableDefinition(@NotNull @NotNull TableDefinition tableDefinition, @NotNull @NotNull Map<String,Object> attributes, boolean isFlat)

    • schemaFromTable

      public static org.apache.arrow.vector.types.pojo.Schema schemaFromTable(@NotNull @NotNull Table table)

    • toSchema

      public static org.apache.arrow.vector.types.pojo.Schema toSchema(TableDefinition definition, Map<String,Object> attributes, boolean isFlat)

    • schemaBytes

      public static com.google.protobuf.ByteString schemaBytes(@NotNull @NotNull ToIntFunction<com.google.flatbuffers.FlatBufferBuilder> schemaPayloadWriter)

    • makeTableSchemaPayload

      public static int makeTableSchemaPayload(@NotNull @NotNull com.google.flatbuffers.FlatBufferBuilder builder, @NotNull @NotNull BarrageOptions options, @NotNull @NotNull TableDefinition tableDefinition, @NotNull @NotNull Map<String,Object> attributes, boolean isFlat)

    • makeSchema

      public static org.apache.arrow.vector.types.pojo.Schema makeSchema(@NotNull @NotNull BarrageOptions options, @NotNull @NotNull TableDefinition tableDefinition, @NotNull @NotNull Map<String,Object> attributes, boolean isFlat)

    • attributesToMetadata

      @NotNull public static @NotNull Map<String,String> attributesToMetadata(@NotNull @NotNull Map<String,Object> attributes)

    • attributesToMetadata

      @NotNull public static @NotNull Map<String,String> attributesToMetadata(@NotNull @NotNull Map<String,Object> attributes, boolean isFlat)

    • columnDefinitionsToFields

      public static Stream<org.apache.arrow.vector.types.pojo.Field> columnDefinitionsToFields(@NotNull @NotNull Map<String,String> columnDescriptions, @Nullable @Nullable InputTableUpdater inputTableUpdater, @NotNull @NotNull TableDefinition tableDefinition, @NotNull @NotNull Collection<ColumnDefinition<?>> columnDefinitions, @NotNull @NotNull Function<String,Map<String,String>> fieldMetadataFactory, @NotNull @NotNull Map<String,Object> attributes)

    • columnDefinitionsToFields

      public static Stream<org.apache.arrow.vector.types.pojo.Field> columnDefinitionsToFields(@NotNull @NotNull Map<String,String> columnDescriptions, @Nullable @Nullable InputTableUpdater inputTableUpdater, @NotNull @NotNull TableDefinition tableDefinition, @NotNull @NotNull Collection<ColumnDefinition<?>> columnDefinitions, @NotNull @NotNull Function<String,Map<String,String>> fieldMetadataFactory, @NotNull @NotNull Map<String,Object> attributes, boolean columnsAsList)

    • putMetadata

      public static void putMetadata(Map<String,String> metadata, String key, String value)

    • getDefaultType

      public static BarrageTypeInfo<org.apache.arrow.vector.types.pojo.Field> getDefaultType(@NotNull @NotNull org.apache.arrow.vector.types.pojo.Field field)

    • convertTableDefinition

      public static TableDefinition convertTableDefinition(@NotNull @NotNull ExportedTableCreationResponse response)

    • convertArrowSchema

      public static BarrageUtil.ConvertedArrowSchema convertArrowSchema(@NotNull @NotNull ExportedTableCreationResponse response)

    • convertArrowSchema

      public static BarrageUtil.ConvertedArrowSchema convertArrowSchema(@NotNull org.apache.arrow.flatbuf.Schema schema)

    • convertArrowSchema

      public static BarrageUtil.ConvertedArrowSchema convertArrowSchema(@NotNull org.apache.arrow.flatbuf.Schema schema, @Nullable @Nullable BarrageOptions options)

    • convertArrowSchema

      public static BarrageUtil.ConvertedArrowSchema convertArrowSchema(org.apache.arrow.vector.types.pojo.Schema schema)

    • convertArrowSchema

      public static BarrageUtil.ConvertedArrowSchema convertArrowSchema(org.apache.arrow.vector.types.pojo.Schema schema, BarrageOptions options)

    • arrowFieldFor

      public static org.apache.arrow.vector.types.pojo.Field arrowFieldFor(String name, Class<?> type, Class<?> componentType, Map<String,String> metadata, boolean columnAsList)

    • flatbufFieldFor

      public static org.apache.arrow.flatbuf.Field flatbufFieldFor(ColumnDefinition<?> columnDefinition, Map<String,String> metadata)

    • flatbufFieldFor

      public static org.apache.arrow.flatbuf.Field flatbufFieldFor(String name, Class<?> type, Class<?> componentType, Map<String,String> metadata)

    • createAndSendStaticSnapshot

      public static void createAndSendStaticSnapshot(BarrageMessageWriter.Factory bmwFactory, BaseTable<?> table, BitSet columns, RowSet viewport, boolean reverseViewport, BarrageSnapshotOptions snapshotRequestOptions, io.grpc.stub.StreamObserver<BarrageMessageWriter.MessageView> listener, BarragePerformanceLog.SnapshotMetricsHelper metrics)

    • targetSnapshotTime

      public static long targetSnapshotTime(UpdateGraph updateGraph)
      Very simplistic logic to take the last snapshot and extrapolate max number of rows that will not exceed the target update graph processing time percentage.
      Parameters:
      updateGraph - the update graph for the table
      Returns:
      the target snapshot time, in nanos

    • createAndSendSnapshot

      public static void createAndSendSnapshot(BarrageMessageWriter.Factory bwmFactory, BaseTable<?> table, BitSet columns, RowSet viewport, boolean reverseViewport, BarrageSnapshotOptions options, io.grpc.stub.StreamObserver<BarrageMessageWriter.MessageView> listener, BarragePerformanceLog.SnapshotMetricsHelper metrics)