Package io.deephaven.kafka.ingest

Class JsonConsumerRecordToTableWriterAdapter.Builder

java.lang.Object
io.deephaven.kafka.ingest.JsonConsumerRecordToTableWriterAdapter.Builder
Enclosing class:
JsonConsumerRecordToTableWriterAdapter
public static class JsonConsumerRecordToTableWriterAdapter.Builder extends Object
A builder to map key and value fields to table columns.

  • Constructor Details

    • Builder

      public Builder()

  • Method Details

    • kafkaPartitionColumnName

      @ScriptApi @NotNull public JsonConsumerRecordToTableWriterAdapter.Builder kafkaPartitionColumnName(@NotNull String kafkaPartitionColumnName)
      Set the name of the column which stores the Kafka partition identifier of the record.
      Parameters:
      kafkaPartitionColumnName - the name of the column in the output table
      Returns:
      this builder

    • kafkaKeyColumnName

      @NotNull @ScriptApi public JsonConsumerRecordToTableWriterAdapter.Builder kafkaKeyColumnName(@NotNull String kafkaKeyColumnName)
      Set the name of the column which stores the Kafka key of the record. Key must be a String.
      Parameters:
      kafkaKeyColumnName - the name of the column in the output table
      Returns:
      this builder

    • offsetColumnName

      @ScriptApi @NotNull public JsonConsumerRecordToTableWriterAdapter.Builder offsetColumnName(@NotNull String offsetColumnName)
      Set the name of the column which stores the Kafka offset of the record.
      Parameters:
      offsetColumnName - the name of the column in the output table
      Returns:
      this builder

    • recvTimeColumnName

      @ScriptApi @NotNull public JsonConsumerRecordToTableWriterAdapter.Builder recvTimeColumnName(@NotNull String recvTimeColumnName)
      Set the name of the column which stores the time that the record was received by the ingester.
      Parameters:
      recvTimeColumnName - the name of the column in the output table
      Returns:
      this builder

    • timestampColumnName

      @ScriptApi @NotNull public JsonConsumerRecordToTableWriterAdapter.Builder timestampColumnName(@NotNull String timestampColumnName)
      Set the name of the column which stores the Kafka timestamp of the record.
      Parameters:
      timestampColumnName - the name of the column in the output table
      Returns:
      this builder

    • addFieldParallel

      @ScriptApi @NotNull public JsonConsumerRecordToTableWriterAdapter.Builder addFieldParallel(@NotNull String column, @NotNull String field)
      Identifies an array field within the message, which generates one row of output for each value in the array. If an array element is null, then it is ignored. If all array elements in this table are null, then there will be no row written unless the builder is instructed to `.allowNullArrays(true)`
      Parameters:
      column - the output column
      field - the input field
      Returns:
      this builder

    • addNestedFieldParallel

      @ScriptApi @NotNull public JsonConsumerRecordToTableWriterAdapter.Builder addNestedFieldParallel(@NotNull JsonConsumerRecordToTableWriterAdapter.Builder.NestedBuilder nestedBuilder, @NotNull String field)
      Add a nested parallel array field to this table. All array fields must be parallel (i.e. the same size). Each element of the array produces a new row. Non-array fields are replicated to each output row from the message. You may have multiply nested fields, but may not have parallel fields within your nested field. If an array element is null, then it is ignored. If all array elements in this table are null, then there will be no row written unless the builder is instructed to `.allowNullArrays(true)`
      Parameters:
      nestedBuilder - the builder for the nested field
      field - the input field
      Returns:
      this builder

    • addNestedField

      @ScriptApi @NotNull public JsonConsumerRecordToTableWriterAdapter.Builder addNestedField(@NotNull JsonConsumerRecordToTableWriterAdapter.Builder.NestedBuilder nestedBuilder, @NotNull String field)
      Add a nested field to this table. A nested field is itself a JSON object, with columns outputs defined by the builder. You may have arbitrarily deeply nested fields. The nested builder may not have auto value mapping enabled or include any parallel arrays. If a nested field is null, and null values are permitted then all columns derived from the nested field will be null.
      Parameters:
      nestedBuilder - the builder for the nested field
      field - the input field
      Returns:
      this builder

    • allowUnmapped

      @ScriptApi @NotNull public JsonConsumerRecordToTableWriterAdapter.Builder allowUnmapped(@NotNull String allowUnmapped)
      Allow the column with the given name to be unmapped in the output. Unmapped columns will have no setter, and will thus be null filled in the output.
      Parameters:
      allowUnmapped - the column name to allow to be unmapped
      Returns:
      this builder

    • addColumnToValueField

      @ScriptApi @NotNull public JsonConsumerRecordToTableWriterAdapter.Builder addColumnToValueField(@NotNull String column, @NotNull String field)
      Map a column to a field in the value record.
      Parameters:
      column - the name of the output column
      field - the name of the field in the value record
      Returns:
      this builder

    • addColumnToValueFunction

      @ScriptApi @NotNull public JsonConsumerRecordToTableWriterAdapter.Builder addColumnToValueFunction(@NotNull String column, @NotNull Function<JsonRecord,?> function)
      Map a column to a function of the value record.
      Parameters:
      column - the name of the output column
      function - the function to apply to the value record
      Returns:
      this builder

    • allowMissingKeys

      @ScriptApi @NotNull public JsonConsumerRecordToTableWriterAdapter.Builder allowMissingKeys(boolean allowMissingKeys)
      If allowMissingKeys is set, then a request for a value using a key that is not found in the record will receive a null value. Otherwise, an exception is thrown when a key is not found in the record.
      Parameters:
      allowMissingKeys - to allow quietly continuing if requested value's key is not in the current record.
      Returns:
      this builder

    • allowNullArrays

      @ScriptApi @NotNull public JsonConsumerRecordToTableWriterAdapter.Builder allowNullArrays(boolean allowNullArrays)
      If allowNullArrays is set, then a Json message which does not have any of the builder-defined arrays populated will output a single row, where each of the array-populated columns will be null. If this flag is not set, then a Json message which does not have any of the builder-defined arrays populated will be skipped, and not output any rows
      Parameters:
      allowNullArrays - to allow writing of a single row when all builder-defined arrays are missing
      Returns:
      this builder

    • allowNullValues

      @ScriptApi @NotNull public JsonConsumerRecordToTableWriterAdapter.Builder allowNullValues(boolean allowNullValues)
      If allowNullValues is set, then records with a null value will have their columns null filled; otherwise an exception is thrown on receipt of a null value. If no value fields are set, then no columns are taken from the value; so this flag has no effect.
      Parameters:
      allowNullValues - if null values are allowed
      Returns:
      this builder

    • autoValueMapping

      @ScriptApi @NotNull public JsonConsumerRecordToTableWriterAdapter.Builder autoValueMapping(boolean autoValueMapping)
      If autoValueMapping is set, then all unused columns are mapped to a field of the same name in the generic record.
      Parameters:
      autoValueMapping - if unused columns should be automatically mapped to a field of the same name
      Returns:
      this builder

    • caseInsensitiveSearch

      @ScriptApi @NotNull public JsonConsumerRecordToTableWriterAdapter.Builder caseInsensitiveSearch(boolean caseInsensitiveSearch)
      If autoValueMapping(boolean) is set to true, then all unused columns are mapped to a field of the same name in the generic record without regard to case. For example, a json record field of widgetId would be matched to a table column named WidgetID.
      Parameters:
      caseInsensitiveSearch - should a case-insensitive search for matching columns be performed?
      Returns:
      this builder

    • parallelParsers

      @ScriptApi @NotNull public JsonConsumerRecordToTableWriterAdapter.Builder parallelParsers(int parallelThreads)
      Define maximum number of dedicated threads to use for parsing of inbound messages.
      Parameters:
      parallelThreads - maximum number of dedicated threads to use for parsing inbound messages
      Returns:
      this builder

    • setFilter

      @ScriptApi @NotNull public JsonConsumerRecordToTableWriterAdapter.Builder setFilter(BiPredicate<org.apache.kafka.clients.consumer.ConsumerRecord<?,?>,JsonRecord> recordPredicate)
      If a filter is added, then all records will be passed through the predicate before being consumed. Only records that match the predicate will be ingested.
      Parameters:
      recordPredicate - returns true if the record should be ingested, otherwise false
      Returns:
      this builder

    • setParseOnce

      @ScriptApi @NotNull public JsonConsumerRecordToTableWriterAdapter.Builder setParseOnce(@Nullable Function<JsonRecord,Object> parseOnce)
      Allows builder to specify an object which may be accessed multiple times without requiring re-parsing of a specific JSON field
      Parameters:
      parseOnce - a function which returns an object to be stored
      Returns:
      this builder