Package io.deephaven.engine.table.impl.select

Class AutoTuningIncrementalReleaseFilter

java.lang.Object
io.deephaven.util.referencecounting.ReferenceCounted
io.deephaven.engine.liveness.ReferenceCountedLivenessReferent
io.deephaven.engine.liveness.ReferenceCountedLivenessNode
io.deephaven.engine.liveness.LivenessArtifact
io.deephaven.engine.table.impl.select.WhereFilterLivenessArtifactImpl
io.deephaven.engine.table.impl.select.BaseIncrementalReleaseFilter
io.deephaven.engine.table.impl.select.AutoTuningIncrementalReleaseFilter
All Implemented Interfaces:
ConcurrencyControl<Filter>, Expression, Filter, LogOutputAppendable, LivenessManager, LivenessNode, LivenessReferent, WhereFilter, NotificationQueue.Dependency, Serializable, Runnable
public class AutoTuningIncrementalReleaseFilter extends BaseIncrementalReleaseFilter
Filter that releases the required number of rows from a table to saturate the UGP cycle. The source table must be static.

The table has an initial size, which can be thought of as the size during query initialization. There is an initial number of rows that are released, which is then used to tune the number of rows to release on the subsequent cycle.

You must invoke the BaseIncrementalReleaseFilter.start() method to begin producing rows.

The targetFactor parameter is multiplied by the UGP's targetCycle. This allows you to determine how busy you want the UGP to be. For example a factor of 1, will attempt to hit the target cycle exactly. A target of 0.5 should result an UGP ratio of about 50%. A factor of 10 would mean that the system will extend beyond the target cycle time, coalesce updates accordingly and have a ratio that is nearly 100%.

The time the rows are released is recorded, and a terminal notification is enqueued to record the end of the cycle. On each cycle, the number of rows per second is computed; and then the number of rows released is the UGP's target cycle multiplied by the rows per second multiplied by the target factor.

The AutotuningIncrementalReleaseFilter can be used to benchmark how many rows of data a query can process. In its simplest form we can measure how many rows a lastBy statement can process. For example: 

 import io.deephaven.engine.table.impl.select.AutoTuningIncrementalReleaseFilter

 quotes = engine.t("FeedOS", "EquityQuoteL1").where("Date=lastBusinessDateNy()")
 filter=new AutoTuningIncrementalReleaseFilter(10000, 10000, 1)
 quotesFiltered = quotes.where(filter)
 currentQuote = quotesFiltered.lastBy("LocalCodeStr").update("Mid=(Bid + Ask)/2")
Produces a currentQuote table, and you can view the Log tab to determine how many rows per second were processed. The summary is sent to the WARN level: 
 12:55:49.985 WARN Completed release 6.97 seconds, rows=19630961, rows/second=2,817,053.86
If verbose mode is enabled, progress is displayed for each cycle at the INFO level.

You may specify a StreamLoggerImpl() to send the data to STDOUT, as follows: 

 import io.deephaven.engine.table.impl.select.AutoTuningIncrementalReleaseFilter

 quotes = engine.t("FeedOS", "EquityQuoteL1").where("Date=lastBusinessDateNy()")
 logger = new io.deephaven.io.logger.StreamLoggerImpl()
 filterQuotes=new AutoTuningIncrementalReleaseFilter(logger, 10000, 10000, 1.0d, true)
 quotesFiltered = quotes.where(filterQuotes)
 currentQuote = quotesFiltered.lastBy("LocalCodeStr").update("Mid=(Bid + Ask)/2")
 filterQuotes.start()

The verbose information and the final report are easily visible on your console.

The AutotuningIncrementalReleaseFilter is best suited for queries that have a single source table with arbitrary amounts of processing on that table. Multiple incremental release filters may be combined, and each filter will report the number of rows that were released per second, however the data is not synchronized between tables and it is not possible to differentiate which table is contributing more to the query's load without examining the performance tables. You may need to adjust the initial size parameters so that one table does not complete processing before another. 

 import io.deephaven.engine.table.impl.select.AutoTuningIncrementalReleaseFilter

 quotes = engine.t("FeedOS", "EquityQuoteL1").where("Date=lastBusinessDateNy()")
 trades = engine.t("FeedOS", "EquityTradeL1").where("Date=lastBusinessDateNy()")
 filterQuotes=new AutoTuningIncrementalReleaseFilter(10000, 10000, 1, true)
 quotesFiltered = quotes.where(filterQuotes)
 filterTrades=new AutoTuningIncrementalReleaseFilter(10000, 10000, 1, true)
 tradesFiltered = trades.where(filterTrades)

 decorated = tradesFiltered.aj(quotesFiltered, "LocalCodeStr,MarketTimestamp", "QuoteTime=MarketTimestamp,Bid,BidSize,Ask,AskSize")

 filterTrades.start()
 filterQuotes.start()
See Also:

  • Constructor Details

    • AutoTuningIncrementalReleaseFilter

      @ScriptApi public AutoTuningIncrementalReleaseFilter(long initialSize, long initialRelease, double targetFactor)
      Create an auto tuning release filter using a real time clock, without printing on each cycle.
      Parameters:
      initialSize - the initial table size
      initialRelease - the initial incremental update; after the first cycle the rows per second is calculated based on the duration of the last cycle and the number of rows released by this filter
      targetFactor - the multiple of the UGP cycle we should aim for

    • AutoTuningIncrementalReleaseFilter

      @ScriptApi public AutoTuningIncrementalReleaseFilter(Logger logger, long initialSize, long initialRelease, double targetFactor)
      Create an auto tuning release filter using a real time clock, without printing on each cycle.
      Parameters:
      logger - the logger the final row/second calculations to
      initialSize - the initial table size
      initialRelease - the initial incremental update; after the first cycle the rows per second is calculated based on the duration of the last cycle and the number of rows released by this filter
      targetFactor - the multiple of the UGP cycle we should aim for

    • AutoTuningIncrementalReleaseFilter

      @ScriptApi public AutoTuningIncrementalReleaseFilter(long initialSize, long initialRelease, double targetFactor, boolean verbose)
      Create an auto tuning release filter using a real time clock.
      Parameters:
      initialSize - the initial table size
      initialRelease - the initial incremental update; after the first cycle the rows per second is calculated based on the duration of the last cycle and the number of rows released by this filter
      targetFactor - the multiple of the UGP cycle we should aim for
      verbose - whether information should be printed on each UGP cycle describing the current rate and number of rows released

    • AutoTuningIncrementalReleaseFilter

      @ScriptApi public AutoTuningIncrementalReleaseFilter(Logger logger, long initialSize, long initialRelease, double targetFactor, boolean verbose)
      Create an auto tuning release filter using a real time clock.
      Parameters:
      logger - the logger to report progress (if verbose is set) and the final row/second calculations
      initialSize - the initial table size
      initialRelease - the initial incremental update; after the first cycle the rows per second is calculated based on the duration of the last cycle and the number of rows released by this filter
      targetFactor - the multiple of the UGP cycle we should aim for
      verbose - whether information should be printed on each UGP cycle describing the current rate and number of rows released

  • Method Details

    • run

      public void run()
      Specified by:
      run in interface Runnable
      Overrides:
      run in class BaseIncrementalReleaseFilter

    • filter

      @NotNull public @NotNull WritableRowSet filter(@NotNull @NotNull RowSet selection, @NotNull @NotNull RowSet fullSet, @NotNull @NotNull Table table, boolean usePrev)
      Description copied from interface: WhereFilter
      Filter selection to only matching rows.
      Specified by:
      filter in interface WhereFilter
      Overrides:
      filter in class BaseIncrementalReleaseFilter
      Parameters:
      selection - the indices that should be filtered. The selection must be a subset of fullSet, and may include rows that the engine determines need not be evaluated to produce the result. Implementations may not mutate or close selection.
      fullSet - the complete RowSet of the table to filter. The fullSet is used for calculating variables like "i" or "ii". Implementations may not mutate or close fullSet.
      table - the table to filter
      usePrev - true if previous values should be used. Implementing previous value filtering is optional, and a WhereFilter.PreviousFilteringNotSupported exception may be thrown. If a PreviousFiltering exception is thrown, then the caller must acquire the PeriodicUpdateGraph lock.
      Returns:
      The subset of selection accepted by this filter; ownership passes to the caller