Working with Live vs Static Data

	Watch the video below titled, "Working with Static/Historical Data vs. Live Ticking Data"

Syntax Differences

Live tables and static tables in Deephaven support identical functionality. Every table operation that can be applied to a static table can also be applied to a live table. The only difference in syntax is when initially retrieving the data.

db.t("<namespace>", "<table>") // Retrieves a static table. The result table will never change.

db.i("<namespace>", "<table>") // Retrieves a live table. As new rows are imported into the system, they are appended to the end of the result table.

Note: The examples above do not include a filter on the partitioning column. However, you should always filter on the partitioning column when pulling up a table. For more information, see Accessing Data.

Performance Considerations

When working with live data, the query engine must update the results of query operations as data changes in each operation's source tables. The results of all query operations update to reflect changing data, including:

Filters (used with where() )
Formulas (such as used with update() or updateView())
Aggregations (such as sumBy(), avgBy(), etc.)
Joins (such as naturalJoin() and aj(), etc.)
Sorts (with sort() and sortDescending())

When running a complex query on fast-changing data, the amount of processing required to update the results of each operation may overwhelm the query server's hardware.

An overwhelmed query processor results in markedly decreased performance. A worker struggling to keep up with changing data will present clear signs in the form of a degraded user experience such as the following:

Sorting and filtering tables will take longer than expected
New queries run from an interactive console will take longer than expected
Tables will refresh infrequently (far less often than once per second). As a result, the data in tables may not be acceptably current.

To learn more about best practices and methods for working with live data, see Common Causes of Performance Issues with Real-Time Data.

Last Updated: 20 August 2019 09:54 -06:00 UTC Deephaven v.1.20180917