Package com.illumon.iris.db.v2

Class HierarchicalTable

java.lang.Object
com.illumon.util.referencecounting.ReferenceCounted
com.illumon.iris.db.util.liveness.ReferenceCountedLivenessNode
com.illumon.iris.db.util.liveness.LivenessArtifact
com.illumon.iris.db.v2.BaseTable
com.illumon.iris.db.v2.QueryTable
com.illumon.iris.db.v2.HierarchicalTable
All Implemented Interfaces:
com.fishlib.base.log.LogOutputAppendable, NotificationQueue.Dependency, Deflatable<Table>, Table, LivenessManager, LivenessNode, LivenessReferent, LongSizedDataStructure, DynamicNode, DynamicTable, NotificationStepReceiver, NotificationStepSource, SystemicObject, Serializable
public class HierarchicalTable extends QueryTable
This class is an extension of QueryTable that overrides many methods from Table which are not valid to perform on Hierarchical tables (Table.treeTable(String, String) and Table.rollup(ComboAggregateFactory, String...)).
See Also:

  • Method Details

    • getInfo

      public HierarchicalTableInfo getInfo()
      Get the HierarchicalTableInfo associated with this table.
      Returns:
      the info for this table

    • getSourceTable

      public Table getSourceTable()
      Get the table on which this hierarchical table was created from.
      Returns:
      the source table

    • withFlatFilterColumns

      public HierarchicalTable withFlatFilterColumns(String... columns)
      Create a shallow copy of this table that will apply filters to the specified columns at each level, instead of filtering the base table and re-treeing.
      Parameters:
      columns - the columns to filter flat.
      Returns:
      a copy of this table that will filter the specified columns per level.

    • copy

      public HierarchicalTable copy(boolean copyAttributes)
      Overrides:
      copy in class QueryTable

    • getRawRootTable

      public Table getRawRootTable()
      Get the table that is the root of the hierarchy
      Returns:
      the root of the hierarchy

    • formatColumns

      public HierarchicalTable formatColumns(String... columnFormats)

    • sort

      public HierarchicalTable sort(SortPair... columnsToSortBy)
      Specified by:
      sort in interface Table
      Overrides:
      sort in class QueryTable

    • slice

      public Table slice(long firstRowInclusive, long lastRowExclusive)
      Description copied from interface: Table
      Extracts a subset of a table by row position. If both firstPosition and lastPosition are positive, then the rows are counted from the beginning of the table. The firstPosition is inclusive, and the lastPosition is exclusive. The Table.head(long)(N) call is equivalent to slice(0, N). The firstPosition must be less than or equal to the lastPosition. If firstPosition is positive and lastPosition is negative, then the firstRow is counted from the beginning of the table, inclusively. The lastPosition is counted from the end of the table. For example, slice(1, -1) includes all rows but the first and last. If the lastPosition would be before the firstRow, the result is an emptyTable. If firstPosition is negative, and lastPosition is zero, then the firstRow is counted from the end of the table, and the end of the slice is the size of the table. slice(-N, 0) is equivalent to Table.tail(long)(N). If the firstPosition is nega tive and the lastPosition is negative, they are both counted from the end of the table. For example, slice(-2, -1) returns the second to last row of the table.
      Specified by:
      slice in interface Table
      Overrides:
      slice in class QueryTable
      Parameters:
      firstRowInclusive - the first position to include in the result
      lastRowExclusive - the last position to include in the result
      Returns:
      a new Table, which is the request subset of rows from the original table

    • head

      public Table head(long size)
      Specified by:
      head in interface Table
      Overrides:
      head in class QueryTable

    • tail

      public Table tail(long size)
      Specified by:
      tail in interface Table
      Overrides:
      tail in class QueryTable

    • leftJoin

      public Table leftJoin(Table table, MatchPair[] columnsToMatch, MatchPair[] columnsToAdd)
      Description copied from interface: Table
      Augments this table with array columns of right-hand side matches.

      The leftJoin() method returns the exact rows of the leftTable. The data joined in from the rightTable are grouped into arrays of data. When no right-hand side data is found, the right hand columns are null.

      Specified by:
      leftJoin in interface Table
      Overrides:
      leftJoin in class QueryTable
      Parameters:
      table - The right side table on the join.
      columnsToMatch - An array of match pair conditions ("leftColumn=rightColumn" or "columnFoundInBoth")
      columnsToAdd - An array of the columns from the right side be added to the left side as a result of the match. If empty, then all columns from the right table are added to the result.
      Returns:
      a new table joined according to the specification in columnsToMatch and columnsToAdd

    • exactJoin

      public Table exactJoin(Table table, MatchPair[] columnsToMatch, MatchPair[] columnsToAdd)
      Description copied from interface: Table
      Identical to naturalJoin, but fail if the right side does not produce a match.
      Specified by:
      exactJoin in interface Table
      Overrides:
      exactJoin in class QueryTable

    • dropColumns

      public Table dropColumns(String... columnNames)
      Specified by:
      dropColumns in interface Table
      Overrides:
      dropColumns in class QueryTable

    • renameColumns

      public Table renameColumns(MatchPair... pairs)
      Specified by:
      renameColumns in interface Table
      Overrides:
      renameColumns in class QueryTable

    • aj

      public Table aj(Table rightTable, MatchPair[] columnsToMatch, MatchPair[] columnsToAdd, Table.AsOfMatchRule asOfMatchRule)
      Description copied from interface: Table
      Looks up the columns in the rightTable that meet the match conditions in the columnsToMatch list. Matching is done exactly for the first n-1 columns and via a binary search for the last match pair. The columns of the original table are returned intact, together with the columns from rightTable defined in a comma separated list "columnsToAdd"
      Specified by:
      aj in interface Table
      Overrides:
      aj in class QueryTable
      Parameters:
      rightTable - The right side table on the join.
      columnsToMatch - A comma separated list of match conditions ("leftColumn=rightColumn" or "columnFoundInBoth")
      columnsToAdd - A comma separated list with the columns from the left side that need to be added to the right side as a result of the match.
      Returns:
      a new table joined according to the specification in columnsToMatch and columnsToAdd

    • raj

      public Table raj(Table rightTable, MatchPair[] columnsToMatch, MatchPair[] columnsToAdd, Table.AsOfMatchRule asOfMatchRule)
      Description copied from interface: Table
      Just like .aj(), but the matching on the last column is in reverse order, so that you find the row after the given timestamp instead of the row before.

      Looks up the columns in the rightTable that meet the match conditions in the columnsToMatch list. Matching is done exactly for the first n-1 columns and via a binary search for the last match pair. The columns of the original table are returned intact, together with the columns from rightTable defined in a comma separated list "columnsToAdd"

      Specified by:
      raj in interface Table
      Overrides:
      raj in class QueryTable
      Parameters:
      rightTable - The right side table on the join.
      columnsToMatch - A comma separated list of match conditions ("leftColumn=rightColumn" or "columnFoundInBoth")
      columnsToAdd - A comma separated list with the columns from the left side that need to be added to the right side as a result of the match.
      Returns:
      a new table joined according to the specification in columnsToMatch and columnsToAdd

    • naturalJoin

      public Table naturalJoin(Table rightTable, MatchPair[] columnsToMatch, MatchPair[] columnsToAdd)
      Description copied from interface: Table
      Augment this table with zero or one row from the right table.

      The result is somewhat like an Excel vlookup or SQL leftJoin.

      • The leftTable always retains the same number of rows and the same columns with which it started.
      • If there are no matching values for a row, the appended cell(s) from the rightTable will contain NULL values.
      • The right side table can only have one row for each key of the join. If duplicate rows exist on the right side, then the operation's initialization or update results in an error.

      When columnsToMatch is empty, then no join keys are used. If there is a row in the right table, it is joined to all rows of the left table. If there are no rows in the right table, then the right columns are null. If there are multiple rows in the right table then there is an error.

      Specified by:
      naturalJoin in interface Table
      Overrides:
      naturalJoin in class QueryTable
      Parameters:
      rightTable - The right side table on the join.
      columnsToMatch - An array of match pair conditions ("leftColumn=rightColumn" or "columnFoundInBoth")
      columnsToAdd - An array of the columns from the right side be added to the left side as a result of the match. If empty, then all columns from the right table are added to the result.
      Returns:
      a new table joined according to the specification in columnsToMatch and columnsToAdd

    • join

      public Table join(Table rightTable, MatchPair[] columnsToMatch, MatchPair[] columnsToAdd, int numRightBitsToReserve)
      Description copied from interface: Table
      Perform a cross join with the right table.

      Returns a table that is the cartesian product of left rows X right rows, with one column for each of the left table's columns, and one column corresponding to each of the right table's columns that are included in the columnsToAdd argument. The rows are ordered first by the left table then by the right table. If columnsToMatch is non-empty then the product is filtered by the supplied match conditions.

      To efficiently produce updates, the bits that represent a key for a given row are split into two. Unless specified, join reserves 16 bits to represent a right row. When there are too few bits to represent all of the right rows for a given aggregation group the table will shift a bit from the left side to the right side. The default of 16 bits was carefully chosen because it results in an efficient implementation to process live updates.

      An OutOfKeySpaceException is thrown when the total number of bits needed to express the result table exceeds that needed to represent Long.MAX_VALUE. There are a few work arounds: - If the left table is sparse, consider flattening the left table. - If there are no key-columns and the right table is sparse, consider flattening the right table. - If the maximum size of a right table's group is small, you can reserve fewer bits by setting numRightBitsToReserve on initialization.

      Note: If you can prove that a given group has at most one right-row then you should prefer using Table.naturalJoin(com.illumon.iris.db.tables.Table, com.illumon.iris.db.tables.select.MatchPair[], com.illumon.iris.db.tables.select.MatchPair[]).

      Specified by:
      join in interface Table
      Overrides:
      join in class QueryTable
      Parameters:
      rightTable - The right side table on the join.
      columnsToMatch - An array of match pair conditions ("leftColumn=rightColumn" or "columnFoundInBoth")
      columnsToAdd - An array of the columns from the right side that need to be added to the left side as a result of the match.
      numRightBitsToReserve - The number of bits to reserve for rightTable groups.
      Returns:
      a new table joined according to the specification in columnsToMatch and columnsToAdd

    • lastBy

      public Table lastBy(SelectColumn... groupByColumns)
      Description copied from interface: Table
      Groups the data column according to groupByColumns and retrieves the last for the rest of the fields
      Specified by:
      lastBy in interface Table
      Overrides:
      lastBy in class QueryTable
      Parameters:
      groupByColumns - The grouping columns Table.by(String...)

    • firstBy

      public Table firstBy(SelectColumn... groupByColumns)
      Description copied from interface: Table
      Groups the data column according to groupByColumns and retrieves the first for the rest of the fields
      Specified by:
      firstBy in interface Table
      Overrides:
      firstBy in class QueryTable
      Parameters:
      groupByColumns - The grouping columns Table.by(String...)

    • minBy

      public Table minBy(SelectColumn[] selectColumns)
      Description copied from interface: Table
      Groups the data column according to groupByColumns and computes the min for the rest of the fields
      Specified by:
      minBy in interface Table
      Overrides:
      minBy in class QueryTable
      Parameters:
      selectColumns - The grouping columns Table.by(String...)

    • maxBy

      public Table maxBy(SelectColumn[] selectColumns)
      Description copied from interface: Table
      Groups the data column according to groupByColumns and computes the max for the rest of the fields
      Specified by:
      maxBy in interface Table
      Overrides:
      maxBy in class QueryTable
      Parameters:
      selectColumns - The grouping columns Table.by(String...) }

    • medianBy

      public Table medianBy(SelectColumn[] selectColumns)
      Description copied from interface: Table
      Groups the data column according to groupByColumns and computes the median for the rest of the fields
      Specified by:
      medianBy in interface Table
      Overrides:
      medianBy in class QueryTable
      Parameters:
      selectColumns - The grouping columns Table.by(String...) }

    • countBy

      public Table countBy(String countColumnName, SelectColumn... groupByColumns)
      Specified by:
      countBy in interface Table
      Overrides:
      countBy in class QueryTable

    • ungroup

      public Table ungroup(boolean nullFill, String... columnsToUngroup)
      Description copied from interface: Table
      Ungroups a table by converting arrays into columns.
      Specified by:
      ungroup in interface Table
      Overrides:
      ungroup in class QueryTable
      Parameters:
      nullFill - indicates if the ungrouped table should allow disparate sized arrays filling shorter columns with null values. If set to false, then all arrays should be the same length.
      columnsToUngroup - the columns to ungroup
      Returns:
      the ungrouped table

    • headPct

      public Table headPct(double percent)
      Description copied from interface: Table
      Provides a head that selects a dynamic number of rows based on a percent.
      Specified by:
      headPct in interface Table
      Overrides:
      headPct in class QueryTable
      Parameters:
      percent - the fraction of the table to return (0..1), the number of rows will be rounded up. For example if there are 3 rows, headPct(50) returns the first two rows.

    • tailPct

      public Table tailPct(double percent)
      Specified by:
      tailPct in interface Table
      Overrides:
      tailPct in class QueryTable

    • by

      public Table by(AggregationStateFactory aggregationStateFactory, SelectColumn... groupByColumns)
      Specified by:
      by in interface Table
      Overrides:
      by in class QueryTable

    • headBy

      public Table headBy(long nRows, String... groupByColumns)
      Specified by:
      headBy in interface Table
      Overrides:
      headBy in class QueryTable

    • tailBy

      public Table tailBy(long nRows, String... groupByColumns)
      Specified by:
      tailBy in interface Table
      Overrides:
      tailBy in class QueryTable

    • applyToAllBy

      public Table applyToAllBy(String formulaColumn, String columnParamName, SelectColumn... groupByColumns)
      Description copied from interface: Table
      Groups data according to groupByColumns and applies formulaColumn to each of columns not altered by the grouping operation. columnParamName is used as place-holder for the name of each column inside formulaColumn.
      Specified by:
      applyToAllBy in interface Table
      Overrides:
      applyToAllBy in class QueryTable
      Parameters:
      formulaColumn - Formula applied to each column
      columnParamName - The parameter name used as a placeholder for each column
      groupByColumns - The grouping columns Table.by(SelectColumn[])

    • sumBy

      public Table sumBy(SelectColumn... groupByColumns)
      Description copied from interface: Table
      Groups the data column according to groupByColumns and computes the sum for the rest of the fields
      Specified by:
      sumBy in interface Table
      Overrides:
      sumBy in class QueryTable
      Parameters:
      groupByColumns - The grouping columns Table.by(String...)

    • absSumBy

      public Table absSumBy(SelectColumn... groupByColumns)
      Description copied from interface: Table
      Groups the data column according to groupByColumns and computes the sum of the absolute values for the rest of the fields
      Specified by:
      absSumBy in interface Table
      Overrides:
      absSumBy in class QueryTable
      Parameters:
      groupByColumns - The grouping columns Table.by(String...)

    • avgBy

      public Table avgBy(SelectColumn... groupByColumns)
      Description copied from interface: Table
      Groups the data column according to groupByColumns and computes the average for the rest of the fields
      Specified by:
      avgBy in interface Table
      Overrides:
      avgBy in class QueryTable
      Parameters:
      groupByColumns - The grouping columns Table.by(String...)

    • wavgBy

      public Table wavgBy(String weightColumn, SelectColumn... groupByColumns)
      Description copied from interface: Table
      Groups the data column according to groupByColumns and computes the weighted average using weightColumn for the rest of the fields
      Specified by:
      wavgBy in interface Table
      Overrides:
      wavgBy in class QueryTable
      Parameters:
      weightColumn - the column to use for the weight
      groupByColumns - The grouping columns Table.by(String...)

    • stdBy

      public Table stdBy(SelectColumn... groupByColumns)
      Specified by:
      stdBy in interface Table
      Overrides:
      stdBy in class QueryTable

    • varBy

      public Table varBy(SelectColumn... groupByColumns)
      Description copied from interface: Table
      Groups the data column according to groupByColumns and computes the variance for the rest of the fields
      Specified by:
      varBy in interface Table
      Overrides:
      varBy in class QueryTable
      Parameters:
      groupByColumns - The grouping columns Table.by(String...)

    • where

      public Table where(SelectFilter... filters)
      Specified by:
      where in interface Table
      Overrides:
      where in class QueryTable

    • whereIn

      public Table whereIn(Table rightTable, boolean inclusion, MatchPair... columnsToMatch)
      Description copied from interface: Table
      Filters this table based on the set of values in the rightTable. Note that when the right table ticks, all of the rows in the left table are going to be re-evaluated, thus the intention is that the right table is fairly slow moving compared with the left table.
      Specified by:
      whereIn in interface Table
      Overrides:
      whereIn in class QueryTable
      Parameters:
      rightTable - the filtering table.
      inclusion - whether things included in rightTable should be passed through (they are exluded if false)
      columnsToMatch - the columns to match between the two tables
      Returns:
      a new table filtered on right table

    • select

      public Table select(SelectColumn... selectColumns)
      Specified by:
      select in interface Table
      Overrides:
      select in class QueryTable

    • selectDistinct

      public Table selectDistinct(SelectColumn... columns)
      Specified by:
      selectDistinct in interface Table
      Overrides:
      selectDistinct in class QueryTable

    • update

      public Table update(SelectColumn... columns)
      Specified by:
      update in interface Table
      Overrides:
      update in class QueryTable

    • view

      public Table view(SelectColumn... columns)
      Specified by:
      view in interface Table
      Overrides:
      view in class QueryTable

    • updateView

      public Table updateView(SelectColumn... columns)
      Specified by:
      updateView in interface Table
      Overrides:
      updateView in class QueryTable

    • lazyUpdate

      public Table lazyUpdate(SelectColumn... columns)
      Description copied from interface: Table
      Compute column formulas on demand.

      Lazy update defers computation until required for a set of values, and caches the results for a set of input values. This uses less RAM than an update statement when you have a smaller set of unique values. Less computation than an updateView is needed, because the results are saved in a cache.

      If you have many unique values, you should instead use an update statement, which will have more memory efficient structures. Values are never removed from the lazyUpdate cache, so it should be used judiciously on a ticking table.

      Specified by:
      lazyUpdate in interface Table
      Overrides:
      lazyUpdate in class QueryTable
      Parameters:
      columns - the columns to add
      Returns:
      a new Table with the columns added; to be computed on demand

    • flatten

      public Table flatten()
      Description copied from interface: Table
      Creates a version of this table with a flat index (V2 only).
      Specified by:
      flatten in interface Table
      Overrides:
      flatten in class QueryTable

    • preemptiveUpdatesTable

      public Table preemptiveUpdatesTable(long updateInterval)
      Specified by:
      preemptiveUpdatesTable in interface Table
      Overrides:
      preemptiveUpdatesTable in class QueryTable

    • preemptiveSnapshotTable

      public Table preemptiveSnapshotTable(long updateInterval)
      Specified by:
      preemptiveSnapshotTable in interface Table
      Overrides:
      preemptiveSnapshotTable in class QueryTable

    • byExternal

      public LocalTableMap byExternal(boolean dropKeys, String... keyColumnNames)
      Description copied from interface: Table
      Create a TableMap from this table, keyed by the specified columns.

      The returned TableMap contains each row in this table in exactly one of the tables within the map. If you have exactly one key column the TableMap is keyed by the value in that column. If you have zero key columns, then the TableMap is keyed by com.fishlib.datastructures.util.SmartKey.EMPTY (and will contain this table as the value). If you have multiple key columns, then the TableMap is keyed by a com.fishlib.datastructures.util.SmartKey. The SmartKey will have one value for each of your column values, in the order specified by keyColumnNames.

      For example if you have a Table keyed by a String column named USym, and a DBDateTime column named Expiry; a value could be retrieved from the TableMap with tableMap.get(new SmartKey("SPY";, DBTimeUtils.convertDateTime("2020-06-19T16:15:00 NY"))). For a table with an Integer column named Bucket, you simply use the desired value as in tableMap.get(1).

      Specified by:
      byExternal in interface Table
      Overrides:
      byExternal in class QueryTable
      Parameters:
      dropKeys - if true, drop key columns in the output Tables
      keyColumnNames - the name of the key columns to use.
      Returns:
      a TableMap keyed by keyColumnNames

    • rollup

      public HierarchicalTable rollup(ComboAggregateFactory comboAggregateFactory, SelectColumn... columns)
      Create a rollup table. A rollup table aggregates by the specified columns, and then creates a hierarchical table which re-aggregates using one less aggregation column on each level. The column that is no longer part of the aggregation key is replaced with null on each level.
      Parameters:
      comboAggregateFactory - the ComboAggregateFactory describing the aggregation
      columns - the columns to group by
      Returns:
      a hierarchical table with the rollup applied
      ImplNote:
      The Preemptive version of this method will return a non-preemptive table

    • treeTable

      public HierarchicalTable treeTable(String idColumn, String parentColumn)
      Create a hierarchical tree table. The structure of the table is encoded by an "id" and a "parent" column. The id column should represent a unique identifier for a given row, and the parent column indicates which row is the parent for a given row. Rows that have a null parent, are shown in the main table. It is possible for rows to be "orphaned", if their parent reference is non-null and does not exist in the table.
      Specified by:
      treeTable in interface Table
      Overrides:
      treeTable in class QueryTable
      Parameters:
      idColumn - the name of a column containing a unique identifier for a particular row in the table
      parentColumn - the name of a column containing the parent's identifier, null for elements that are part of the root table
      Returns:
      a hierarchical table grouped according to the parentColumn
      ImplNote:
      The Preemptive version of this method will return a non-preemptive table

    • reverse

      public Table reverse()
      Description copied from class: QueryTable
      The reverse operation returns a new table that is the same as the original table, but the first row is last, and the last row is first. This is an internal API to be used by .raj(), but is accessible for unit tests.
      Specified by:
      reverse in interface Table
      Overrides:
      reverse in class QueryTable
      Returns:
      the reversed table

    • snapshot

      public Table snapshot(Table rightTable, boolean doInitialSnapshot, String... stampColumns)
      Description copied from interface: Table
      Snapshot "rightTable", triggered by "this" Table, and return a new Table as a result. "this" Table is the triggering table, i.e. the table whose change events cause a new snapshot to be taken. The result table includes a "snapshot key" which is a subset (possibly all) of this Table's columns. The remaining columns in the result table come from "rightTable", the table being snapshotted.
      Specified by:
      snapshot in interface Table
      Overrides:
      snapshot in class QueryTable
      Parameters:
      rightTable - The table to be snapshotted
      doInitialSnapshot - Take the first snapshot now (otherwise wait for a change event)
      stampColumns - The columns forming the "snapshot key", i.e. some subset of this Table's columns to be included in the result at snapshot time. As a special case, an empty stampColumns is taken to mean "include all columns".
      Returns:
      The result table

    • snapshotIncremental

      public Table snapshotIncremental(Table rightTable, boolean doInitialSnapshot, String... stampColumns)
      Specified by:
      snapshotIncremental in interface Table
      Overrides:
      snapshotIncremental in class QueryTable

    • snapshotHistory

      public Table snapshotHistory(Table rightTable)
      Specified by:
      snapshotHistory in interface Table
      Overrides:
      snapshotHistory in class QueryTable

    • getSubTable

      public QueryTable getSubTable(Index index)
      Specified by:
      getSubTable in interface Table
      Overrides:
      getSubTable in class QueryTable