Logging to Deephaven from C++

As with Java, Deephaven supports native logging of data from C++ applications.

The Deephaven C++ logging libraries require C++11 because they require support for variadic template arguments. Deephaven uses gcc 4.8 and newer, but any compiler supporting C++11 should work.

There are two components to a new C++ logger:

A constructor, which defines the column names and types.
A function to low a row of data, with arguments that match the column types. This function simply wraps illumon::binarystore::BinaryStoreWriter::writeRow. (Calling writeRow directly with incorrect arguments will compile, but will fail at runtime. Calling the wrapper function with incorrect arguments will fail during compilation.)

Source Code

The source code for the Deephaven C++ loggers is available in a ZIP archive.

In addition to the source code for the C++ logging framework, the ZIP archive contains the complete TradeWriter example referenced in the Example section below. The archive's "examples" directory includes the full implementation of TradeWriter, along with a usage example and sample data.

The example files are:

TradeWriter.hpp — the logger class's header
TradeWriter.cpp — the logger class's implementation
TradeWriterExample.cpp — an example application that reads CSV data and writes it using the TradeWriter class
trades.csv — example market trade data, to be read by TradeWriterExample and written in Deephaven's binary format
trades-in.bin — the example data, stored in Deephaven's binary format

Null Values

Deephaven uses reserved constants to represent null values for each primitive type. These are defined by BinaryStoreWriter.hpp under illumon::binarystore.

static const uint8_t NULL_CHAR = 0;
static const int8_t NULL_BYTE = std::numeric_limits<int8_t>::min(); // -128
static const int16_t NULL_SHORT = std::numeric_limits<int16_t>::min(); // -32768
static const int32_t NULL_INT = std::numeric_limits<int32_t>::min(); // -2147483648
static const int64_t NULL_LONG = std::numeric_limits<int64_t>::min(); // - 9223372036854775808
static const float NULL_FLOAT = (-std::numeric_limits<float>::max()); // -3.40282e+38
static const double NULL_DOUBLE = (-std::numeric_limits<double>::max()); //-1.79769e+308

Example

An abridged version of an example C++ logger, TradeWriter, is provided below. The TradeWriter demonstrates how to use Deephaven's BinaryStoreWriter to log stock market data. This example logs data for a table's Timestamp, Exchange, USym, Sym, Price, and Size columns.

Note that the table's partitioning column, Date, is excluded from the logger, since a table's partition value is determined by the Data Import Server.

TradeWriter.hpp:

#pragma once

#include "BinaryStoreWriter.hpp"

/*
* This is an example wrapper for logging market trades to a Deephaven binary store.
*
* The resultant output file can then be streamed by a log tailer to the data
* import server, which makes it available for use in queries.
*
*/
class TradeWriter
: illumon::binarystore::BinaryStoreWriter
{
public:
/* The constructor initializes the fields that will be written.
*
* The field declarations must match the schema expected by the data import server. */
TradeWriter();

/*
* Write a trade entry row to the binary store.
*
* Timestamps are represented as nanoseconds from the Unix epoch.
*/

void writeTrade(uint64_t insertTime, std::string const & usym, std::string const & sym, std::string const & side, double price, int size, uint64_t exchangeTimestamp, std::string const & exchange);

/* Pass through the open and close calls. */
using illumon::binarystore::BinaryStoreWriter::open;
using illumon::binarystore::BinaryStoreWriter::close;
};

TradeWriter.cpp:

#include "TradeWriter.hpp"

/*
* On construction, the TradeWriter declares the columns that are part of the
* log file. These must match the schema definition. In this case, the schema
* used is:
*
* <Table name="StockTrades" namespace="LearnIris" storageType="NestedPartitionedOnDisk">
* <Partitions keyFormula="__PARTITION_AUTOBALANCE_SINGLE__"/>
* <Column name="Date" dataType="String" columnType="Partitioning"/>
*   <Column name="Timestamp" dataType="DateTime" columnType="Normal"/>
* <Column name="Exchange" dataType="String" columnType="Normal"/>
* <Column name="USym" dataType="String" columnType="Grouping"/>
* <Column name="Sym" dataType="String" columnType="Grouping"/>
* <Column name="Last" dataType="double" columnType="Normal"/>
* <Column name="Size" dataType="int" columnType="Normal"/>
* <Column name="ExchangeTimestamp" dataType="DateTime" columnType="Normal"/>
* </Table>
*
* The "Date" column is not included because it is a Partitioning column, and
* the data import server implicitly stores the partitioning column as part of
* the directory structure. Similarly, values with schema attribute of
* intradayType="none" should not be included.
*
* All other columns must be included with the same type (Boolean, Byte, Char,
* Double, Float, Int, Long, String) as in the schema definition.
*
* DateTime columns are represented as Long nanoseconds since the Unix epoch.
*
* Blobs and Enums are not supported from C++.
*/
TradeWriter::TradeWriter() {
addColumn("Timestamp", illumon::binarystore::SupportedType::Long);
addColumn("Exchange", illumon::binarystore::SupportedType::String);
addColumn("USym", illumon::binarystore::SupportedType::String);
addColumn("Sym", illumon::binarystore::SupportedType::String);
addColumn("Side", illumon::binarystore::SupportedType::String);
addColumn("Last", illumon::binarystore::SupportedType::Double);
addColumn("Size", illumon::binarystore::SupportedType::Int);
addColumn("ExchangeTimestamp", illumon::binarystore::SupportedType::Long);
}

/*
* This is a thin wrapper to the underlying writeRow in BinaryStoreWriter.*
* The arguments to writeRow must be in the same order that the constructor
* declared the columns using addColumn; otherwise a logic_error will be
* thrown at runtime.
*/
void TradeWriter::writeTrade(uint64_t insertTime, std::string const & usym, std::string const & sym, std::string const & side, double price, int size, uint64_t exchangeTimestamp, std::string const & exchange)
{
writeRow(
  insertTime,
  exchange,
  usym,
  sym,
  side,
  price,
  size,
  exchangeTimestamp);
}

Last Updated: 20 August 2019 09:54 -06:00 UTC Deephaven v.1.20180917