Custom Importer

If one of the existing import tools does not support your input data, it might be useful to create a custom importer.

ExampleImporter.java extends BaseImporter.java in a very basic way in order to illustrate how the classes allow easy functional additions. BaseImporter handles most of the setup, freeing you to focus on just moving the data. ExampleImporter.java has extensive documentation that should explain what the class does and what needs to be overridden or copied.

You will need to create your own class extending BaseImporter, and

Create an Arguments class extending StandardImporterArguments. This class needs to add arguments for the information your importer needs to locate and process the data to be imported.
Copy main, replacing occurrences of ExampleImporter.Arguments with your Arguments class, and ExampleImporter with your importer class.
Implement processData in your importer class. This method must set all the column values using the TableWriter instance and call TableWriter.writeRow for each row of output data.
ExampleImporter includes implementations at two extremes - one is completely hardcoded and the other is completely dynamic.

Prerequisites

The namespace and table name must exist in the System namespace, and the table definition must exist. This will be true if a schema has been deployed for the table.

Executing the program

This importer should be run on a merge server host, with local access to intraday data..

All Deephaven processes expect certain properties. Generally these are provided by scripts or service configuration. In this case, you need to provide them as you run the import program.

Classpath

Your ingester's classpath should include installed Deephaven files, the default override locations, and the path to your own java code.

/etc/sysconfig/illumon.d/hotfixes

/etc/sysconfig/illumon.d/override

/etc/sysconfig/illumon.d/resources

/etc/sysconfig/illumon.d/java_lib/*

/usr/illumon/latest/etc

/usr/illumon/latest/java_lib/*

process.name

Set this to whatever seems appropriate. Only one instance of each process.name can run at any given time. Pass this as a JVM argument:

-Dprocess.name=example_importer

Configuration.rootFile

The root property file specifying configuration properties for your process. Pass this as a JVM argument:

-DConfiguration.rootFile=iris-common.prop

devroot

Can be /usr/illumon/latest, or the resolved target of that link.

-Ddevroot=/usr/illumon/latest

workspace

This governs where log files will go. Matching the other import processes should be fine.

-Dworkspace=/db/TempFiles/dbmerge/example_importer

Class to execute

Replace com.illumon.iris.importers.ExampleImporter with your class.

Program Arguments

Standard importer arguments, plus any new arguments you add. Standard arguments are:

-dd or --destinationDirectory <path>
-dp or --destinationPartition <internal partition name / partitioning value>
-ns or --namespace <namespace>
-tn or --tableName <name>
-om or --outputMode <import behavior>

(Note: See Tables & Schemas for more information.)

This will give you a command line something like:

export EXAMPLECLASSPATH=/etc/sysconfig/illumon.d/hotfixes:/etc/sysconfig/illumon.d/override:/etc/sysconfig/illumon.d/resources:/etc/sysconfig/illumon.d/java_lib/*:/usr/illumon/latest/etc:/usr/illumon/latest/java_lib/*

sudo java -cp $EXAMPLECLASSPATH -Dprocess.name=example_importer -DConfiguration.rootFile=iris-common.prop -Ddevroot=/usr/illumon/latest -Dworkspace=/db/TempFiles/dbmerge/example_importer com.illumon.iris.importers.ExampleImporter -dp hostname/2018-03-07 -om REPLACE -ns ExampleNamespace -tn ExampleTable [custom arguments]

Importing as Query

ExampleImporter and ExampleImporter.Arguments include overrides that bypass the command line. Importers that override these methods in a similar way can be called from an Iris console or persistent query.

import com.illumon.iris.importers.ExampleImporter

// Call the static convenience method
ExampleImporter.importData(log, "ExampleNamespace", "ExampleTable", "hostname/2018-01-01", "REPLACE", 12, 23)

// Create an object and invoke doImport()
e = new com.illumon.iris.importers.ExampleImporter(log, "ExampleNamespace", "ExampleTable", "hostname/2018-03-08", "REPLACE", 12, 3)
e.doImport()

Last Updated: 16 February 2021 18:06 -04:00 UTC Deephaven v.1.20200928 (See other versions)