Package com.illumon.iris.importers.example

Class ExampleImporter

java.lang.Object
com.illumon.iris.importers.BaseImporter<com.illumon.iris.importers.example.ExampleImporter.Arguments>
com.illumon.iris.importers.example.ExampleImporter
public class ExampleImporter
extends BaseImporter<com.illumon.iris.importers.example.ExampleImporter.Arguments>

This file is an example importer that illustrates the extensions to BaseImporter needed to create a custom importer.

There are two examples. Both manufacture data based on a row index.

processDataSimple() will write data to a table defined by the following schema: 


 <Table name="DemoTable" namespace="ExampleNamespace" rethrowLoggerExceptionsAsIOExceptions="false" storageType="NestedPartitionedOnDisk">
      <Partitions keyFormula="${autobalance_by_first_grouping_column}" />

      <Column name="StringCol" dataType="String" />
      <Column name="DateTimeCol" dataType="DateTime" />
      <Column name="IntegerCol" dataType="Integer" />
      <Column name="LongCol" dataType="Long" />
      <Column name="DoubleCol" dataType="Double" />
      <Column name="FloatCol" dataType="Float" />
      <Column name="BooleanCol" dataType="Boolean" />
      <Column name="CharCol" dataType="Char" />
      <Column name="ByteCol" dataType="Byte" />
      <Column name="ShortCol" dataType="Short" />
      <Column name="Date" dataType="String" columnType="Partitioning" />
 </Table>

The getters and setters are hard-coded and stored in variables. This method is simple, but tedious.

A valid invocation will include options similar to the following (note that the namespace and table name parameters match the example schema above:

-dp localhost/2018-02-28 -ns ExampleNamespace -tn DemoTable -om REPLACE -nd 0 -st 100 -nr 123

processDataDynamic() will write data to any table with an existing schema. The code to get the data is a calculation based on the index and data type. There are several layers of code to move calculations out of the row processing loop, and to ensure that unnecessary inefficiency (e.g. boxing of primitives) is avoided.

A valid invocation will include options similar to this:

-dp localhost/2018-02-28 -ns AnyNamespace -tn AnyTable -om REPLACE -nd 0 -st 100 -nr 123

  • Constructor Details

    • ExampleImporter

      public ExampleImporter​(@NotNull com.fishlib.io.logger.Logger log, @NotNull com.illumon.iris.importers.example.ExampleImporter.Arguments arguments)
      Construct the ExampleImporter.

      Can be private, because it's only constructed from this class's main(). If queries are used to set up and run this importer, other access specifiers or constructors might be needed.

      Parameters:
      log - use this Logger for feedback
      arguments - importer arguments defining the import run

    • ExampleImporter

      public ExampleImporter​(@NotNull com.fishlib.io.logger.Logger log, @NotNull String namespace, @NotNull String tableName, @NotNull String partitionString, @NotNull String outputMode, int startIndex, int numRows)
      Alternate constructor for usage without a command line.
      Parameters:
      log - Logger for monitoring and debugging messages
      namespace - namespace of the destination table
      tableName - name of the destination table
      partitionString - internal and column partition values ("internalPartition/columnPartition")
      outputMode - a valid ImportOutputMode value
      startIndex - see the -st command line option: "Generate rows starting at startIndex (default is 0)"
      numRows - see the -nr command line option: "Generate this many rows"

  • Method Details

    • processData

      protected void processData()
      This will be called by the base importer superclass.

      Override this method to open and process your data source. For each row of the input, call tableWriter.getSetter(...).set(value) for each column, and then tableWriter.writeRow().

      Specified by:
      processData in class BaseImporter<com.illumon.iris.importers.example.ExampleImporter.Arguments>

    • importData

      public static void importData​(@NotNull com.fishlib.io.logger.Logger log, @NotNull String namespace, @NotNull String tableName, @NotNull String partitionString, @NotNull String outputMode, int startIndex, int numRows)
      Static helper method to avoid fluff in queries.
      Parameters:
      log - Logger for monitoring and debugging messages
      namespace - namespace of the destination table
      tableName - name of the destination table
      partitionString - internal and column partition values ("internalPartition/columnPartition")
      outputMode - a valid ImportOutputMode value
      startIndex - see the -st command line option: "Generate rows starting at startIndex (default is 0)"
      numRows - see the -nr command line option: "Generate this many rows"

    • main

      public static void main​(String... args)
      Example importer.
      Parameters:
      args - command line arguments to the process