Package com.illumon.iris.importers

Class GeneralXmlImporter

java.lang.Object
com.illumon.iris.importers.GeneralImporter<CsvFieldWriter>
com.illumon.iris.importers.GeneralXmlImporter
public class GeneralXmlImporter
extends GeneralImporter<CsvFieldWriter>
General XML importer class to handle single-table XML imports

  • Constructor Details

    • GeneralXmlImporter

      public GeneralXmlImporter​(com.fishlib.io.logger.Logger log, ImportTableWriterFactory importTableWriterFactory, String intradayPartitionColumn, org.jdom2.Element sourceElement, List<File> sourceFiles, CsvFieldWriter.Factory factory, char delimiter, boolean strict, String constantColumnValue, String elementType, int startIndex, int startDepth, int maxDepth, boolean useElementValues, boolean useAttributeValues, boolean namedValues) throws IOException
      Constructor used when importing from a set of files. Writer will be closed automatically when the set has been imported.
      Parameters:
      log - Logger created upstream
      sourceElement - XML schema element of the ImportSource block to be used
      importTableWriterFactory - Provides table writers on demand based on the type of import (single vs multi partition)
      intradayPartitionColumn - Column to use for determining the target partition for multi partition imports (generally Date)
      sourceFiles - Stream of File objects from which to read CSV data
      factory - CSVFieldWriter factory that will create the field writers (setters) for the table's columns
      delimiter - Single character to be used as a delimiter - normally this is a comma
      strict - Whether to fail if a field fails numeric conversion or a column is missing a setter
      constantColumnValue - A String to materialize as the source column when an ImportColumn is defined with a sourceType of CONSTANT (aka ImporterColumnDefinition$IrisImportConstant). Can be null.
      elementType - A string element name to match when finding elements to import from the XML
      startIndex - Number of elements after the root element to start looking for elements to import
      startDepth - How far under the element obtained from startIndex to start looking for elements to import
      maxDepth - How far to recurse into import elements when searching for values to import
      useElementValues - Whether to use values that are stored as the contents of elements
      useAttributeValues - Whether to user values that are stored as attributes
      namedValues - True to use values by name, false to use values by position
      Throws:
      IOException - If there are problems reading the source file or stream

    • GeneralXmlImporter

      public GeneralXmlImporter​(com.fishlib.io.logger.Logger log, ImportTableWriterFactory importTableWriterFactory, String intradayPartitionColumn, org.jdom2.Element sourceElement, InputStream sourceStream, CsvFieldWriter.Factory factory, char delimiter, boolean strict, String constantColumnValue, String elementType, Map<String,​String> importProperties, int startIndex, int startDepth, int maxDepth, boolean useElementValues, boolean useAttributeValues, boolean namedValues) throws IOException
      Constructor used when importing from an InputStream - e.g. QuandlImporter.
      Parameters:
      log - Logger created upstream
      importTableWriterFactory - Provides table writers on demand based on the type of import (single vs multi partition)
      intradayPartitionColumn - Column to use for determining the target partition for multi partition imports (generally Date)
      sourceElement - XML schema element of the ImportSource block to be used
      sourceStream - InputStream from which to read CSV data
      factory - CSVFieldWriter factory that will create the field writers (setters) for the table's columns
      delimiter - Single character to be used as a delimiter - normally this is a comma
      strict - Whether to fail if a field fails numeric conversion or a column is missing a setter
      constantColumnValue - A String to materialize as the source column when an ImportColumn is defined with a sourceType of CONSTANT (aka ImporterColumnDefinition$IrisImportConstant). Can be null.
      elementType - A string element name to match when finding elements to import from the XML
      importProperties -
      startIndex - Number of elements after the root element to start looking for elements to import
      startDepth - How far under the element obtained from startIndex to start looking for elements to import
      maxDepth - How far to recurse into import elements when searching for values to import
      useElementValues - Whether to use values that are stored as the contents of elements
      useAttributeValues - Whether to user values that are stored as attributes
      namedValues - True to use values by name, false to use values by position
      Throws:
      IOException - If there are problems reading the source file or stream

  • Method Details

    • getLineCount

      public long getLineCount()

    • run

      public void run()
      Iterates the file set, or passes the InputStream directly to importSingleStream to be imported.