Package io.deephaven.importers.csv

Class GeneralCsvImporter

java.lang.Object
io.deephaven.importers.GeneralImporter<CsvFieldWriter>
io.deephaven.importers.csv.GeneralCsvImporter
public class GeneralCsvImporter extends GeneralImporter<CsvFieldWriter>
General CSV importer class to handle standard CSV imports

  • Constructor Details

    • GeneralCsvImporter

      public GeneralCsvImporter(@NotNull @NotNull Logger log, @NotNull @NotNull ImportTableWriterFactory importTableWriterFactory, @Nullable @Nullable String intradayPartitionColumn, @Nullable @Nullable io.deephaven.shadow.enterprise.org.jdom2.Element sourceElement, char delimiter, boolean strict, String fileFormat, List<File> sourceFiles, InputStream sourceStream, @Nullable @Nullable String constantColumnValue, Map<String,String> customImportProperties, int skipHeaderLines, int skipFooterLines, boolean trim, boolean noHeader, @Nullable @Nullable List<String> columnNames)
      Constructor used when importing from a set of files. Writer will be closed automatically when the set has been imported. For backwards-compatibility with callers that expect to always have a header row in the CSV and that don't support skipping footer lines.
      Parameters:
      log - Logger created upstream
      importTableWriterFactory - Provides table writers on demand based on the type of import (single vs multi partition)
      intradayPartitionColumn - Column to use for determining the target partition for multi partition imports (generally Date)
      sourceElement - InputStream from which to read CSV data
      sourceFiles - Stream of File objects from which to read CSV data
      sourceStream - The @{link InputStream} from which to read CSV data
      delimiter - Single character to be used as a delimiter - normally this is a comma
      strict - Whether to fail if a field fails numeric conversion or a column is missing a setter
      fileFormat - Apache commons CSV file format to use
      constantColumnValue - A String to materialize as the source column when an ImportColumn is defined with a sourceType of CONSTANT (aka ImporterColumnDefinition$IrisImportConstant)
      skipHeaderLines - Number of lines to skip before the header
      trim - If trim is true, use CSVFormat that trims leading and trailing blanks.
      customImportProperties - Custom import properties
      skipFooterLines - Number of lines to skip at the end of the file
      noHeader - Whether the file has a header row with column names
      columnNames - the column names to import

    • GeneralCsvImporter

      public GeneralCsvImporter(@NotNull @NotNull Logger log, @NotNull @NotNull ImportTableWriterFactory importTableWriterFactory, String intradayPartitionColumn, io.deephaven.shadow.enterprise.org.jdom2.Element sourceElement, List<File> sourceFiles, char delimiter, boolean strict, String fileFormat, String constantColumnValue, int skipHeaderLines, boolean trim)
      Constructor used when importing from a set of files. Writer will be closed automatically when the set has been imported. For backwards-compatibility with callers that expect to always have a header row in the CSV and that don't support skipping footer lines.
      Parameters:
      log - Logger created upstream
      importTableWriterFactory - Provides table writers on demand based on the type of import (single vs multi partition)
      intradayPartitionColumn - Column to use for determining the target partition for multi partition imports (generally Date)
      sourceElement - InputStream from which to read CSV data
      sourceFiles - Stream of File objects from which to read CSV data
      delimiter - Single character to be used as a delimiter - normally this is a comma
      strict - Whether to fail if a field fails numeric conversion or a column is missing a setter
      fileFormat - Apache commons CSV file format to use
      constantColumnValue - A String to materialize as the source column when an ImportColumn is defined with a sourceType of CONSTANT (aka ImporterColumnDefinition$IrisImportConstant). Can be null.
      skipHeaderLines - Number of lines to skip before the header. Use this for files that have leading "garbage".
      trim - If trim is true, use CSVFormat that trims leading and trailing blanks.

    • GeneralCsvImporter

      public GeneralCsvImporter(Logger log, ImportTableWriterFactory importTableWriterFactory, String intradayPartitionColumn, io.deephaven.shadow.enterprise.org.jdom2.Element sourceElement, List<File> sourceFiles, char delimiter, boolean strict, String fileFormat, String constantColumnValue, int skipHeaderLines, int skipFooterLines, boolean trim, boolean noHeader, List<String> columnNames) throws IOException
      Constructor used when importing from a set of files. Writer will be closed automatically when the set has been imported.
      Parameters:
      log - Logger created upstream
      importTableWriterFactory - Provides table writers on demand based on the type of import (single vs multi partition)
      intradayPartitionColumn - Column to use for determining the target partition for multi partition imports (generally Date)
      sourceElement - InputStream from which to read CSV data
      sourceFiles - Stream of File objects from which to read CSV data
      delimiter - Single character to be used as a delimiter - normally this is a comma
      strict - Whether to fail if a field fails numeric conversion or a column is missing a setter
      fileFormat - Apache commons CSV file format to use
      constantColumnValue - A String to materialize as the source column when an ImportColumn is defined with a sourceType of CONSTANT (aka ImporterColumnDefinition$IrisImportConstant). Can be null.
      skipHeaderLines - Number of lines to skip before the header. Use this for files that have leading "garbage".
      skipFooterLines - Number of lines to skip at the end of the file. Use this for files that have trailing "garbage".
      trim - If trim is true, use CSVFormat that trims leading and trailing blanks.
      noHeader - Whether the CSV does not include a header row with column names.
      columnNames - A list of column names to use instead of a header from the CSV.
      Throws:
      IOException - is thrown in case of IO failure when handling given source files

    • GeneralCsvImporter

      public GeneralCsvImporter(Logger log, ImportTableWriterFactory importTableWriterFactory, String intradayPartitionColumn, io.deephaven.shadow.enterprise.org.jdom2.Element sourceElement, InputStream sourceStream, char delimiter, boolean strict, String fileFormat, String constantColumnValue, int skipLines, boolean trim) throws IOException
      Constructor used when importing from an InputStream - e.g. QuandlImporter.*

      Note that the CSV importer will always close any table writers it uses, so it is up to the factory to provide appending writers if appropriate.

      Parameters:
      log - Logger created upstream
      importTableWriterFactory - Provides table writers on demand based on the type of import (single vs multi partition)
      intradayPartitionColumn - Column to use for determining the target partition for multi partition imports (generally Date)
      sourceElement - InputStream from which to read CSV data
      sourceStream - InputStream from which to read CSV data
      delimiter - Single character to be used as a delimiter - normally this is a comma
      strict - Whether to fail if a field fails numeric conversion or a column is missing a setter
      fileFormat - Apache commons CSV file format to use
      constantColumnValue - A String to materialize as the source column when an ImportColumn is defined with a sourceType of CONSTANT (aka ImporterColumnDefinition$IrisImportConstant). Can be null.
      skipLines - Number of lines to skip before the header. Use this for files that have leading "garbage".
      trim - If trim is true, use CSVFormat that trims leading and trailing blanks.
      Throws:
      IOException - is thrown in case of IO failure when handling given sourceStream

    • GeneralCsvImporter

      public GeneralCsvImporter(Logger log, ImportTableWriterFactory importTableWriterFactory, String intradayPartitionColumn, io.deephaven.shadow.enterprise.org.jdom2.Element sourceElement, InputStream sourceStream, char delimiter, boolean strict, String fileFormat, String constantColumnValue, int skipLines, boolean trim, boolean noHeader, List<String> columnNames) throws IOException
      Constructor used when importing from an InputStream - e.g. QuandlImporter.*

      Note that the CSV importer will always close any table writers it uses, so it is up to the factory to provide appending writers if appropriate.

      Parameters:
      log - Logger created upstream
      importTableWriterFactory - Provides table writers on demand based on the type of import (single vs multi partition)
      intradayPartitionColumn - Column to use for determining the target partition for multi partition imports (generally Date)
      sourceElement - InputStream from which to read CSV data
      sourceStream - InputStream from which to read CSV data
      delimiter - Single character to be used as a delimiter - normally this is a comma
      strict - Whether to fail if a field fails numeric conversion or a column is missing a setter
      fileFormat - Apache commons CSV file format to use
      constantColumnValue - A String to materialize as the source column when an ImportColumn is defined with a sourceType of CONSTANT (aka ImporterColumnDefinition$IrisImportConstant). Can be null.
      skipLines - Number of lines to skip before the header. Use this for files that have leading "garbage".
      trim - If trim is true, use CSVFormat that trims leading and trailing blanks.
      noHeader - Whether the CSV does not include a header row with column names.
      columnNames - A list of column names to use instead of a header from the CSV.
      Throws:
      IOException - is thrown in case of IO failure when handling given sourceStream

    • GeneralCsvImporter

      public GeneralCsvImporter(Logger log, ImportTableWriterFactory importTableWriterFactory, String intradayPartitionColumn, io.deephaven.shadow.enterprise.org.jdom2.Element sourceElement, InputStream sourceStream, char delimiter, boolean strict, String fileFormat, String constantColumnValue, int skipHeaderLines, int skipFooterLines, boolean trim, boolean noHeader, List<String> columnNames) throws IOException
      Constructor used when importing from an InputStream - e.g. QuandlImporter.*

      Note that the CSV importer will always close any table writers it uses, so it is up to the factory to provide appending writers if appropriate.

      Parameters:
      log - Logger created upstream
      importTableWriterFactory - Provides table writers on demand based on the type of import (single vs multi partition)
      intradayPartitionColumn - Column to use for determining the target partition for multi partition imports (generally Date)
      sourceElement - InputStream from which to read CSV data
      sourceStream - InputStream from which to read CSV data
      delimiter - Single character to be used as a delimiter - normally this is a comma
      strict - Whether to fail if a field fails numeric conversion or a column is missing a setter
      fileFormat - Apache commons CSV file format to use
      constantColumnValue - A String to materialize as the source column when an ImportColumn is defined with a sourceType of CONSTANT (aka ImporterColumnDefinition$IrisImportConstant). Can be null.
      skipHeaderLines - Number of lines to skip before the header. Use this for files that have leading "garbage".
      skipFooterLines - Number of lines to skip at the end of the file. Use this for files that have trailing "garbage".
      trim - If trim is true, use CSVFormat that trims leading and trailing blanks.
      noHeader - Whether the CSV does not include a header row with column names.
      columnNames - A list of column names to use instead of a header from the CSV.
      Throws:
      IOException - is thrown in case of IO failure when handling given sourceStream

  • Method Details