Package com.illumon.iris.db.tables.dataimport.logtailer

Interface DataImportProcessor

All Superinterfaces:
CheckpointRecord.SourceFileSizeRecord, com.fishlib.base.log.LogOutputAppendable
All Known Implementing Classes:
BinaryStoreImportProcessor, CsvImportProcessor
public interface DataImportProcessor extends CheckpointRecord.SourceFileSizeRecord
Interface to abstract details of consuming remote file data.

  • Field Summary

    Fields
    Modifier and Type
    Field
    Description
    static final int
    END_OF_DATA_FLAG
     
    static final int
    TRUNCATED_FLAG
     

  • Method Summary

    Modifier and Type
    Method
    Description
    default com.fishlib.base.log.LogOutput
    append(com.fishlib.base.log.LogOutput logOutput)
     
    String
    getName()
    Get the name of the file.
    long
    getSize()
    Get the size of the file.
    default boolean
    isDirty()
    Does this import processor contain inconsistent state that should prevent checkpointing?
    void
    onNewFile(FilePosition filePosition)
    Notify the import processor that a new file is available.
    int
    processContent(ByteBuffer dataBuffer, long bufferSentTimestamp)
    Request that the import processor consume data from the supplied buffer, the first byte of which is associated with the supplied filePosition.
    default void
    setContext(DataImportStreamContext newContext)
    After truncating a partition, a new context might be required to update checkpoints and file writers.
    default boolean
    shouldBeginTransaction(long nowMillis)
    Check if a transaction should be started.
    default boolean
    shouldEndTransaction(long nowMillis, boolean isNewFile)
    Check if any currently in-progress transactions should be completed.

  • Field Details

  • Method Details

    • onNewFile

      void onNewFile(@NotNull FilePosition filePosition)
      Notify the import processor that a new file is available.
      Parameters:
      filePosition - The new file's initial position

    • processContent

      int processContent(@NotNull ByteBuffer dataBuffer, long bufferSentTimestamp)
      Request that the import processor consume data from the supplied buffer, the first byte of which is associated with the supplied filePosition.
      Parameters:
      dataBuffer - The data to process
      Returns:
      the reason flags for returning from the method

    • setContext

      default void setContext(DataImportStreamContext newContext)
      After truncating a partition, a new context might be required to update checkpoints and file writers.
      Parameters:
      newContext - The replacement DataImportStreamContext

    • isDirty

      default boolean isDirty()
      Does this import processor contain inconsistent state that should prevent checkpointing?
      Returns:
      Whether this import processor contains inconsistent state that should prevent checkpointing

    • shouldEndTransaction

      default boolean shouldEndTransaction(long nowMillis, boolean isNewFile)
      Check if any currently in-progress transactions should be completed.
      Parameters:
      nowMillis - The current time
      isNewFile - Whether the method is being invoked because of a new file
      Returns:
      Whether the transaction should be completed

    • shouldBeginTransaction

      default boolean shouldBeginTransaction(long nowMillis)
      Check if a transaction should be started.
      Parameters:
      nowMillis - The current time
      Returns:
      Whether a transaction should begin

    • getName

      String getName()
      Get the name of the file. Must not be null, unless size is 0.
      Returns:
      The name of the file, as in File.getName()

    • getSize

      long getSize()
      Get the size of the file. Must be 0 if name is null.
      Returns:
      The size associated with the file

    • append

      default com.fishlib.base.log.LogOutput append(@NotNull com.fishlib.base.log.LogOutput logOutput)
      Specified by:
      append in interface com.fishlib.base.log.LogOutputAppendable