Package com.illumon.iris.db.tables.dataimport.logtailer

Interface DataImportProcessor

All Superinterfaces:
CheckpointRecord.SourceFileSizeRecord, com.fishlib.base.log.LogOutputAppendable
All Known Implementing Classes:
BinaryStoreImportProcessor, CsvImportProcessor
public interface DataImportProcessor
extends CheckpointRecord.SourceFileSizeRecord
Interface to abstract details of consuming remote file data.

  • Field Summary

    Fields 
    Modifier and Type Field Description
    static int END_OF_DATA_FLAG  
    static int TRUNCATED_FLAG  

  • Method Summary

    Modifier and Type Method Description
    default boolean isDirty()
    Does this import processor contain inconsistent state that should prevent checkpointing?
    void onNewFile​(FilePosition filePosition)
    Notify the import processor that a new file is available.
    int processContent​(ByteBuffer dataBuffer, long bufferSentTimestamp)
    Request that the import processor consume data from the supplied buffer, the first byte of which is associated with the supplied filePosition.
    default void setContext​(DataImportStreamContext newContext)
    After truncating a partition, a new context might be required to update checkpoints and file writers.
    default boolean shouldBeginTransaction​(long nowMillis)
    Check if a transaction should be started.
    default boolean shouldEndTransaction​(long nowMillis, boolean isNewFile)
    Check if any currently in-progress transactions should be completed.

  • Field Details

  • Method Details

    • onNewFile

      void onNewFile​(@NotNull FilePosition filePosition)
      Notify the import processor that a new file is available.
      Parameters:
      filePosition - The new file's initial position

    • processContent

      int processContent​(@NotNull ByteBuffer dataBuffer, long bufferSentTimestamp)
      Request that the import processor consume data from the supplied buffer, the first byte of which is associated with the supplied filePosition.
      Parameters:
      dataBuffer - The data to process
      Returns:
      the reason flags for returning from the method

    • setContext

      default void setContext​(DataImportStreamContext newContext)
      After truncating a partition, a new context might be required to update checkpoints and file writers.
      Parameters:
      newContext - The replacement DataImportStreamContext

    • isDirty

      default boolean isDirty()
      Does this import processor contain inconsistent state that should prevent checkpointing?
      Returns:
      Whether this import processor contains inconsistent state that should prevent checkpointing

    • shouldEndTransaction

      default boolean shouldEndTransaction​(long nowMillis, boolean isNewFile)
      Check if any currently in-progress transactions should be completed.
      Parameters:
      nowMillis - The current time
      isNewFile - Whether the method is being invoked because of a new file
      Returns:
      Whether the transaction should be completed

    • shouldBeginTransaction

      default boolean shouldBeginTransaction​(long nowMillis)
      Check if a transaction should be started.
      Parameters:
      nowMillis - The current time
      Returns:
      Whether a transaction should begin