Package com.illumon.iris.db.tables.dataimport.logtailer

Class LogtailerDestinationManager

java.lang.Object
com.illumon.iris.db.tables.dataimport.logtailer.LogtailerDestinationManager
public class LogtailerDestinationManager
extends Object
A LogtailerDestinationManager instance sends data for a fully-defined set of logs to a specific destination. If multiple destinations are required, each one requires a LogtailerDestinationManager instance. A fully-defined set of logs includes the following: - namespace/table - internal partition - column partition - log path prefix (i.e. the full path and filename prefix for this LogtailerDestinationManager) - destination details

  • Constructor Details

    • LogtailerDestinationManager

      public LogtailerDestinationManager​(com.fishlib.io.logger.Logger log, Comparator<File> fileComparator, SocketAddress remoteAddress, @NotNull FullTableLocationKey table, Long idleTimeMillis, String pathPrefix, @Nullable TokenBucketThrottle tableThrottle, @Nullable TokenBucketThrottle destinationThrottle, boolean sendDoneMessage, boolean logDetailedFileInfo, boolean logBytesSent, int retryCount, long retryPauseMillis, long pollPauseMillis, File[] matchingFiles, String destination, DataImportFormat streamFormat, Map<String,​String> metadata, String tableLocationIdentifierString)
      Construct a manager to tail a log stream to one destination.
      Parameters:
      log - the log to which errors will be sent
      fileComparator - a valid file comparator for the binary log files, returns a result based on their inherent ordering
      remoteAddress - the address of the remote DataImportServer for this LogtailerDestinationManager
      table - the table for which this LogtailerDestinationManager will send data
      idleTimeMillis - if not null, if the thread doesn't send data in this amount of time, it should terminate
      pathPrefix - the fully-qualified path this LogtailerDestinationManager will use for file searches, the search will look for any files beginning with this value
      tableThrottle - if provided, a throttle to be applied at the table level
      destinationThrottle - if provided, a throttle to be applied at the destination level
      sendDoneMessage - specifies whether or not to send the done message to the DIS when the thread completes
      logDetailedFileInfo - - if true, detailed information on file reads is logged
      logBytesSent - - if true, log events detailing the number of bytes sent are generated every time data is sent to a DIS
      retryCount - number of retries for remote destination
      retryPauseMillis - pause in millis between connection retries (the pause will be longer if too many errors occur, as in sleepWithBackoff(String))
      pollPauseMillis - pause in millis between file polling
      matchingFiles - the current array of files for this instance
      destination - the destination name, used for logging
      tableLocationIdentifierString - the table location identifier string, used for logging

  • Method Details

    • setMatchingFiles

      public void setMatchingFiles​(File[] matchingFiles)

    • run

      public void run​(Runnable fileFinderMethod, Runnable completedMethod, String tableIdentifierDetails)
      Run this LogtailerDestinationManager. If exceptions occur, keep trying until the termination time has passed.
      Parameters:
      fileFinderMethod - a method to be called to update the file list in the event that files seem to disappear
      completedMethod - a method to be called when this thread completes, it must return true if it's okay to terminate and call setCompleted()
      tableIdentifierDetails - the details for this table identifier, used when logging

    • startIfNeeded

      public void startIfNeeded​(@NotNull File newFile, Runnable fileFinderMethod, Runnable completedMethod, Runnable startedMethod, String tableIdentifierDetails)
      Start a new destination thread if needed.
      Parameters:
      newFile - the new file that triggered this start request
      fileFinderMethod - a method to be called to update the file list in the event that files seem to disappear
      completedMethod - a method to be called when this thread completes, it must return true if it's okay to terminate and call setCompleted()
      startedMethod - a method to be called if the thread starts
      tableIdentifierDetails - the details for this table identifier, used when logging

    • requestShutdown

      public void requestShutdown()
      Request that the LogtailerDestinationManager thread terminate