Package com.illumon.iris.db.tables.dataimport.logtailer

Class LogtailerDestinationManager

java.lang.Object
com.illumon.iris.db.tables.dataimport.logtailer.LogtailerDestinationManager
public class LogtailerDestinationManager extends Object
A LogtailerDestinationManager instance sends data for a fully-defined set of logs to a specific destination. If multiple destinations are required, each one requires a LogtailerDestinationManager instance. A fully-defined set of logs includes the following:
  • namespace/table
  • internal partition
  • column partition
  • log path prefix (i.e. the full path and filename prefix for this LogtailerDestinationManager)
  • destination details

  • Constructor Details

    • LogtailerDestinationManager

      public LogtailerDestinationManager(@NotNull com.fishlib.io.logger.Logger log, @NotNull Comparator<File> fileComparator, @NotNull DataRoutingService.DeferredEndpoint remoteAddress, @NotNull FullTableLocationKey table, @NotNull Long idleTimeMillis, @NotNull String pathPrefix, @Nullable TokenBucketThrottle tableThrottle, @Nullable TokenBucketThrottle destinationThrottle, boolean sendDoneMessage, boolean logDetailedFileInfo, boolean logBytesSent, int retryCount, long retryPauseMillis, long pollPauseMillis, long terminationWaitTimeMillis, @NotNull File[] matchingFiles, @NotNull String destination, @NotNull DataImportFormat streamFormat, @NotNull Map<String,String> metadata, @NotNull TableLocationIdentifier tableLocationIdentifier, @NotNull LogtailerDestinationManager.FinishedFileConsumer finishedFileConsumer)
      Construct a manager to tail a log stream to one destination.
      Parameters:
      log - the log to which errors will be sent
      fileComparator - a valid file comparator for the binary log files, returns a result based on their inherent ordering
      remoteAddress - the address of the remote DataImportServer for this LogtailerDestinationManager
      table - the table for which this LogtailerDestinationManager will send data
      idleTimeMillis - if not null, if the thread doesn't send data in this amount of time, it should terminate
      pathPrefix - the fully-qualified path this LogtailerDestinationManager will use for file searches, the search will look for any files beginning with this value
      tableThrottle - if provided, a throttle to be applied at the table level
      destinationThrottle - if provided, a throttle to be applied at the destination level
      sendDoneMessage - specifies whether to send the done message to the DIS when the thread completes
      logDetailedFileInfo - - if true, detailed information on file reads is logged
      logBytesSent - - if true, log events detailing the number of bytes sent are generated every time data is sent to a DIS
      retryCount - number of retries for remote destination
      retryPauseMillis - pause in millis between connection retries (the pause will be longer if too many errors occur, as in sleepWithBackoff(String, long, BooleanSupplier)
      pollPauseMillis - pause in millis between file polling
      terminationWaitTimeMillis - the max time to wait while shutting down destination managers. This governs sleep and wait cycles.
      matchingFiles - the current array of files for this instance
      destination - the destination name, used for logging
      streamFormat - the DataImportFormat of the current stream
      metadata - Any metadata provided by the source
      tableLocationIdentifier - the table location identifier, used for logging and determination of endpoints during cleanup
      finishedFileConsumer - Consumer that is called when a binlog file has been processed

  • Method Details

    • setMatchingFiles

      public void setMatchingFiles(File[] matchingFiles)

    • run

      public void run(Runnable fileFinderMethod, Runnable completedMethod, String tableIdentifierDetails)
      Run this LogtailerDestinationManager. If exceptions occur, keep trying until the termination time has passed.
      Parameters:
      fileFinderMethod - a method to be called to update the file list in the event that files seem to disappear
      completedMethod - a method to be called when this thread completes, it must return true if it's okay to terminate and call setCompleted()
      tableIdentifierDetails - the details for this table identifier, used when logging

    • startIfNeeded

      public void startIfNeeded(@NotNull File newFile, Runnable fileFinderMethod, Runnable completedMethod, Runnable startedMethod, String tableIdentifierDetails)
      Start a new destination thread if needed.
      Parameters:
      newFile - the new file that triggered this start request
      fileFinderMethod - a method to be called to update the file list in the event that files seem to disappear
      completedMethod - a method to be called when this thread completes, it must return true if it's okay to terminate and call setCompleted()
      startedMethod - a method to be called if the thread starts
      tableIdentifierDetails - the details for this table identifier, used when logging

    • requestShutdown

      public void requestShutdown()
      Request that the LogtailerDestinationManager thread terminate

    • getRemoteEndpointName

      protected String getRemoteEndpointName()