Package com.illumon.iris.importers

Class BaseMergeData

java.lang.Object

com.illumon.iris.importers.BaseMergeData

Direct Known Subclasses:

MergeIntradayData, RemergeData

public abstract class BaseMergeData
extends Object

Re-writes an input table to multi-day storage, (merged, partitioned, grouped, and sorted as appropriate). Create concrete instances via MergeData

TODO: Experiment with sequential reading and splayed writing, instead of the current (reversed) operation

Field Summary

Fields

Modifier and Type	Field	Description
protected final Database	database
protected final com.fishlib.io.logger.Logger	logger
protected final MergeParameters	params

Constructor Summary

Constructors

Constructor	Description
BaseMergeData(Database db, MergeParameters params, com.fishlib.io.logger.Logger log)

Method Summary

Modifier and Type	Method	Description
static void	awaitJobs(List<Future<?>> futures)
static void	awaitSyncOperators(com.fishlib.io.logger.Logger log, String logPrefix, SyncOperator... syncOperators)
protected File[]	filterWritablePartitions(File[] originalWritablePartitions)	Filter the input array of candidate writable partitions.
static ExecutorService	makeExecutor(String name, int poolSize)	Make a suitable ExecutorService for use in merge processing.
final void	merge(com.fishlib.util.process.FatalErrorReporter fatalErrorReporter, StatusCallback statusCallback)	Read, partition, group, sort input data and re-write it into multi-day locations or parquet.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

logger

protected final com.fishlib.io.logger.Logger logger

database

protected final Database database

params

protected final MergeParameters params

Constructor Details

BaseMergeData

public BaseMergeData(@Nullable
 Database db,
 @NotNull
 MergeParameters params,
 @NotNull
 com.fishlib.io.logger.Logger log)

Method Details

merge

public final void merge(@NotNull
 com.fishlib.util.process.FatalErrorReporter fatalErrorReporter,
 @NotNull
 StatusCallback statusCallback)

Read, partition, group, sort input data and re-write it into multi-day locations or parquet.

Parameters:

fatalErrorReporter - An error reporter

statusCallback - A callback for status getInputData(String, TableDefinition)

filterWritablePartitions

protected File[] filterWritablePartitions(@NotNull
 File[] originalWritablePartitions)

Filter the input array of candidate writable partitions. This can be used by an overriding class to merge only a subset of the possible partitions. Partitions removed from the originalWritablePartitions array will not be touched.

Parameters:

originalWritablePartitions - the list of writable partitions to filter

Returns:

all partitions for which the merge should continue.

awaitJobs

public static void awaitJobs(@NotNull
 List<Future<?>> futures)

awaitSyncOperators

public static void awaitSyncOperators(@NotNull
 com.fishlib.io.logger.Logger log,
 @NotNull
 String logPrefix,
 @NotNull
 SyncOperator... syncOperators)

makeExecutor

public static ExecutorService makeExecutor(@NotNull
 String name,
 int poolSize)

Make a suitable ExecutorService for use in merge processing.

Parameters:

name - The name of the executor to be used in thread naming

poolSize - The number of threads that will be allowed in the executor (core and maximum)

Returns:

The new ExecutorService