Package com.illumon.iris.db.tables.dataimport.logtailer

Class DataImportServer

java.lang.Object

com.fishlib.util.process.OnetimeShutdownTask

com.illumon.iris.db.tables.dataimport.logtailer.DataImportServer

All Implemented Interfaces:

com.fishlib.util.process.ShutdownManager.Task, IDataImportServer, DataRoutingConfigListener

public class DataImportServer
extends com.fishlib.util.process.OnetimeShutdownTask
implements IDataImportServer, DataRoutingConfigListener

Data Import Server: Consumes row-oriented data from log tailers (or similar data sources) and persists Iris' column-oriented table format, simultaneously serving said data consistently to remote table data clients.

Field Summary

Fields

Modifier and Type	Field	Description
static final long	LOG_TAILER_ACK_TIMEOUT_DEFAULT
static final String	LOG_TAILER_ACK_TIMEOUT_PROP
static final long	LOG_TAILER_POLL_PAUSE_DEFAULT
static final String	LOG_TAILER_POLL_PAUSE_PROP

Constructor Summary

Constructors

Constructor	Description
DataImportServer(com.fishlib.io.logger.Logger logNotUsed, DataImportServiceConfig disConfig, com.fishlib.configuration.Configuration configuration)	Deprecated. use DataImportServer(DataImportServerLogFactory, String, Configuration, SchemaService, DataRoutingService, String)
DataImportServer(DataImportServerLogFactory logFactory, DataImportServiceConfig disConfig, com.fishlib.configuration.Configuration configuration, SchemaService schemaService)	Deprecated. use DataImportServer(DataImportServerLogFactory, String, Configuration, SchemaService, DataRoutingService, String)
DataImportServer(DataImportServerLogFactory logFactory, String disName, com.fishlib.configuration.Configuration configuration, SchemaService schemaService, DataRoutingService routingService, String storageRoot)	Construct a DataImportServer.

Method Summary

Modifier and Type	Method	Description
SimpleDataImportStreamProcessor	createSimpleProcessor(String description, FullTableLocationKey streamKey, Runnable closeCallback, SimpleDataImportStreamProcessor.OnReject onReject)	Create a SimpleDataImportStreamProcessor for importing data into this DataImportServer.
static DataImportServer	getDataImportServer(com.fishlib.io.logger.Logger log, DataImportServiceConfig disConfig, com.fishlib.configuration.Configuration configuration, SchemaService schemaService)	Deprecated.
static DataImportServer	getDataImportServer(DataImportServerLogFactory logFactory, DataImportServiceConfig disConfig, com.fishlib.configuration.Configuration configuration, SchemaService schemaService)	Deprecated. use getDataImportServer(DataImportServerLogFactory, String, Configuration, SchemaService, DataRoutingService, String)
static DataImportServer	getDataImportServer(DataImportServerLogFactory logFactory, String disName, com.fishlib.configuration.Configuration configuration, SchemaService schemaService, DataRoutingService routingService, String storageRoot)	Create a new DataImportServer instance, according to the configuration passed in.
SchemaService	getSchemaService()
TableDataService	getTableDataService()
static void	main(String... args)
void	onConfigChange(long version)	Called when the configuration changes.
void	shutdown()
void	start()	Deprecated. use startInWorker() instead
void	start(io.deephaven.enterprise.stats.StatsIntradayLogger statsIntradayLogger)	Beginning tracking statistics and accepting connections.
void	startInWorker()	Start accepting connections in-worker.

Methods inherited from class com.fishlib.util.process.OnetimeShutdownTask

adapt, awaitShutdown, awaitShutdown, invoke, isShutdown

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface com.illumon.iris.db.tables.dataimport.logtailer.IDataImportServer

isShutdown

Field Details

LOG_TAILER_POLL_PAUSE_PROP

public static final String LOG_TAILER_POLL_PAUSE_PROP

See Also:

• Constant Field Values

LOG_TAILER_ACK_TIMEOUT_PROP

public static final String LOG_TAILER_ACK_TIMEOUT_PROP

See Also:

• Constant Field Values

LOG_TAILER_POLL_PAUSE_DEFAULT

public static final long LOG_TAILER_POLL_PAUSE_DEFAULT

See Also:

• Constant Field Values

LOG_TAILER_ACK_TIMEOUT_DEFAULT

public static final long LOG_TAILER_ACK_TIMEOUT_DEFAULT

See Also:

• Constant Field Values

Constructor Details

DataImportServer

@Deprecated
public DataImportServer(@Nullable
 DataImportServerLogFactory logFactory,
 @NotNull
 DataImportServiceConfig disConfig,
 @NotNull
 com.fishlib.configuration.Configuration configuration,
 @NotNull
 SchemaService schemaService)
                 throws IOException

Deprecated.

use DataImportServer(DataImportServerLogFactory, String, Configuration, SchemaService, DataRoutingService, String)

Delegate to DataImportServer(DataImportServerLogFactory, String, Configuration, SchemaService, DataRoutingService, String) with assumed defaults where required.

Parameters:

logFactory - optional DataImportServerLogFactory, will be used to create logger. If null, a global instance will be used, if available.

disConfig - used only to extract the config name

configuration - the configuration to use when fetching settings that aren't included in disConfig

schemaService - the SchemaService to use

Throws:

IOException - It doesn't really throw this.

DataImportServer

@Deprecated
public DataImportServer(@Nullable
 com.fishlib.io.logger.Logger logNotUsed,
 @NotNull
 DataImportServiceConfig disConfig,
 @NotNull
 com.fishlib.configuration.Configuration configuration)
                 throws IOException

Deprecated.

use DataImportServer(DataImportServerLogFactory, String, Configuration, SchemaService, DataRoutingService, String)

Removed deprecated method. This constructor throws a clear error to any users and should be removed in the next release (San Luis).

Parameters:

logNotUsed - not used

disConfig - not used

configuration - not used

Throws:

IOException

DataImportServer

public DataImportServer(@Nullable
 DataImportServerLogFactory logFactory,
 @NotNull
 String disName,
 @NotNull
 com.fishlib.configuration.Configuration configuration,
 @NotNull
 SchemaService schemaService,
 @NotNull
 DataRoutingService routingService,
 @Nullable
 String storageRoot)

Construct a DataImportServer. The routingService will be used to retrieve and monitor the DIS configuration, and keep it up to date if it is a DataRoutingService that supports change notification.

Parameters:

logFactory - optional DataImportServerLogFactory, will be used to create logger. If null, a global instance will be used, if available.

disName - the name of the DIS configuration in the data routing service

configuration - the configuration to use when fetching settings that aren't included in disConfig

schemaService - the SchemaService to use

routingService - the DataRoutingService to use for configuration

storageRoot - optional storage root to use if the DIS configuration specifies "private"

Method Details

getTableDataService

public TableDataService getTableDataService()

Specified by:

getTableDataService in interface IDataImportServer

getSchemaService

public SchemaService getSchemaService()

Specified by:

getSchemaService in interface IDataImportServer

start

@Deprecated
public void start()
           throws IOException

Deprecated.

use startInWorker() instead

Removed deprecated method. Delete in next release (San Luis).

Throws:

IOException

startInWorker

public void startInWorker()
                   throws IOException

Start accepting connections in-worker.

Specified by:

startInWorker in interface IDataImportServer

Throws:

IOException

start

public void start(@NotNull
 io.deephaven.enterprise.stats.StatsIntradayLogger statsIntradayLogger)
           throws IOException

Beginning tracking statistics and accepting connections.

Parameters:

statsIntradayLogger - the stats intraday logger

Throws:

IOException

shutdown

public void shutdown()

Specified by:

shutdown in interface IDataImportServer

Specified by:

shutdown in class com.fishlib.util.process.OnetimeShutdownTask

createSimpleProcessor

public SimpleDataImportStreamProcessor createSimpleProcessor(@NotNull
 String description,
 @NotNull
 FullTableLocationKey streamKey,
 @NotNull
 Runnable closeCallback,
 @NotNull
 SimpleDataImportStreamProcessor.OnReject onReject)

Create a SimpleDataImportStreamProcessor for importing data into this DataImportServer.

Specified by:

createSimpleProcessor in interface IDataImportServer

Parameters:

description - a description for the data connection

streamKey - destination information

closeCallback - callback to be invoked when the simple data stream is closed

onReject - callback to be invoked if the simple data stream is rejected

Returns:

a new SimpleDataImportStreamProcessor

main

public static void main(String... args)
                 throws IOException

Throws:

IOException

getDataImportServer

@Deprecated
public static DataImportServer getDataImportServer(@Nullable
 DataImportServerLogFactory logFactory,
 DataImportServiceConfig disConfig,
 com.fishlib.configuration.Configuration configuration,
 SchemaService schemaService)
                                            throws IOException

Deprecated.

use getDataImportServer(DataImportServerLogFactory, String, Configuration, SchemaService, DataRoutingService, String)

Deprecated. Use getDataImportServer(DataImportServerLogFactory, String, Configuration, SchemaService, DataRoutingService, String) instead.

Parameters:

logFactory - optional DataImportServerLogFactory, will be used to create logger. If null, a global instance will be used, if available.

disConfig - used only to extract the config name

configuration - the configuration to use when fetching settings that aren't included in disConfig

schemaService - the SchemaService to use

Returns:

a new DataImportServer instance

Throws:

IOException

getDataImportServer

@Deprecated
public static DataImportServer getDataImportServer(com.fishlib.io.logger.Logger log,
 DataImportServiceConfig disConfig,
 com.fishlib.configuration.Configuration configuration,
 SchemaService schemaService)
                                            throws IOException

Deprecated.

Removed deprecated method. Throws a clear error message and should be removed in the next release (San Luis).

Returns:

never returns.

Throws:

IOException

getDataImportServer

public static DataImportServer getDataImportServer(@Nullable
 DataImportServerLogFactory logFactory,
 @NotNull
 String disName,
 @NotNull
 com.fishlib.configuration.Configuration configuration,
 @NotNull
 SchemaService schemaService,
 @NotNull
 DataRoutingService routingService,
 @Nullable
 String storageRoot)

Create a new DataImportServer instance, according to the configuration passed in. The routingService will be used to retrieve and monitor the DIS configuration, and keep it up to date if it is a DataRoutingService that supports change notification.

Parameters:

logFactory - optional DataImportServerLogFactory, will be used to create logger. If null, a global instance will be used, if available.

disName - the name of the DIS configuration to use

configuration - the configuration to use when fetching settings that aren't included in disConfig

schemaService - the SchemaService to use

routingService - the DataRoutingService to use for configuration

storageRoot - optional storage root to use when configured storage is "private"

Returns:

a new DataImportServer instance

onConfigChange

public void onConfigChange(long version)

Description copied from interface: DataRoutingConfigListener

Called when the configuration changes. Additional heartbeat notifications are sent periodically.

Specified by:

onConfigChange in interface DataRoutingConfigListener

Parameters:

version - the new/current version of the configuration.