Package com.illumon.iris.controller

Class PersistentQueryController

java.lang.Object

com.fishlib.util.process.OnetimeShutdownTask

com.illumon.iris.controller.PersistentQueryController

All Implemented Interfaces:

com.fishlib.util.process.ShutdownManager.Task

public final class PersistentQueryController
extends com.fishlib.util.process.OnetimeShutdownTask

Stand-alone controller for Iris.

Hosts a configuration PHT, a state PDHT, monitors server status, and dispatches stop/start commands.

Thread-safety notes:

• All changes to IrisQueryInfos are protected by the lock on the associated PDHT's map.
• All PersistentQueryConfiguration instances that are published (in the associated PHT or PersistentQueryInfo) are guaranteed not to be further mutated. Hence, they can be read safely at all times.
• All changes to a PersistentQueryConfiguration (by serial, not by instance, since published instances are immutable) are protected by a serial-specific lock. Query starts/stops are also protected, since they must always use the most up-to-date version of the configuration.

Configuration properties

• PersistentQueryController.host - The hostname of this controller
• PersistentQueryController.port - The port for communication
• PersistentQueryController.useLocalGit (optional) - When set to true use the local filesystem as a repository
• PersistentQueryController.gitGcEnabled (optional) - When set to true a git gc will be periodically performed
• PersistentQueryController.statusCheckTimes - When an AnomalyClient is used, defines the times to check in with status
• PersistentQueryController.failureWindowMillis - How long failures are queued
• PersistentQueryController.failureNumQueriesThreshold - How many failures are required before notifying the AnomalyClient
• PersistentQueryController.failureResendMillis - The minimum gap between failure messages
• PersistentQueryController.commitCheckpointImmediate (optional) - Immediately checkpoint the PDHT state on startup
• RemoteProcessingRequestProfile.defaultProfile - The default JVM args profile to use
• RemoteProcessingRequestProfile.noneProfile - The JVM profile to use when "None" is selected

Field Summary

Fields

Modifier and Type	Field	Description
static final double	DEFAULT_MAX_HEAP_SIZE_GB
static final int	ETCD_PRESENCE_KVS_LOAD_BATCH_SIZE
static final int	ETCD_PRESENCE_KVS_LOAD_TIMEOUT_SECONDS
static final int	ETCD_PRESENCE_LEASE_KEEPALIVE_TIMEOUT_MS	How long will we wait for our etcd lease to be granted or renewed, in milliseconds before determining that we had a failure.
static final int	ETCD_PRESENCE_LEASE_RENEW_RETRY_DELAY_SECONDS	What is the maximum frequency which we will try to renew a lease.
static final int	ETCD_PRESENCE_LEASE_RENEW_SECONDS	How often, in seconds, do we renew our etcd lease.
static final int	ETCD_PRESENCE_LEASE_TTL_SECONDS	How long does our etcd lease last in seconds? If we do not renew the lease for this many seconds, then etcd will automatically revoke it, and our presence KV will be removed.
static final int	MAX_DESYNC_RETRIES
static final int	MAX_POSSIBLE_HEAP_GB

Constructor Summary

Constructors

Constructor

Description

PersistentQueryController(String processNameForLogging, String processInfoId, io.deephaven.enterprise.auth.audit.AuditEventLogger auditEventLogger, PersistentQueryStateLogger stateLogLogger, PersistentQueryConfigurationLogger configurationLogLogger, ResourceUtilizationLogger resourceUtilizationLogger, com.fishlib.io.sched.Scheduler scheduler, com.fishlib.net.ConnectionMonitor.Params connectionMonitorParams, com.fishlib.anomalyclient.SimpleAnomalyClientImpl chatBotClient, ScriptPathLoaderManager scriptPathLoaderManager, String errorEmailRecipientAddress, IrisGroupProvider irisGroupProvider, ScheduledExecutorService executorService, io.deephaven.enterprise.auth.AuthenticationClient authClient, Function<String,PublicPrivateKeyEncryption> encryptionFactory, ControllerDispatcherHandler.ConnectionFactory dispatcherConnectionFactory, io.deephaven.shadow.jetcd.io.etcd.jetcd.Client etcdClient, io.deephaven.enterprise.etcdclient.presence.PresenceLeaseHandler.Factory presenceLeaseHandlerFactory)

Method Summary

Modifier and Type	Method	Description
io.deephaven.enterprise.auth.AuthenticationClient	getAuthClient()
PersistentQueryConfigurationLogger	getConfigurationLogLogger()
static String	getHost()
static String	getName()
static int	getPort()
QueryHandleContext.PresenceManager	getPresenceManager()
Map<String,WorkerKind>	getWorkerKindMap()
static void	main(String... args)
void	shutdown()
void	start(io.deephaven.shadow.core.io.grpc.internal.AbstractServerImplBuilder<?> serverBuilder, Supplier<PqStorage> pqStorageFactory, boolean loadStateFromPresenceKvs)

Methods inherited from class com.fishlib.util.process.OnetimeShutdownTask

adapt, awaitShutdown, awaitShutdown, invoke, isShutdown

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

MAX_POSSIBLE_HEAP_GB

public static final int MAX_POSSIBLE_HEAP_GB

See Also:

• Constant Field Values

DEFAULT_MAX_HEAP_SIZE_GB

public static final double DEFAULT_MAX_HEAP_SIZE_GB

MAX_DESYNC_RETRIES

public static final int MAX_DESYNC_RETRIES

ETCD_PRESENCE_LEASE_TTL_SECONDS

public static final int ETCD_PRESENCE_LEASE_TTL_SECONDS

How long does our etcd lease last in seconds? If we do not renew the lease for this many seconds, then etcd will automatically revoke it, and our presence KV will be removed. Any process using our presence KV as sign of liveness will consider us dead.

ETCD_PRESENCE_LEASE_RENEW_SECONDS

public static final int ETCD_PRESENCE_LEASE_RENEW_SECONDS

How often, in seconds, do we renew our etcd lease. This must be less than ETCD_PRESENCE_LEASE_TTL_SECONDS; for safety it should be no more than 50% of the TTL value.

ETCD_PRESENCE_LEASE_RENEW_RETRY_DELAY_SECONDS

public static final int ETCD_PRESENCE_LEASE_RENEW_RETRY_DELAY_SECONDS

What is the maximum frequency which we will try to renew a lease. If a lease renewal times out, we will try again more quickly than the standard ETCD_LEASE_RENEW_SECONDS.

ETCD_PRESENCE_LEASE_KEEPALIVE_TIMEOUT_MS

public static final int ETCD_PRESENCE_LEASE_KEEPALIVE_TIMEOUT_MS

How long will we wait for our etcd lease to be granted or renewed, in milliseconds before determining that we had a failure.

ETCD_PRESENCE_KVS_LOAD_TIMEOUT_SECONDS

public static final int ETCD_PRESENCE_KVS_LOAD_TIMEOUT_SECONDS

ETCD_PRESENCE_KVS_LOAD_BATCH_SIZE

public static final int ETCD_PRESENCE_KVS_LOAD_BATCH_SIZE

Constructor Details

PersistentQueryController

public PersistentQueryController(@NotNull
 String processNameForLogging,
 @NotNull
 String processInfoId,
 @NotNull
 io.deephaven.enterprise.auth.audit.AuditEventLogger auditEventLogger,
 @NotNull
 PersistentQueryStateLogger stateLogLogger,
 @NotNull
 PersistentQueryConfigurationLogger configurationLogLogger,
 @NotNull
 ResourceUtilizationLogger resourceUtilizationLogger,
 @NotNull
 com.fishlib.io.sched.Scheduler scheduler,
 @Nullable
 com.fishlib.net.ConnectionMonitor.Params connectionMonitorParams,
 @Nullable
 com.fishlib.anomalyclient.SimpleAnomalyClientImpl chatBotClient,
 @NotNull
 ScriptPathLoaderManager scriptPathLoaderManager,
 @Nullable
 String errorEmailRecipientAddress,
 @NotNull
 IrisGroupProvider irisGroupProvider,
 @NotNull
 ScheduledExecutorService executorService,
 @NotNull
 io.deephaven.enterprise.auth.AuthenticationClient authClient,
 @NotNull
 Function<String,PublicPrivateKeyEncryption> encryptionFactory,
 @NotNull
 ControllerDispatcherHandler.ConnectionFactory dispatcherConnectionFactory,
 @NotNull
 io.deephaven.shadow.jetcd.io.etcd.jetcd.Client etcdClient,
 @NotNull
 io.deephaven.enterprise.etcdclient.presence.PresenceLeaseHandler.Factory presenceLeaseHandlerFactory)
                          throws IOException,
ConfigurationVerificationException

Throws:

IOException

ConfigurationVerificationException

Method Details

getName

public static String getName()

getHost

public static String getHost()

getPort

public static int getPort()

getWorkerKindMap

public Map<String,WorkerKind> getWorkerKindMap()

start

public void start(@NotNull
 io.deephaven.shadow.core.io.grpc.internal.AbstractServerImplBuilder<?> serverBuilder,
 @NotNull
 Supplier<PqStorage> pqStorageFactory,
 boolean loadStateFromPresenceKvs)
           throws IOException,
PqStorageException

Throws:

IOException

PqStorageException

shutdown

public void shutdown()

Specified by:

shutdown in class com.fishlib.util.process.OnetimeShutdownTask

main

public static void main(String... args)
                 throws IOException,
ClassNotFoundException,
ConfigurationVerificationException,
PqStorageException

Throws:

IOException

ClassNotFoundException

ConfigurationVerificationException

PqStorageException

getConfigurationLogLogger

public PersistentQueryConfigurationLogger getConfigurationLogLogger()

getAuthClient

@NotNull
public io.deephaven.enterprise.auth.AuthenticationClient getAuthClient()

getPresenceManager

public QueryHandleContext.PresenceManager getPresenceManager()