Configuration Overview

Most Deephaven server configuration is in the following locations:

/etc/sysconfig - This is where the Deephaven installation-specific files reside. Files under this path are not modified when a system update occurs. Of particular interest:

/etc/sysconfig/illumon.confs/illumon.iris.hostconfig - Also may be found via the soft-link /etc/sysconfig/illumon. This file controls the environment for each Deephaven server-side process (run via service iris <service name>).
/etc/sysconfig/illumon.d/resources - Configuration files for each Deephaven server-side process (referenced by the hostconfig).
/etc/sysconfig/illumon.d/schema - Database schema files (XML). This is the recommended location for customer database schema files.
/etc/sysconfig/illumon.d/java_lib - Custom user and third-party Java libraries. "Extension" libraries placed here will be found on the classpath.
/etc/sysconfig/illumon.d/override - This path contains any class files that should override those contained in java_lib jars.
/etc/sysconfig/illumon.d/hotfixes - This path contains any jar files that should be used instead of those contained in the java_lib directories.

/usr/illumon - This is where Deephaven system files reside. These are not meant to be modified except when a new release occurs. The current release is referenced by the soft-link /usr/illumon/latest. While it should not be necessary to modify these files, it may be useful to be aware of the following:

/usr/illumon/latest/etc - System-wide internal configuration and defaults.
/usr/illumon/latest/java_lib - Deephaven system Java libraries.

Deephaven uses several formats of configuration data to control its operation. For core Deephaven services, there are three main forms of configuration:

Java properties files - These are text files, usually named with a .prop extension, that store key-value pairs. These are the largest stores of Deephaven configuration data, and control most of its operations and behavior.
XML files - Some Deephaven processes and utilities use eXtensible Markup Language files. XML is a text format, but with specific requirements for delimitation of data elements, and with support for hierarchical representations of information.
YAML files - These files, named with a .yml extension, are used to configure the Data Routing Service. The YAML configuration file format centralizes the information governing the locations, servers, and services that determine how data is handled.
Other configuration files - These are also text files, but their format is specified by the applications that use them. The main example for Deephaven of this type of file is the hostconfig file that configures process startup for services. In this case, the file is actually a shell script.

There are also other configuration settings and other uses of the principal configuration files in Deephaven. The details of these are described in the following sections.

Properties Files

Most Deephaven processes have their own unique configuration (properties) file. The file is specified in hostconfig as CONFIGFILE=... for each Deephaven process. Each properties file includes IRIS-CONFIG.prop where common settings are specified, as well as iris-defaults.prop, where the Deephaven default properties are specified.

The Deephaven process properties file and IRIS-CONFIG.prop need to be in the CLASSPATH and are therefore stored in:

/etc/sysconfig/illumon.d/resources/*.prop

The iris-defaults.prop file is stored in /usr/illumon/latest/etc/ and should not be edited directly by customers. Any property in the default properties file can be overridden in IRIS-CONFIG.prop to be picked up by all Deephaven processes, or overridden in a later properties file for a specific Deephaven process.

Some of the properties refer to other configuration files (of various types) including .txt, .xml, or .yml files as needed by the various Deephaven processes. If an absolute system path is specified by the property, it will be used. Otherwise, they are loaded from the CLASSPATH. These files typically are stored in the /etc/sysconfig/illumon.d/resources/ folder.

Properties Files Syntax

Properties files, or "prop" files, are Java constructs that provide applications with easy access to a collection of settings. Properties files can contain comments (starting with a pound/hash sign [#]) and key=value settings. Deephaven extensions allow other files to be included in properties files using an includefiles directive, as shown in the example below:

includefiles=iris-defaults.prop,my-settings.prop

...

########## Iris DB RemoteQueryDispatcher settings ##########

RemoteQueryUtils.hostnameSuffixesToTruncateForDisplay=

RemoteQueryDispatcherParameters.name=default
RemoteQueryDispatcherParameters.host=localhost
RemoteQueryDispatcherParameters.queryPort=22013
RemoteQueryDispatcher.workerPort=22012
RemoteQueryDispatcher.workerServerPorts=23000-23999
RemoteQueryDispatcher.webPort=8084

Note that everything before the first equals sign is the key name, and everything after it is the value. If a value is a list, it is up to the application to define how that list should be formatted. Some Deephaven lists (such as tailer configuration file names and includefiles names) are comma-delimited, while others (such as schema paths) are semi-colon delimited.

A property name without an assigned value (such as the RemoteQueryUtils.hostnameSuffixesToTruncateForDisplay property above) will be treated as defining the property to have an empty value (i.e. an empty string). This is not the same as if the property does not exist in the properties files.

Only one includefiles directive is allowed in a properties file, but it can list multiple files to include, and these can, in turn, include other files.

There is no uniqueness requirement when adding a setting into a properties file. Instead, the properties collection that applications use within Java provides a "last in, wins" view of the settings. If a value is set multiple times, in a single properties file, or in multiple properties files linked with includefiles, whichever setting is applied last is the one that is used. Properties files are evaluated from top to bottom. Properties file entries are always case-sensitive for the property name, and often case-sensitive for the value.

The iris-defaults.prop properties file is delivered with the Deephaven installation. It provides default values for settings that can be changed, but rarely need to be, and also provides initial values for properties that will most likely need to be updated in production environments. These latter properties might work in a single-machine test environment, but need to be set differently when Deephaven is scaled to multiple servers. As mentioned above, the recommendation is to have a custom properties file (or, potentially, multiple custom properties files for different services, servers, or users) that includes iris-defaults.prop or IRIS-CONFIG.prop at the top, and then has its own specific settings further down that add to or override the inherited settings.

Java also allows for properties to be set for a process by passing them as JVM arguments. These are of the form:

-DpropertyName=propertyValue

Settings using -D are applied after properties file(s) settings, so they have the highest priority and override settings taken from properties files.

The properties file that a Deephaven process will load is set with a -D property as well:

-DConfiguration.rootFile=myIrisSettings.prop

The process will attempt to load this file (myIrisSettings.prop) from the Java classpath. This involves searching both in filesystem paths and also inside any JAR files that are included in the classpath. The first instance of a file in the classpath that matches the name will be loaded. Any included files encountered while loading properties file(s) will also be found from the classpath. This loading happens when the Deephaven Configuration class is initialized by the process. After that, all properties are kept in memory, so in most cases changes to properties file settings will not be picked up until the processes that use them are restarted.

The Property Inspector Utility

The Property Inspector class in the Deephaven Util JAR can inspect properties files or chains of properties files to provide visibility into what properties files are being included, where properties are set, and where properties are duplicated or overridden.

On a Deephaven server, this utility can be executed using iris_exec, as follows:

/usr/illumon/latest/bin/iris_exec property_inspector

Parameter Value	Description
`-pf`,`--propertyFile <value>`	(required) The starting properties file to read.
`-cs`,`--csv`	(optional) Format output as CSV (for easy import into a spreadsheet).
`-fm`,`--mode <value in {ALL (default)`, `ALL_DUPLICATES, FILE_DUPLICATES}>`	(optional) Search mode. `ALL_DUPLICATES` will show properties whose value is set more than once anywhere in the chain of properties files, while `FILE_DUPLICATES` will only show properties whose values are set more than once in the same file.
`-pn`,`--propertyName <value>`	(optional) A specific property to search for. Asterisks can be used as wildcard characters here.

Example:

The command below searches for all properties whose names start with SchemaConfig.: (Note: the "--" is required. It separates arguments to iris_exec from arguments to property_inspector. See: Schema Inference.)

/usr/illumon/latest/bin/iris_exec property_inspector -- -pf iris-console.prop -pn SchemaConfig.*

When the command above is run, the following results are generated (edited for brevity/clarity):

Searching for: SchemaConfig.*, starting with file: "iris-console.prop"

Name: SchemaConfig.resourcePath.irisInternal --- Value: <devroot>/etc/dbinternal;<devroot>/configs/dbinternal --- Line: 408 --- File: iris-defaults.prop

Name: SchemaConfig.internalNamespaces --- Value: DbInternal --- Line: 409 --- File: iris-defaults.prop

Name: SchemaConfig.suffix --- Value: .schema --- Line: 410 --- File: iris-defaults.prop

Name: SchemaConfig.classPrefix --- Value: com.illumon.iris. --- Line: 411 --- File: iris-defaults.prop

Name: SchemaConfig.defaultListenerPackagePrefix --- Value: com.illumon.iris.intraday.gen. --- Line: 412 --- File: iris-defaults.prop

Name: SchemaConfig.resourcePath --- Value: /etc/sysconfig/illumon.d/schema/ --- Line: 40 --- File: IRIS-CONFIG.prop

Name: SchemaConfig.defaultListenerPackagePrefix --- Value: com.illumon.intraday.gen. --- Line: 41 --- File: IRIS-CONFIG.prop

Name: SchemaConfig.internalPath --- Value: <devroot>/java_lib --- Line: 7 --- File: iris-console.prop

Name: SchemaConfig.resourcePath --- Value: <devroot>/java_lib --- Line: 8 --- File: iris-console.prop

Configuration Files

Configuration files are text files whose format varies depending on the requirements of the applications that use them. The main use of conf files in Deephaven is the hostconfig file, which is used in all configurations.

Hostconfig

The hostconfig file is normally located under /etc/sysconfig/illumon.confs, and named illumon.iris.hostconfig. The hostconfig file is sourced (run) by another bash script, which means it supports bash scripting to make if/then and case statement type decisions. This file provides information needed to start and monitor Deephaven processes. It contains details that must be passed to the processes as they are started, such as how much memory should be allocated, what account they should run under, and what properties file they should use.

Whenever a Deephaven service is started by Monit, it uses the Deephaven launch script to start the service, which sources the hostconfig file. For more information on Monit Services, see the Monit Manual.

For example, the following is a section of a hostconfig file that configures the Deephaven Controller process:

iris_controller)
   CONFIGFILE=iris-controller.prop
   EXTRA_ARGS="-j -Xms4G -j -Xmx4G -j -Xmn32m"
   RUN_AS=irisadmin
   WORKSPACE=/db/TempFiles/$RUN_AS/$proc
   ;;

In this case, the process will be directed to use the iris-controller.prop properties file, run under the irisadmin account, and be allocated 4GB of initial, and maximum, heap space, with 32MB of Eden space. In the EXTRA_ARGS line, the -j indicates to the Deephaven scripts that these are parameters to be passed through directly to the JVM.

If needed, additional processes to be managed by Monit can be added to the hostconfig file. If an existing process needs changes to allocated memory or other startup properties, those changes must be made in this file. See the Deephaven Operations Guide for additional documentation on Deephaven's use of Monit.

Other Configuration Files

Most other configuration files in Deephaven are text files that contain lists of values for various purposes:

db_rqc_default_class_push_list.txt
default_static_imports.txt
default_class_imports.txt
default_package_imports.txt
authusers.txt
superusers.txt
dsakeys.txt

1. `db_rqc_default_class_push_list.txt`

This is the default file for listing classes that exist in the classpath of the client system and should be pushed to the class cache of the query worker at startup. An alternate file for this purpose can be specified by setting the property RemoteQueryClient.defaultClassPushList.

In general, classes pushed from this file will be classes that the user is developing and needs to test, or classes that only one or a very few users need. Production classes that are to be usable by most users should be deployed to query servers by building them into JARs and placing these JARs under /etc/sysconfig/illumon.d/java_lib.

2. Default files

These are the default files that contain a list of Java classes that are added to the query library used by Deephaven queries:

default_static_imports.txt

default_class_imports.txt

default_package_imports.txt

These file names can be overridden by the properties:

IrisDB.queryLibrary.defaultPackageImportList=

IrisDB.queryLibrary.defaultClassImportList=

IrisDB.queryLibrary.defaultStaticImportList=

3. `/etc/sysconfig/illumon.d/resources/authusers.txt`

The set of Deephaven users, groups and ACLs are stored in MySQL database. Customers can choose to use LDAP for storing user credentials. In some situations, customers may wish to maintain the set of Deephaven users in a local file by setting the property:

authentication.server.localusers.enabled=true

In this case, the authusers.txt file contains the list of Deephaven users in htpasswd format.

4. `/etc/sysconfig/illumon.d/resources/superusers.txt`

This file lists the local superusers as long as the property is set as follows:

authentication.server.localsuperusers.enabled=true

5. `/etc/sysconfig/illumon.d/resources/dsakeys.txt`

This filename can be overridden by the property:

authentication.server.authorizedkeys.file=...

The dsakeys.txt file lists the DSA keys for Deephaven users as long as the property is set as follows:

authentication.server.authorizedkeys.enabled=true

DSA keys are used in place of passwords for non-interactive users in scripts or batch jobs or delegated API calls. See: Database Authentication to learn more.

YAML Files

YAML is a human-readable data serialization language commonly used for configuration files. In Deephaven, a YAML file (which use the extension .yml) is used to configure the Data Routing Service.

See Data Routing Service Configuration via YAML for detailed information.

XML Files

XML (eXtensible Markup Language) files are also text files, and nominally human-readable, but, unlike configuration files and properties files, XML files include more requirements around how data is formatted and delimited. The main structure of XML files is based around elements, which can be single line or multi-line, and which can encapsulate other elements to form hierarchies. Elements can also contain embedded text data directly as content and/or in the form of named attributes.

The main places where XML is used to store Deephaven configuration settings are:

Tailer configuration files, which tell a tailer what files and paths it should monitor.
Import/Merge script configuration files, which provide one way of automating frequently executed data import and merge operations.
Schema files, each of which is an XML file without a root element.
Workspace files, which persist the layout of a Deephaven client application (tabs, queries, etc).

See the Tailer XML Configuration Parameters and Schemas sections in the Importing Data guide for detailed information.

Environment Variables

Environment variables are used to set some of the global configuration for Deephaven, most commonly for bin utilities, which rely on JAVA_HOME, and also for the import/merge scripts.

In order to use Deephaven tools like iriscat (/usr/illumon/latest/bin/iriscat), JAVA_HOME must first be set to point to an installation directory that contains a Java 8 installation. This path will include a bin subdirectory that contains a Java runtime executable. Typically, this will be set to /usr/java/latest or /usr/java/latest-64. On Linux, this can be checked by running:

export | grep JAVA_HOME

It can be set similarly with:

export JAVA_HOME=path_to_java

Other environment variables, primarily used by the import/merge scripts, are covered in Appendix A of the Deephaven Importing Data guide.

Last Updated: 23 September 2019 12:17 -04:00 UTC Deephaven v.1.20181212 (See other versions)