Import Java classes and Python modules

Software libraries allow users to share and reuse code, making the software development process more efficient. We'll cover all of the ways that libraries can be used in Deephaven. Many functions are built into Deephaven and can be manually imported in your queries. However, you can also take advantage of libraries from outside sources (e.g., Seaborn, Apache, etc.). We'll discuss how to import and install these libraries.

Simple Import#

Python, Groovy, Java, and other supported languages have built-in mechanisms to import code. These mechanisms can be used to import code from Deephaven, third-party vendors, or you, as long as the code is available to the programming language.

For example, in Python, the Deephaven TableTools module can be imported and used to create a table with 10 rows.

import deephaven
t = deephaven.TableTools.emptyTable(10).update("X=3")

The same results can be obtained more succinctly by just importing the emptyTable function.

from deephaven.TableTools import emptyTable
t = emptyTable(10).update("X=3")

In Groovy, this would look like:

import com.illumon.iris.db.tables.utils.TableTools
t = TableTools.emptyTable(10).update("X=3")

or

import static com.illumon.iris.db.tables.utils.TableTools.emptyTable
t = emptyTable(10).update("X=3")

Groovy can be configured to import commonly-used classes by default. emptyTable is one such default import. As a result, in Groovy, the query can be condensed down to.

t = emptyTable(10).update("X=3")

Because Python can not be configured to import commonly-used classes by default, almost all Python queries require imports.

Using Java in Python#

Because Deephaven’s core database is written in Java, we have made it easy for Java and Python to work together. You can use Java libraries to solve problems in Python.

To illustrate using Java in Python, let’s create a Java String within a Python session, call Java’s split method on the String, and iterate over the resulting Java String array.

from deephaven import jpy
JString = jpy.get_type("java.lang.String")
s1 = JString("ABC/XYZ")
print(s1)
print(type(s1))
s2 = s1.split("/")
print(type(s2))
for x in s2:
print(x)

Occasionally, you may need to create a Java array for use with a library. This is also easy.

from deephaven import jpy
jarray = jpy.array("int", 10)
jarray[2] = 3
for x in jarray:
print(x)

Groovy Source#

In Deephaven’s Groovy implementation, files can be executed within a Groovy script. This can be useful to load functions, variables, and configurations into Groovy scripts.

To source a file:

source("/path/to/script.groovy")

To source a file only once during a session:

sourceOnce("/path/to/script.groovy")

Scripts are sourced using a script path loader - they will be retrieved from the Iris controller. The path will be based on the git parameters defined for the controller. The script must be available to the controller based on the following properties:

  • iris.scripts.repo.irisrepo.root - defines the root directory used to create the search paths
  • iris.scripts.repo.irisrepo.paths - defines the subdirectories which will be searched for the scripts

In a default installation, these properties resolve to the following path:

/db/TempFiles/irisadmin/git/irisrepo/groovy

Extend The Query Language#

Because Deephaven’s database is implemented in Java, it is possible to use Java within query strings.

In Groovy, this can be done by providing a fully qualified path for the class.

t = emptyTable(10).update("X = java.math.BigInteger.ONE", "Y = new java.math.BigInteger(`11`)")

The query strings can be made more concise by importing the class or static methods into the query language.

db.importClass(java.math.BigInteger)
db.importStatic(java.math.BigInteger)
t = emptyTable(10).update("X = ONE", "Y = new BigInteger(`11`)")

In Python, these queries are:

from deephaven.TableTools import emptyTable
t = emptyTable(10).update("X = java.math.BigInteger.ONE", "Y = new java.math.BigInteger(`11`)")

and

from deephaven.TableTools import emptyTable
from deephaven import jpy
BigInteger = jpy.get_type("java.math.BigInteger")
db.importClass(BigInteger.jclass)
db.importStatic(BigInteger.jclass)
t = emptyTable(10).update("X = ONE", "Y = new BigInteger(`11`)")

Built-In Libraries#

Documentation for built-in libraries can be found at:

Installing Libraries#

You can also install other libraries for use in Deephaven. To use a java library, it must be available on the java classpath. The classpath directory must be readable by the server running your query. This can be done in a few ways.

  1. Create a Deephaven plugin containing the jar.
  2. Copy the jar file to /etc/sysconfig/illumon.d/java_lib/.
  3. Copy the jar file to a directory listed in the CUSTOMER_JAR_DIR environment variable in the hostconfig file (/etc/sysconfig/illumon.confs/illumon.iris.hostconfig).

Deploying an open source Python package from PyPi is most commonly done by adding the new Python package to requirements.txt and regenerating the Python virtual environment (/usr/illumon/latest/install/python/auto_provision.sh). If the virtual environments are not crossmounted on all query hosts, the virtual environments must be regenerated on each query host.

You can also make your own custom Python modules available to Deephaven queries. Your code must be on the python path. This can be done in a few ways:

  1. Set the PYTHONPATH environment variable when launching your query.
  2. Append to the python path from within your query.
import sys
sys.path.append("/your/python/module/path/")

Remember that your python path must be mounted on the node where your query is run. Also, the path must be readable by the dbquery Unix user, which is the user queries run as.