Code Studio

When using the Deephaven Web interface, the Code Studio is used to write console-based queries for analyzing data. The tables, charts, and widgets generated from those queries (as well as other tables, charts, and widgets) can then be arranged to create customized workspaces in the Deephaven interface.

When you open a new Code Studio, the New Console Connection screen provides options to specify the Server you want to use, the Heap Size (memory size) requested to run your queries, and the language you plan on using for the query (Groovy or Python), as shown below:

Server

Available servers are presented in alphabetical order in the drop-down menu. Because every installation is unique, please consult your system administrator for additional information about available servers.

Heap Size

This setting refers to the dynamic memory (RAM) you want Deephaven to use to process the instructions and the data needed to run this specific query. Memory listed in this setting is noted in gigabytes (GB). Note: memory is a finite resource on the system. It is usually best to start small when estimating the amount of memory needed to run the query, and then adjust it accordingly as needed.

Language

Deephaven scripts can be written in Groovy or Python. This value defaults to Groovy. If you prefer to write your scripts in Python, select Python from the drop-down menu.

Additional connection options are available by selecting Show Advanced.

ClosedClick to see options available under Show Advanced

If you click Show Advanced, additional options will appear in the console window, as shown below.

  • Extra JVM Arguments - This field allows users to access different Java utilities that are not included in core Deephaven installation. For example, one may want to run a different profiler or debugging processor. These items can be included in the Deephaven configuration by typing the extra JVM arguments in this field.
  • Extra Env Variables - If needed, this field can be used to pass Extra Environment Variables to the Deephaven configuration.
  • JVM Profile - This drop-down field allows users to select an alternative JVM profile for new persistent queries. Remote processing profiles are used to specify JVM parameters and environment variables for persistent queries and interactive consoles. Several predefined profiles are available, such as CMS GC or G1 GC, and new ones can be defined using controller properties.
  • Enable detailed GC logs - Garbage Collection (GC) is a JVM memory management program that frees unreachable memory by getting rid of objects not being used by a Java application. When the check box is selected, Garbage Collection information will be included in the server-side dispatcher logs.

After adjusting the applicable settings in the dialog box, click Connect.

After establishing a connection to the server, the Code Studio window will look similar to the image below. Two nested panels, Console and Log, will fill the screen. (Note: the Command History section on the right is part of the Console panel. It is visible by default but can be hidden via More Actions. )

The Console panel on the left is for writing and executing queries. After the query executes, any tables, plots and/or widgets generated by the query will appear in the workspace as a new panel.

If desired, additional Code Studios can be added to the interface by using the New button in the Navigation bar at the top. Consider creating separate code studios for each console and server configuration you commonly access. All Code Studios (and Dashboards) can be renamed by right-clicking the tab, as shown below:

Console

The Console window is used for writing and executing queries. The tables, plots, and widgets generated by those queries can then be used to build your workspace.

Queries

Simply speaking, a query is a way to ask the data system for information. For example, you can write a query to filter, group, and aggregate data; to join different tables; to create different views; or to build plots and other widgets. To learn more, see Writing Queries.

There are two types of queries: console-based and persistent.

  • Console-based queries are typed in the console or accessed via Git or other types of repository. Typically, console-based queries are used for one-off questions or data extractions and the development of new Persistent Queries. Tables and charts generated by console-based queries are added to the Code Studio window as new panels.
  • Persistent queries use the same language as console-based queries. However, persistent queries can be saved*, thereby enabling the automation of repetitive and/or timed operations. Persistent queries can also be shared with other individuals or groups. Tables and charts generated by persistent queries are accessible from the Panels menu. To learn more, see Panels.

* Currently, persistent queries can only be saved when using the Swing version of Deephaven. However, the next release of Deephaven will enable users to save persistent queries via the web interface.

Example of a Console-based Query

The following code block is an example of a query using the Deephaven Query Language. When executed in the console, it will open a table of stock market data and then filter that table to a specific date and Sym (AAPL):

newTable=db.t("LearnDeephaven", "StockTrades")
    .where("Date=`2017-08-25`")
    .where("Sym=`AAPL`")

The specifics about writing queries are covered elsewhere (see Writing Queries). However, as you can see below, the language is very intuitive and easy to follow.

Query

Explanation

newTable= db.t("LearnDeephaven", "StockTrades")

This line tells the system to:

- Look in the namespace called "LearnDeephaven"

- Open a table called "StockTrades"

- Store the contents of the table in a variable named "newTable"

    .where("Date=`2017-08-25`")

This line filters the data to include only those rows on a given date.

    .where("Sym=`AAPL`")

This line filters the data to include only those rows that have the value AAPL in the Sym column.

After the query runs, the generated table will appear in a new panel in the Code Studio window.

As shown above, the new table holds information pertaining only to trades of Apple (AAPL) securities on August 25, 2017.

After the query executes, a blue button will appear in the Console to show the name of the table that was just created. Clicking this button will refresh the table, or reopen the table if it had been subsequently closed.

Now that we have the table created, let's add another query to create a simple line plot using data from the previous table.

newPlot = plot("AAPL", newTable, "Timestamp", "Last")
     .xBusinessTime()
     .show()

After the query executes, the plot is added as a new panel in the Code Studio window. As before, the blue button in the Console panel can be used to refresh or reopen the plot.

This is a very simple example. However, Deephaven is capable of much more complex analyses using much larger datasets.

Console Menu

The menu at the top of the Console window provides the following information and options:

  • Worker status - This shows the current state of the worker: Idle (hollow circle), Working (solid circle), or Disconnected (hollow red circle).
  • Query worker name - This is the name of the Deephaven "worker" assigned to this console
  • Server IP Address - This is the server assigned to this process
  • Script type - This tells users which programming language is being used. Options include Groovy (default) and Python.
  • Memory (Heap) - This is the amount of memory assigned to process queries in this console session.
  • Open Notebook - This opens a notebook panel in the Code Studio. See: Notebook.
  • Show Table - This option becomes available if any console-based queries have generated a table. Selecting an item in the drop-down menu will bring it into focus. If a table has been subsequently closed, you can re-open the table by selecting its name from the drop-down list. (Note: This only applies if the table has been generated in the same session.)

  • Show Widget - This option becomes available if any console-based queries have generated a plot or other widget. Selecting an item in the drop-down menu will bring it into focus. If a plot or widget has been subsequently closed, you can re-open it by selecting its name from the drop-down list. (Note: This only applies if the plot or widget has been generated in the same session.

  • More Actions - This opens the More Actions menu, as shown below:

    • Hide/Show History - Selecting Hide History closes the Command History panel. Selecting Show History re-opens the panel.
    • Print Stdout - If selection, "Stdout" (standard output) will be included in the console history. If your script generates a large amount of logging to stdout, you may wish to turn this off.
    • Auto Launch Panels - If selected, any console-based queries will automatically open in new panels in the Code Studio window. If unselected, tables or widgets produced from console-based queries will not automatically open as new panels. However, these can be opened by clicking the blue button under the query or from the Show Table/Show Widget menus.
    • Close Panels on Disconnect - Note: Not yet implemented. If selected, all open panels will be closed when the console disconnects, or when you exit the web browser. If unselected, panels will remain open upon disconnect or a new session.
    • Open New Log Panel - Note: Not yet implemented. Opens a new log panel if you have closed the log panel for this console.
    • Restart Session - Note: Not yet implemented. Reconnects the console to the query server.
    • Disconnect - Disconnects the console from the query server and will return you to the New Console connection screen.
    • Upload .csv - Note: Not yet implemented. Enables the user to upload a CSV to Deephaven.
    • Thread dump - Note: Not yet implemented. Similar to a stack trace, the thread dump shows a snapshot of the current state of threads running in the application, and can be useful when troubleshooting unexpected behavior. This option is most often used by advanced users, as well as support.
    • Send thread dump - Note: Not yet implemented. Provides users with the option of sending the thread dump to Deephaven Support.

Command History

The Deephaven Command History can be used to review, retrieve and rerun queries that were used in previous queries. Command History is shown by default but may be hidden or re-opened from the More Actions menu in the Console.

When visible, Command History always appears to the right of the console window, as shown below.

The most recently used commands are shown at the bottom of the list. When the panel is full of previous commands, scrolling enables users to find older commands.

The history will show as much of the line as fits the window width. Hovering over that line will open a flyout menu to the left with the full query and its runtime.

You can highlight one or more commands in the History panel by clicking or shift clicking on the commands. If you double-click on a command in the panel or hit enter, that command will be copied to the console.

At the top of the Command History are three options:

  • The Search field can be used to find a specific keyword in a previous command.
  • Open selected commands in the console copies the selected commands into the console.
  • Open selected commands in the notebook copies the selected commands into the notebook panel, or if none are present, creates a new notebook panel with the commands.

Command History Context Menu

If you right-click on a command in the Command History panel, the following options are presented:

  • Copy - Copies the selected command(s) to the clipboard.
  • Run in Console - Runs the selected command(s) in the console.
  • Send to Console - Copies the selected command(s) into the console, where they can be edited before the user runs the query.
  • Send to Notebook - Copies the selected command(s) to the most recently opened notebook panel in the console.
  • New Notebook… - Copies the selected command(s) into a new notebook panel.

Log

The Log panel tab is located beside the Console panel and presents a record (log) of what your query programming is doing, including data input/output, and processing. This feature is primarily designed for more advanced programmers, but the log can help all users find and resolve errors that may occur.

There are eight different logging levels in Deephaven:

  1. FATAL - Details about severe events that will likely cause the query to abort.
  2. STDERR - Presents details about events occurring in the stream where a program writes its own error data.
  3. ERROR - Presents details about more harmful errors that are occurring even though the query is still running.
  4. WARN - Presents details about potentially harmful events that are occurring.
  5. STDOUT - Presents details about events occurring in the stream where a program writes its output data.
  6. INFO - Presents general information showing the progress of the query
  7. DEBUG - Presents more specific information that should be useful for debugging
  8. TRACE - Presents very fine-grained details about events as they occur.

Log Search

The Log search allows you to search the logs for specific keywords/phrases. Simply type the keyword into the Find field and then press Enter. Search will find partial matches and is case-insensitive by default. However, you can choose alternative search options in the search field, including Match Case, Match Whole Word, and Use Regular Expression. If the search term is present in the log, the keyword will be highlighted (Note: this can be turned off and on with the Find in selection button in the search toolbar). If there are multiple instances of the keyword, you can either scroll through the log window or click on the left and right arrows in the search toolbar to jump directly to the next instance of the search term.

Log Settings

Clicking the "gear" icon at the right of the Log panel will open the Settings menu:

The Display Log Levels drop-down menu allows you to choose the level of details presented in the log window when a query is running.

By default, only five levels are shown, including FATAL, STDERR, ERROR, WARN, and STDOUT. Each log level may be turned on or off individually. Selecting Toggle All will show every detail of logging presented in Deephaven. Select Reset to restore the default levels.

Log Context Menu

Right-clicking within the log panel opens a context menu with the following options:

  • Copy - Copies either the line from where the right-click was generated, or selected lines, to the clipboard.
  • Command Palette - Opens the Command Palette, which provides an exhaustive list of commands for editing, including keyboard shortcuts for the most common operations.

Notebooks

The Notebook is a Deephaven feature in which users can develop query components, stage draft queries, or simply store notes. Think of the Notebook as a "sandbox" in which queries can be developed and refined. To learn more, see Deephaven Notebooks.

Notebooks can be added to your workspace by clicking the Open Notebook button in the menu at the top right of the Console window:

A blank notebook will open below the Console window, as shown below:

The Notebook is now ready for use.

Notebook Toolbar

Content entered in the Notebook panel can be acted upon by using the buttons in the Notebook Toolbar, which runs down the left side of the Notebook panel.

  • Run - Automatically copies all of the content in the Notebook window, and then executes the content in the Console
  • Run Selected - Copies the content you have highlighted in the Notebook window and then executes those selected lines in the Console. This option runs the entire selected line (or lines), even if you have highlighted only a portion of the line.
  • Save - Saves all of the content of the Notebook to a file on the user's system. When Save is clicked for the first time, the Save dialog window prompts the user to enter a filename and select the file location. Depending upon the content in the Notebook, the file is saved as .txt, Groovy or Python filetype. Clicking Save again overwrites the existing file with the current Notebook content. After the file is saved, the name shown on the Notebook tab changes to match the filename used during the Save process.
  • Open File Explorer - Note: Not yet implemented. The File Explorer provides a place to save and re-open your files on the server. In the future, users will be able to import local files that can be saved on the server.
  • Git - Note: Not yet implemented.
  • PQ Explorer - Note: Not yet implemented.
  • Search - Note: Not yet implemented.

Notebook Context Menu

Right-clicking within the body of the Notebook panel opens a context menu with following options:

  • Run - Automatically copies all of the content in the Notebook window, and then executes the content in the Console.
  • Run Selected - Copies the content you have highlighted in the Notebook window and then executes those selected lines in the Console. This option runs the entire selected line (or lines), even if you have highlighted only a portion of the line.
  • Change All Occurrences - Highlights all other occurrences of the selected text within the Notebook window, and when the selected text is edited, all occurences change simultaneously.
  • Cut - Deletes the content you have highlighted in the Notebook window.
  • Copy - Copies the content you have highlighted in the Notebook window and saves it to the clipboard.
  • Command Palette - Opens the Command Palette, which provides an exhaustive list of commands for editing, including keyboard shortcuts for the most common operations.

Autocomplete

As you write queries in the Deephaven web console, the autocomplete feature speeds up your query writing by suggesting possible values. As you type in the Console window, a drop-down list will appear with values relevant to your query, such as available namespaces, table names, and table operations.

For example, the animation below shows Deephaven's autocomplete feature generating suggestions (in drop-down menus) as portions of the query are typed.

To select one of the suggested values, scroll to the desired value, and click or press Return/Enter on the highlighted item.

Last Updated: 02 March 2020 14:02 -05:00 UTC    Deephaven v.1.20200121  (See other versions)

Deephaven Documentation     Copyright 2016-2020  Deephaven Data Labs, LLC     All Rights Reserved