deephaven.pandas

This module supports the conversion between Deephaven tables and pandas DataFrames.

to_pandas(table, cols=None, dtype_backend='numpy_nullable', conv_null=True)[source]

Produces a pandas DataFrame from a table.

Note that the entire table is going to be cloned into memory, so the total number of entries in the table should be considered before blindly doing this. For large tables, consider using the Deephaven query language to select a subset of the table before using this method.

Parameters:
  • table (Table) – the source table

  • cols (Optional[list[str]]) – the source column names, default is None which means include all columns

  • dtype_backend (Literal[None, "pyarrow", "numpy_nullable"]) – which dtype backend to use. “numpy_nullable” and “pyarrow” convert the table through pyarrow. “numpy_nullable” uses pandas nullable dtypes for the numeric, boolean, and string types that have one. “pyarrow” uses pd.ArrowDtype for numeric, boolean, string, and timestamp types, and requires pandas 2.0.0 or later. Other types use pyarrow’s default conversion. Deephaven nulls become pd.NA in the columns that use these dtypes. Columns that use pyarrow’s default conversion get its missing-value marker instead, such as NaT for timestamp columns with “numpy_nullable”. None reads the column data into NumPy-backed Series; when conv_null is True, boolean, char, and numeric columns then use pandas nullable dtypes so that Deephaven nulls become pd.NA. Default is “numpy_nullable”.

  • conv_null (bool) – when dtype_backend is set to None, whether to check for Deephaven nulls in the data and automatically replace them with pd.NA. default is True.

Return type:

DataFrame

Returns:

a pandas DataFrame

Raises:

DHError –

to_table(df, cols=None, infer_objects=True)[source]

Creates a new table from a pandas DataFrame.

Column names are converted in one of two ways. With pandas 2.0.0 or later, when a selected column has a pd.ArrowDtype, the table is created through pyarrow, which removes characters that aren’t allowed in column names: “Shades.1” becomes “Shades1”. Otherwise, including when the only pyarrow-backed columns are pd.StringDtype(“pyarrow”) columns, or if that conversion fails, each run of characters other than letters, digits, and underscores becomes a single underscore, and such characters at the start or end of a name are removed: “Shades.1” becomes “Shades_1”. This second conversion changes nothing else, so a name such as “123” or the reserved word “in” is still rejected.

Parameters:
  • df (DataFrame) – the pandas DataFrame instance

  • cols (Optional[list[str]]) – the dataframe column names, default is None which means including all columns in the DataFrame

  • infer_objects (bool) – whether to infer the best possible types for columns of the generic ‘object’ type in the DataFrame before creating the table, default is True. When True, pandas convert_dtypes() method is called to perform the conversion. Note that any conversion will make a copy of the data.

Return type:

Table

Returns:

a Deephaven table

Raises:

DHError –