deephaven_enterprise.iceberg

This module is used to define Iceberg endpoints and to discover and deploy Deephaven schemas for Iceberg tables.

class DiscoveryResult(j_result)

Bases: object

Contains the result of an Iceberg discovery (discover())

deploy_embedded()

Deploy the discovered Iceberg schema to Deephaven, embedding the endpoint within the schema.

Return type:

None

deploy_named()

Deploy the discovered Iceberg schema to Deephaven linking the endpoint by its name.

Return type:

None

class IcebergEndpoint(j_endpoint)

Bases: object

An Iceberg Endpoint defines both the location of the Iceberg Catalog and data warehouse. It also provides information required to create the catalog and retrieve data from the store as a Deephaven table. Optionally users may provide a set of properties, secrets, and hadoop settings to be passed down to the Iceberg APIs.

Secrets passed to IcebergEndpoint instances are simply named references, never actual secret values. For example, you may provide S3 credentials as a secret defined as

IcebergEndpoint(“rest”, “http://catalog/”, “s3://warehouse”, data_instructions, secrets = [“s3.access.key” : “s3.dev_access_key” ])

Deephaven will locate the secret within the set of SecretsProviders to discover the actual value.

Create a new IcebergEndpoint.

deploy(overwrite_existing=False)

Deploy this endpoint to Deephaven. This will fail unless the endpoint was created with a name.

Return type:

None

discover(table_id, endpoint, snapshot_id=None, namespace=None, table_name=None, reference_definition=None)

Discover an Iceberg table.

Parameters:

• table_id (str) – the Iceberg table identifier

• endpoint (IcebergEndpoint) – the endpoint to use for discovery

• snapshot_id (Optional[str]) – (optional) the Iceberg snapshot ID to use for discovery

• namespace (Optional[str]) – (optional) A user specified namespace. If not set, the namespace is derived from the table_id

• table_name (Optional[str]) – (optional) A user specified table name. If not set, the table name is derived from the table_id

• reference_definition (Optional[TableDefinition]) – (optional) A user specified reference TableDefinition. The discovery process guarantees that the result is compatible with this definition.

Return type:

DiscoveryResult

Returns:

a DiscoveryResult that can be used to deploy Deephaven schemas.

get_named_endpoint(endpoint_name)

Get an IcebergEndpoint from Deephaven by name.

Parameters:

endpoint_name (str) – the name of the endpoint

Return type:

IcebergEndpoint

Returns:

the IcebergEndpoint

make_endpoint(catalog_type, catalog_uri, warehouse_uri, data_instructions, endpoint_name=None, properties=None, secrets=None, hadoop_opt=None)

Create a new IcebergEndpoint. If the endpoint_name is set, the resulting endpoint can be deployed to Deephaven for reuse with the deploy() method.

Any provided properties and secrets will be merged into a single collection of properties to be passed on to the Iceberg APIs

Parameters:

• catalog_type (str) – the type of catalog. Possible values include “rest”, “glue”, “hive”, “nessie”, “hadoop”, and “jdbc”

• catalog_uri (str) – the URI of the catalog

• warehouse_uri (str) – the URI of the data

• data_instructions (Any) – the Data Instructions for Deephaven to use when fetching table data.

• endpoint_name (Optional[str]) – (optional) a name for this endpoint. This must be set in order to deploy this endpoint

• properties (Optional[Dict[str, str]]) – (optional) a map of properties to be passed to the Iceberg API

• secrets (Optional[Dict[str, str]]) – a map of secrets to be resolved and passed to the Iceberg API

• hadoop_opt (Optional[Dict[str, str]]) – a map of hadoop specific options to be passed to the Iceberg API

Return type:

IcebergEndpoint