Package io.deephaven.iceberg.util

Class IcebergToolsS3

java.lang.Object
io.deephaven.iceberg.util.IcebergToolsS3
public final class IcebergToolsS3 extends Object
Tools for accessing tables in the Iceberg table format from S3.

  • Constructor Details

    • IcebergToolsS3

      public IcebergToolsS3()

  • Method Details

    • createS3Rest

      public static IcebergCatalogAdapter createS3Rest(@Nullable @Nullable String name, @NotNull @NotNull String catalogURI, @NotNull @NotNull String warehouseLocation, @Nullable @Nullable String region, @Nullable @Nullable String accessKeyId, @Nullable @Nullable String secretAccessKey, @Nullable @Nullable String endpointOverride)
      Create an Iceberg catalog adapter for a REST catalog backed by S3 storage. If null is provided for a value, the system defaults will be used.
      Parameters:
      name - the name of the catalog; if omitted, the catalog URI will be used to generate a name
      catalogURI - the URI of the Iceberg REST catalog
      warehouseLocation - the location of the S3 datafiles backing the catalog
      region - the AWS region; if omitted, system defaults will be used
      accessKeyId - the AWS access key ID; if omitted, system defaults will be used
      secretAccessKey - the AWS secret access key; if omitted, system defaults will be used
      endpointOverride - the S3 endpoint override; this is useful for testing with a S3-compatible local service such as MinIO or LocalStack
      Returns:
      the Iceberg catalog adapter

    • createGlue

      public static IcebergCatalogAdapter createGlue(@Nullable @Nullable String name, @NotNull @NotNull String catalogURI, @NotNull @NotNull String warehouseLocation)
      Create an Iceberg catalog adapter for an AWS Glue catalog. System defaults will be used to populate the region and credentials. These can be configured by following AWS Authentication and access credentials guide.
      Parameters:
      name - the name of the catalog; if omitted, the catalog URI will be used to generate a name
      catalogURI - the URI of the AWS Glue catalog
      warehouseLocation - the location of the S3 datafiles backing the catalog
      Returns:
      the Iceberg catalog adapter

    • createAdapter

      public static IcebergCatalogAdapter createAdapter(@Nullable @Nullable String name, @NotNull @NotNull Map<String,String> properties, @NotNull @NotNull Map<String,String> hadoopConfig, @NotNull @NotNull S3Instructions instructions)
      Create an Iceberg catalog adapter.

      This is the preferred way to configure an Iceberg catalog adapter when the caller is responsible for providing AWS / S3 connectivity details; specifically, this allows for the parity of construction logic between Iceberg-managed and Deephaven-managed AWS clients. For advanced use-cases, users are encouraged to use profiles which allows a rich degree of configurability. The instructions will automatically be used as special instructions if IcebergReadInstructions.dataInstructions() is not explicitly set. The caller is still responsible for providing any other properties necessary to configure their Catalog implementation.

      In cases where the caller prefers to use Iceberg's AWS properties (found amongst AwsProperties, S3FileIOProperties, and HttpClientProperties), they should use IcebergTools directly. In this case, parity will be limited to what S3InstructionsProviderPlugin is able to infer; in advanced cases, it's possible that there will be a difference in construction logic between the Iceberg-managed and Deephaven-managed AWS clients which manifests itself as being able to browse Catalog metadata, but not retrieve Table data.

      Note: this method does not explicitly set, nor impose, that S3FileIO be used. It's possible that a Catalog implementations depends on an AWS client for purposes unrelated to storing the warehouse data via S3.

      Parameters:
      name - the name of the catalog; if omitted, the catalog URI will be used to generate a name
      properties - a map containing the Iceberg catalog properties to use
      hadoopConfig - a map containing Hadoop configuration properties to use
      instructions - the s3 instructions
      Returns:
      the Iceberg catalog adapter