Support GCS vended credentials in Iceberg REST catalog

[kaveti]

Description

The Iceberg REST catalog spec allows the catalog server to return file IO properties in the load-table response. These can include cloud-specific credentials that override the connector's static settings on a per-table basis ("vended credentials").

Trino already supports vended S3 credentials (access key, secret key, session token), but GCS properties from the REST catalog response are currently ignored. This means users connecting to a REST catalog that vends GCS credentials (e.g., Tabular, Polaris, or custom implementations) cannot use them for GCS-backed tables.

For example, a REST catalog server managing tables across multiple GCP
projects might vend different gcs.project-id and gcs.oauth2.token
values per table. Without this change, Trino would always use the
statically configured credentials, which may not have access to all
tables.

This change adds support for the following GCS properties from vended credentials:

• gcs.oauth2.token / gcs.oauth2.token-expires-at — short-lived OAuth2 access token and its expiration
• gcs.project-id — GCP project ID (can differ per table if the catalog manages tables across projects)

These properties are extracted from fileIoProperties in IcebergRestCatalogFileSystemFactory and passed through ConnectorIdentity extra credentials to the GCS filesystem layer, where GcsStorageFactory creates an authenticated Storage client using the vended token.

These properties are extracted from fileIoProperties in
[IcebergRestCatalogFileSystemFactory] and passed through
ConnectorIdentity extra credentials to the GCS filesystem layer:

Fixes #24518

Release notes

## Iceberg
* Add support for vended credentials on GCS. ({issue}`24518 `)

[findinpath]

Thank you for your contribution - please focus on adding integration tests for the functionality.

[kaveti]

thank you reviewing the PR. @findinpath i have addressed all your review comments. could you please take a look again.

[kekwan]

Much needed PR for GCS. Thanks @kaveti. PR LGTM, looks in-line with S3 implementation.

[kaveti]

@findinpath could you please review again. i have addressed your review comments.

[findinpath]

@kaveti reviewing again today.

[findinpath]

Relevant information from @kaveti for the maintainer:

we have an internal irc service where the catalogs are configured to point to diff gcp projects. so the usecase is from trino we have one catalog and based on the table load IRC will give necessary tokens and project id. otherwise we have keep the static project on trino catalog level.

[rambleraptor]

I did some local testing on this PR and it appears to be working properly

[ebyhr]

/test-with-secrets sha=b9c270df30c5335ec8115d5d855ec26d4310eb41

[github-actions]

The CI workflow run with tests that require additional secrets finished as failure: https://github.com/trinodb/trino/actions/runs/23368838842

[cla-bot]

The cla-bot has been summoned, and re-checked this pull request!

[findinpath]

.put("gcs.auth-type", "APPLICATION_DEFAULT")

This setting has been forgotton.

https://github.com/trinodb/trino/actions/runs/23368838842

TestIcebergGcsVendingRestCatalogConnectorSmokeTest.: plugin/trino-iceberg/src/test/java/io/trino/plugin/iceberg/catalog/rest/TestIcebergGcsVendingRestCatalogConnectorSmokeTest.java#L118Configuration errors: 1) Error: Invalid configuration property with prefix '': Either gcs.json-key or gcs.json-key-file-path must be set (for class GcsServiceAccountAuthConfig.authMethodValid) 1 error ====================== Full classname legend: ====================== GcsServiceAccountAuthConfig: "io.trino.filesystem.gcs.GcsServiceAccountAuthConfig" ======================== End of classname legend: ========================
--
 

<br class="Apple-interchange-newline">[TestIcebergGcsVendingRestCatalogConnectorSmokeTest.: plugin/trino-iceberg/src/test/java/io/trino/plugin/iceberg/catalog/rest/TestIcebergGcsVendingRestCatalogConnectorSmokeTest.java#L118](https://github.com/trinodb/trino/commit/32b874a28de86f98f4d7fbe47a20326432ff2e26#annotation_48518247998)
Configuration errors: 1) Error: Invalid configuration property with prefix '': Either gcs.json-key or gcs.json-key-file-path must be set (for class GcsServiceAccountAuthConfig.authMethodValid) 1 error ====================== Full classname legend: ====================== GcsServiceAccountAuthConfig: "io.trino.filesystem.gcs.GcsServiceAccountAuthConfig" ======================== End of classname legend: ========================

[kaveti]

done

[kaveti]

/test-with-secrets sha=b9c270df30c5335ec8115d5d855ec26d4310eb41

[kaveti]

/test-with-secrets sha=b9c270df30c5335ec8115d5d855ec26d4310eb41

[ebyhr]

@kaveti Note that only maintainers can run /test-with-secrets command.

[ebyhr]

/test-with-secrets sha=60eeffde9984b934ce01892f19253498d2fae984

[github-actions]

The CI workflow run with tests that require additional secrets finished as failure: https://github.com/trinodb/trino/actions/runs/23415433626

[findinpath]

Error:  io.trino.plugin.iceberg.catalog.rest.TestIcebergGcsVendingRestCatalogConnectorSmokeTest.testDropTableWithMissingDataFile -- Time elapsed: 3.294 s <<< ERROR!
io.trino.filesystem.TrinoFileSystemException: GCS service error listing files: gs://trino-ci-test/gcs-vending-rest-test-j8tipshu69/tpch/test_drop_table_with_missing_data_file_t2q9q86zbq-c16f14599c9848f4983388f5b89b7194/data/
	at io.trino.filesystem.gcs.GcsUtils.handleGcsException(GcsUtils.java:46)
	at io.trino.filesystem.gcs.GcsFileSystem.listFiles(GcsFileSystem.java:222)
	at io.trino.filesystem.switching.SwitchingFileSystem.listFiles(SwitchingFileSystem.java:116)
	at io.trino.filesystem.tracing.TracingFileSystem.lambda$listFiles$0(TracingFileSystem.java:119)
	at io.trino.filesystem.tracing.Tracing.withTracing(Tracing.java:51)
	at io.trino.filesystem.tracing.TracingFileSystem.listFiles(TracingFileSystem.java:119)
	at io.trino.filesystem.tracking.TrackingFileSystem.listFiles(TrackingFileSystem.java:95)
	at io.trino.filesystem.cache.CacheFileSystem.listFiles(CacheFileSystem.java:106)
	at io.trino.plugin.iceberg.BaseIcebergConnectorSmokeTest.testDropTableWithMissingDataFile(BaseIcebergConnectorSmokeTest.java:660)

https://github.com/trinodb/trino/actions/runs/23415433626/job/68110003934

We're still missing

    @Override
    @BeforeAll
    public void initFileSystem()
    {
        byte[] jsonKeyBytes = Base64.getDecoder().decode(gcpCredentialKey);
        GcsFileSystemConfig config = new GcsFileSystemConfig();
        GcsServiceAccountAuthConfig authConfig = new GcsServiceAccountAuthConfig().setJsonKey(new String(jsonKeyBytes, UTF_8));
        GcsStorageFactory storageFactory;
        try {
            storageFactory = new GcsStorageFactory(config, new GcsServiceAccountAuth(authConfig));
        }
        catch (IOException e) {
            throw new RuntimeException(e);
        }
        this.fileSystem = new GcsFileSystemFactory(config, storageFactory).create(SESSION);
    }

I'm still testing locally whether it works.

[findinpath]

I've found and proposed a fix for the issue here apache/iceberg#15734

@ebyhr in the context of dealing with the above mentioned apache/iceberg issue - could we proceed with temporarily overriding the failing methods with a TODO to re-enable them once that the fix will be avaiable?

[ebyhr]

/test-with-secrets sha=327412fbd4c966f70f77a412d252bc6f14d28ef9

[github-actions]

The CI workflow run with tests that require additional secrets has been started: https://github.com/trinodb/trino/actions/runs/23476214902