Skip to main content

Gcs

Google Cloud Storage Support

Capabilities​

This service can be used to:

  • stat
  • read
  • write
  • create_dir
  • delete
  • copy
  • rename
  • list
  • presign
  • blocking

Configuration​

  • root: Set the work directory for backend
  • bucket: Set the container name for backend
  • endpoint: Customizable endpoint setting
  • credential: Service Account or External Account JSON, in base64
  • credential_path: local path to Service Account or External Account JSON file
  • service_account: name of Service Account
  • predefined_acl: Predefined ACL for GCS
  • default_storage_class: Default storage class for GCS

Refer to public API docs for more information. For authentication related options, read on.

Options to authenticate to GCS​

OpenDAL supports the following authentication options:

  1. Provide a base64-ed JSON key string with credential
  2. Provide a JSON key file at explicit path with credential_path
  3. Provide a JSON key file at implicit path
  4. Fetch access token from VM metadata
    • Only works when running inside Google Cloud.
    • If a non-default Service Account name is required, set with service_account. Otherwise, nothing need to be set.
  5. A custom TokenLoader via GcsBuilder.customed_token_loader()

Notes:

  • When a Service Account key is provided, it will be used to create access tokens (VM metadata will not be used).
  • Explicit Service Account key, in json or path, always take precedence over ADC-defined key paths.
  • Due to limitation in GCS, a private key is required to create Pre-signed URL. Currently, OpenDAL only supports Service Account key.

Example​

Via Builder​

use anyhow::Result;
use opendal::services::Gcs;
use opendal::Operator;

#[tokio::main]
async fn main() -> Result<()> {
    // create backend builder
    let mut builder = Gcs::default();

    // set the storage bucket for OpenDAL
    builder.bucket("test");
    // set the working directory root for GCS
    // all operations will happen within it
    builder.root("/path/to/dir");
    // set the credentials with service account
    builder.credential("service account JSON in base64");
    // set the predefined ACL for GCS
    builder.predefined_acl("publicRead");
    // set the default storage class for GCS
    builder.default_storage_class("STANDARD");

    let op: Operator = Operator::new(builder)?.finish();
    Ok(())
}

Via Config​

use anyhow::Result;
use opendal::Operator;
use opendal::Scheme;
use std::collections::HashMap;

#[tokio::main]
async fn main() -> Result<()> {
    let mut map = HashMap::new();
    map.insert("bucket".to_string(), "test".to_string());
    map.insert("root".to_string(), "/path/to/dir".to_string());
    map.insert("credential".to_string(), "authentication token".to_string());
    map.insert("predefined_acl".to_string(), "publicRead".to_string());
    map.insert("default_storage_class".to_string(), "STANDARD".to_string());

    let op: Operator = Operator::via_map(Scheme::Gcs, map)?;
    Ok(())
}