storage.storage

Module Contents

Classes

UploadType	Create a collection of name/value pairs.
StreamResponse	This class provides an abstraction between the slightly different
Storage

Functions

init_api_root(api_root)
choose_boundary()	Stolen from urllib3.filepost.choose_boundary() as of v1.26.2.
encode_multipart_formdata(fields, boundary)	Stolen from urllib3.filepost.encode_multipart_formdata() as of v1.26.2.

Attributes

MAX_CONTENT_LENGTH_SIMPLE_UPLOAD
SCOPES
log

storage.storage.MAX_CONTENT_LENGTH_SIMPLE_UPLOAD

storage.storage.SCOPES = ['https://www.googleapis.com/auth/devstorage.read_write']

storage.storage.log

storage.storage.init_api_root(api_root)

Parameters:

api_root (Optional[str]) –

Return type:

Tuple[bool, str]

storage.storage.choose_boundary()

Stolen from urllib3.filepost.choose_boundary() as of v1.26.2.

Return type:

str

storage.storage.encode_multipart_formdata(fields, boundary)

Stolen from urllib3.filepost.encode_multipart_formdata() as of v1.26.2.

Very heavily modified to be compatible with our gcloud-rest converter and to avoid unnecessary urllib3 dependencies (since that’s only included with requests, not aiohttp).

Parameters:

• fields (List[Tuple[Dict[str, str], bytes]]) –

• boundary (str) –

Return type:

Tuple[bytes, str]

class storage.storage.UploadType(*args, **kwds)

Bases: enum.Enum

Create a collection of name/value pairs.

Example enumeration:

>>> class Color(Enum):
...     RED = 1
...     BLUE = 2
...     GREEN = 3

Access them by:

• attribute access:

  >>> Color.RED
  <Color.RED: 1>
  

• value lookup:

  >>> Color(1)
  <Color.RED: 1>
  

• name lookup:

  >>> Color['RED']
  <Color.RED: 1>
  

Enumerations can be iterated over, and know how many members they have:

>>> len(Color)
3

>>> list(Color)
[<Color.RED: 1>, <Color.BLUE: 2>, <Color.GREEN: 3>]

Methods can be added to enumerations, and members can have their own attributes – see the documentation for details.

SIMPLE = 1

RESUMABLE = 2

MULTIPART = 3

class storage.storage.StreamResponse(response)

This class provides an abstraction between the slightly different recommended streaming implementations between requests and aiohttp.

Parameters:

response (Any) –

property content_length: int

Return type:

int

async read(size=-1)

Parameters:

size (int) –

Return type:

bytes

async __aenter__()

Return type:

Any

async __aexit__(*exc_info)

Parameters:

exc_info (Any) –

Return type:

None

class storage.storage.Storage(*, service_file=None, token=None, session=None, api_root=None)

Parameters:

• service_file (Optional[Union[str, IO[AnyStr]]]) –

• token (Optional[gcloud.aio.auth.Token]) –

• session (Optional[requests.Session]) –

• api_root (Optional[str]) –

_api_root: str

_api_is_dev: bool

_api_root_read: str

_api_root_write: str

async _headers()

Return type:

Dict[str, str]

async list_buckets(project, *, params=None, headers=None, session=None, timeout=DEFAULT_TIMEOUT)

Parameters:

• project (str) –

• params (Optional[Dict[str, str]]) –

• headers (Optional[Dict[str, Any]]) –

• session (Optional[requests.Session]) –

• timeout (int) –

Return type:

List[storage.bucket.Bucket]

get_bucket(bucket_name)

Parameters:

bucket_name (str) –

Return type:

storage.bucket.Bucket

async copy(bucket, object_name, destination_bucket, *, new_name=None, metadata=None, params=None, headers=None, timeout=DEFAULT_TIMEOUT, session=None)

When files are too large, multiple calls to rewriteTo are made. We refer to the same copy job by using the rewriteToken from the previous return payload in subsequent rewriteTo calls.

Using the rewriteTo GCS API is preferred in part because it is able to make multiple calls to fully copy an object whereas the copyTo GCS API only calls rewriteTo once under the hood, and thus may fail if files are large.

In the rare case you need to resume a copy operation, include the rewriteToken in the params dictionary. Once you begin a multi-part copy operation, you then have 1 week to complete the copy job.

See https://cloud.google.com/storage/docs/json_api/v1/objects/rewrite

Parameters:

• bucket (str) –

• object_name (str) –

• destination_bucket (str) –

• new_name (Optional[str]) –

• metadata (Optional[Dict[str, Any]]) –

• params (Optional[Dict[str, str]]) –

• headers (Optional[Dict[str, str]]) –

• timeout (int) –

• session (Optional[requests.Session]) –

Return type:

Dict[str, Any]

async delete(bucket, object_name, *, timeout=DEFAULT_TIMEOUT, params=None, headers=None, session=None)

Parameters:

• bucket (str) –

• object_name (str) –

• timeout (int) –

• params (Optional[Dict[str, str]]) –

• headers (Optional[Dict[str, str]]) –

• session (Optional[requests.Session]) –

Return type:

str

async download(bucket, object_name, *, headers=None, timeout=DEFAULT_TIMEOUT, session=None)

Parameters:

• bucket (str) –

• object_name (str) –

• headers (Optional[Dict[str, Any]]) –

• timeout (int) –

• session (Optional[requests.Session]) –

Return type:

bytes

async download_to_filename(bucket, object_name, filename, **kwargs)

Parameters:

• bucket (str) –

• object_name (str) –

• filename (str) –

• kwargs (Any) –

Return type:

None

async download_metadata(bucket, object_name, *, headers=None, session=None, timeout=DEFAULT_TIMEOUT)

Parameters:

• bucket (str) –

• object_name (str) –

• headers (Optional[Dict[str, Any]]) –

• session (Optional[requests.Session]) –

• timeout (int) –

Return type:

Dict[str, Any]

async download_stream(bucket, object_name, *, headers=None, timeout=DEFAULT_TIMEOUT, session=None)

Download a GCS object in a buffered stream.

Parameters:

• bucket (str) – The bucket from which to download.

• object_name (str) – The object within the bucket to download.

• headers (Optional[Dict[str, Any]]) – Custom header values for the request, such as range.

• timeout (int) – Timeout, in seconds, for the request. Note that with this function, this is the time to the beginning of the response data (TTFB).

• session (Optional[requests.Session]) – A specific session to (re)use.

Returns:

A object encapsulating the stream, similar to io.BufferedIOBase, but it only supports the read() function.

Return type:

StreamResponse

async list_objects(bucket, *, params=None, headers=None, session=None, timeout=DEFAULT_TIMEOUT)

Parameters:

• bucket (str) –

• params (Optional[Dict[str, str]]) –

• headers (Optional[Dict[str, Any]]) –

• session (Optional[requests.Session]) –

• timeout (int) –

Return type:

Dict[str, Any]

async upload(bucket, object_name, file_data, *, content_type=None, parameters=None, headers=None, metadata=None, session=None, force_resumable_upload=None, zipped=False, timeout=30)

Parameters:

• bucket (str) –

• object_name (str) –

• file_data (Any) –

• content_type (Optional[str]) –

• parameters (Optional[Dict[str, str]]) –

• headers (Optional[Dict[str, str]]) –

• metadata (Optional[Dict[str, Any]]) –

• session (Optional[requests.Session]) –

• force_resumable_upload (Optional[bool]) –

• zipped (bool) –

• timeout (int) –

Return type:

Dict[str, Any]

async upload_from_filename(bucket, object_name, filename, **kwargs)

Parameters:

• bucket (str) –

• object_name (str) –

• filename (str) –

• kwargs (Any) –

Return type:

Dict[str, Any]

static _get_stream_len(stream)

Parameters:

stream (IO[AnyStr]) –

Return type:

int

static _preprocess_data(data)

Parameters:

data (Any) –

Return type:

IO[Any]

static _compress_file_in_chunks(input_stream, chunk_size=8192)

Reads the contents of input_stream and writes it gzip-compressed to output_stream in chunks. The chunk size is 8Kb by default, which is a standard filesystem block size.

Parameters:

• input_stream (IO[AnyStr]) –

• chunk_size (int) –

Return type:

IO[bytes]

static _decide_upload_type(force_resumable_upload, content_length)

Parameters:

• force_resumable_upload (Optional[bool]) –

• content_length (int) –

Return type:

UploadType

static _split_content_type(content_type)

Parameters:

content_type (str) –

Return type:

Tuple[str, Optional[str]]

static _format_metadata_key(key)

Formats the fixed-key metadata keys as wanted by the multipart API.

Ex: Content-Disposition –> contentDisposition

Parameters:

key (str) –

Return type:

str

async _download(bucket, object_name, *, params=None, headers=None, timeout=DEFAULT_TIMEOUT, session=None)

Parameters:

• bucket (str) –

• object_name (str) –

• params (Optional[Dict[str, str]]) –

• headers (Optional[Dict[str, str]]) –

• timeout (int) –

• session (Optional[requests.Session]) –

Return type:

bytes

async _download_stream(bucket, object_name, *, params=None, headers=None, timeout=DEFAULT_TIMEOUT, session=None)

Parameters:

• bucket (str) –

• object_name (str) –

• params (Optional[Dict[str, str]]) –

• headers (Optional[Dict[str, str]]) –

• timeout (int) –

• session (Optional[requests.Session]) –

Return type:

StreamResponse

async _upload_simple(url, object_name, stream, params, headers, *, session=None, timeout=30)

Parameters:

• url (str) –

• object_name (str) –

• stream (IO[AnyStr]) –

• params (Dict[str, str]) –

• headers (Dict[str, str]) –

• session (Optional[requests.Session]) –

• timeout (int) –

Return type:

Dict[str, Any]

async _upload_multipart(url, object_name, stream, params, headers, metadata, *, session=None, timeout=30)

Parameters:

• url (str) –

• object_name (str) –

• stream (IO[AnyStr]) –

• params (Dict[str, str]) –

• headers (Dict[str, str]) –

• metadata (Dict[str, Any]) –

• session (Optional[requests.Session]) –

• timeout (int) –

Return type:

Dict[str, Any]

async _upload_resumable(url, object_name, stream, params, headers, *, metadata=None, session=None, timeout=30)

Parameters:

• url (str) –

• object_name (str) –

• stream (IO[AnyStr]) –

• params (Dict[str, str]) –

• headers (Dict[str, str]) –

• metadata (Optional[Dict[str, Any]]) –

• session (Optional[requests.Session]) –

• timeout (int) –

Return type:

Dict[str, Any]

async _initiate_upload(url, object_name, params, headers, *, metadata=None, timeout=DEFAULT_TIMEOUT, session=None)

Parameters:

• url (str) –

• object_name (str) –

• params (Dict[str, str]) –

• headers (Dict[str, str]) –

• metadata (Optional[Dict[str, Any]]) –

• timeout (int) –

• session (Optional[requests.Session]) –

Return type:

str

async _do_upload(session_uri, stream, headers, *, retries=5, session=None, timeout=30)

Parameters:

• session_uri (str) –

• stream (IO[AnyStr]) –

• headers (Dict[str, str]) –

• retries (int) –

• session (Optional[requests.Session]) –

• timeout (int) –

Return type:

Dict[str, Any]

async patch_metadata(bucket, object_name, metadata, *, params=None, headers=None, session=None, timeout=DEFAULT_TIMEOUT)

Parameters:

• bucket (str) –

• object_name (str) –

• metadata (Dict[str, Any]) –

• params (Optional[Dict[str, str]]) –

• headers (Optional[Dict[str, str]]) –

• session (Optional[requests.Session]) –

• timeout (int) –

Return type:

Dict[str, Any]

async get_bucket_metadata(bucket, *, params=None, headers=None, session=None, timeout=DEFAULT_TIMEOUT)

Parameters:

• bucket (str) –

• params (Optional[Dict[str, str]]) –

• headers (Optional[Dict[str, str]]) –

• session (Optional[requests.Session]) –

• timeout (int) –

Return type:

Dict[str, Any]

async close()

Return type:

None

async __aenter__()

Return type:

Storage

async __aexit__(*args)

Parameters:

args (Any) –

Return type:

None