How to append upload files using Python SDK V2 - - Alibaba Cloud Documentation Center

You can append content to existing appendable objects. This topic describes how to use OSS Python SDK to perform append upload.

Considerations

The example code in this topic uses the region ID cn-hangzhou of China (Hangzhou) and the public endpoint by default. If you want to access OSS from other Alibaba Cloud services in the same region as OSS, use the internal endpoint. For more information about the mapping between regions and endpoints of OSS, see OSS regions and endpoints.
When you call an append upload operation, if the object to which you want to append content does not exist, an appendable object is created.
If the object to which you want to append content exists:
- If the object is an appendable object and the specified position from which the append operation starts equals the current object length, the content is appended to the end of the object.
- If the object is an appendable object but the specified position from which the append operation starts does not equal the current object length, the PositionNotEqualToLength exception is thrown.
- If the object is not an appendable object, such as a Normal object uploaded by using simple upload, the ObjectNotAppendable exception is thrown.

Permissions

By default, an Alibaba Cloud account has full permissions. RAM users or RAM roles under an Alibaba Cloud account do not have any permissions by default. The Alibaba Cloud account or account administrator must grant operation permissions through RAM Policy or Bucket Policy.

API

Action

Definition

AppendObject

oss:PutObject

You can call this operation to upload an object by appending the object to an existing object.

oss:PutObjectTagging

When uploading an object by appending the object to an existing object, if you specify object tags through x-oss-tagging, this permission is required.

Method definition

For append upload scenarios, Python SDK V2 adds the AppendFile method to simulate file read and write behaviors for operating objects in buckets. The following table lists the descriptions of AppendFile and AppendObject methods:

The method name

Description

AppendFile

Has the same capabilities as the AppendObject method

Optimize fault tolerance after the object reupload fails

AppendObject

Perform append upload to upload an object that is up to 5 GiB in size

Support CRC-64 (enabled by default)

Supports progress bars

Premium Edition append upload API: AppendFile

You can call the AppendFile method to upload data by appending the data to an existing object. If the object does not exist, the AppendFile operation creates an appendable object. If the object exists but is not the appendable type, an error is returned.

The following code provides the definition of the AppendFile method.

append_file(bucket: str, key: str, request_payer: str | None = None, create_parameter: AppendObjectRequest | None = None, **kwargs) → AppendOnlyFile

Request parameters

Parameter	Type	Description
bucket	str	The name of the bucket.
key	str	The name of the object.
RequestPayer	str	Specifies that if pay-by-requester is enabled, RequestPayer must be set to requester.
CreateParameter	AppendObjectRequest	Used to set the object metadata during the first upload, including ContentType, Metadata, permissions, storage class, etc. For more information, see AppendObjectRequest

Return values

Type	Description
AppendOnlyFile	The instance of the append file. For more information, see AppendOnlyFile

The following table describes the methods included in the AppendOnlyFile class:

Method	Description
Close()	Closes the file handles to release resources.
write(b)	Writes byte data to the file and returns the number of bytes written.
write_from(b: str \| bytes \| Iterable[bytes] \| IO[str] \| IO[bytes])	Writes arbitrary data to the file and returns the number of bytes written.

For the complete definition of the AppendFile method, see append_file.

Basic append upload API: AppendObject

append_object(request: AppendObjectRequest, **kwargs) → AppendObjectResult

Request parameters

Parameter	Type	Description
request	AppendObjectRequest	Sets the request parameters. For more information, see AppendObjectRequest

Return values

Type	Description
AppendObjectResult	The return value. For more information, see AppendObjectResult

For the complete definition of the AppendObject method, see append_object.

Sample code

(Recommended) Use AppendFile for append upload

import argparse
import alibabacloud_oss_v2 as oss

# Create a command-line argument parser, and describe the script purpose: example shows how to append data to an OSS object
parser = argparse.ArgumentParser(description="append file sample")

# Specify the --region parameter, which specifies the region in which the bucket is located. This command line parameter is required.
parser.add_argument('--region', help='The region in which the bucket is located.', required=True)
# Specify the --bucket parameter, which specifies the name of the bucket. This command line parameter is required.
parser.add_argument('--bucket', help='The name of the bucket.', required=True)
# Specify the --endpoint parameter, which specifies the endpoint that other services can use to access OSS. This command line parameter is optional.
parser.add_argument('--endpoint', help='The domain names that other services can use to access OSS')
# Specify the --key parameter, which specifies the name of the object. This command line parameter is required.
parser.add_argument('--key', help='The name of the object.', required=True)

def main():
    # Parse the command-line parameters to obtain the specified values.
    args = parser.parse_args()

    # Obtain access credentials from environment variables for authentication.
    credentials_provider = oss.credentials.EnvironmentVariableCredentialsProvider()

    # Use the default configurations of the SDK to create a configuration object and specify the credential provider.
    cfg = oss.config.load_default()
    cfg.credentials_provider = credentials_provider
    cfg.region = args.region

    # If a custom endpoint is provided, update the endpoint attribute of the cfg object with the provided endpoint.
    if args.endpoint is not None:
        cfg.endpoint = args.endpoint

    # Use the preceding configuration to initialize the OSSClient instance.
    client = oss.Client(cfg)

    # Define the data to append
    data1 = b'hello'
    data2 = b' world. '

    # First append operation
    with client.append_file(bucket=args.bucket, key=args.key) as f:
        append_f = f
        f.write(data1)
    # Print the file status after the first append
    print(f'closed: {append_f.closed},'
          f' name: {append_f.name}'
    )

    # Second append operation
    with client.append_file(bucket=args.bucket, key=args.key) as f:
        append_f = f
        f.write(data2)
    # Print the file status after the second append
    print(f'closed: {append_f.closed},'
          f' name: {append_f.name}'
    )

    # Get the content of the appended object
    result = client.get_object(oss.GetObjectRequest(
        bucket=args.bucket,
        key=args.key,
    ))
    # Print the result of getting the object
    print(f'status code: {result.status_code},'
          f' request id: {result.request_id},'
          f' content: {result.body.content.decode("utf-8")}'
    )

# Call the main function to start the processing logic when the script is directly run.
if __name__ == "__main__":
    main()  # Script entry point, program flow starts here

Use AppendObject for append upload

import argparse
import alibabacloud_oss_v2 as oss

# Create a command-line argument parser.
parser = argparse.ArgumentParser(description="append object sample")

# Add command line parameters
# --region: Specify the region in which the OSS bucket is located
parser.add_argument('--region', help='The region in which the bucket is located.', required=True)
# --bucket: Specify the name of the bucket to operate on
parser.add_argument('--bucket', help='The name of the bucket.', required=True)
# --endpoint: Optional parameter, specify the domain name for accessing OSS service
parser.add_argument('--endpoint', help='The domain names that other services can use to access OSS')
# --key: Specify the key name of the object (file) in OSS
parser.add_argument('--key', help='The name of the object.', required=True)

def main():
    # Parse the command line parameters.
    args = parser.parse_args()

    # Load the authentication information required to access OSS from the environment variables.
    credentials_provider = oss.credentials.EnvironmentVariableCredentialsProvider()

    # Use the default configuration to create a configuration object.
    cfg = oss.config.load_default()

    # Set the credential provider to the previously created object
    cfg.credentials_provider = credentials_provider

    # Set the region for the OSS client based on user input
    cfg.region = args.region

    # If the user provided a custom endpoint, update the configuration
    if args.endpoint is not None:
        cfg.endpoint = args.endpoint

    # Create an OSS client instance using the above configuration
    client = oss.Client(cfg)

    # Define the data to append
    data1 = b'hello'
    data2 = b' world'

    # First append operation
    result = client.append_object(oss.AppendObjectRequest(
        bucket=args.bucket,  # Specify the bucket name.
        key=args.key,  # Specify the object key name
        position=0,  # The starting position for append, initially 0
        body=data1,  # The data to append
    ))

    # Print the result of the first append
    print(f'status code: {result.status_code},'
          f' request id: {result.request_id},'
          f' version id: {result.version_id},'
          f' hash crc64: {result.hash_crc64},'
          f' next position: {result.next_position},' 
    )

    # Second append operation
    result = client.append_object(oss.AppendObjectRequest(
        bucket=args.bucket,  # Specify the bucket name.
        key=args.key,  # Specify the object key name
        position=result.next_position,  # Start from the next position after the previous append
        body=data2,  # The data to append
    ))

    # Print the result of the second append
    print(f'status code: {result.status_code},'
          f' request id: {result.request_id},'
          f' version id: {result.version_id},'
          f' hash crc64: {result.hash_crc64},'
          f' next position: {result.next_position},'
    )

# Call the main function when the script is directly run.
if __name__ == "__main__":
    main()

Common scenarios

Display progress bar during append upload

import argparse
import alibabacloud_oss_v2 as oss

# Create a command-line argument parser.
parser = argparse.ArgumentParser(description="append object sample")

# Add command line parameters
# --region: Specify the region in which the OSS bucket is located
parser.add_argument('--region', help='The region in which the bucket is located.', required=True)
# --bucket: Specify the name of the bucket to operate on
parser.add_argument('--bucket', help='The name of the bucket.', required=True)
# --endpoint: Optional parameter, specify the domain name for accessing OSS service
parser.add_argument('--endpoint', help='The domain names that other services can use to access OSS')
# --key: Specify the key name of the object (file) in OSS
parser.add_argument('--key', help='The name of the object.', required=True)

def main():
    # Parse the command line parameters to obtain the values entered by the user.
    args = parser.parse_args()

    # From the environment variables, load the authentication information required to access OSS.
    credentials_provider = oss.credentials.EnvironmentVariableCredentialsProvider()

    # Use the default configuration to create a configuration object.
    cfg = oss.config.load_default()

    # Set the credential provider to the previously created object
    cfg.credentials_provider = credentials_provider

    # Set the region for the OSS client based on user input
    cfg.region = args.region

    # If the user provided a custom endpoint, update the configuration
    if args.endpoint is not None:
        cfg.endpoint = args.endpoint

    # Create an OSS client instance using the above configuration
    client = oss.Client(cfg)

    # Define a dictionary variable progress_state to store upload progress status, with initial value 0
    progress_state = {'saved': 0}
    def _progress_fn(n, written, total):
        # Use dictionary to store cumulative written bytes, avoiding global variables
        progress_state['saved'] += n

        # Calculate current upload percentage by dividing written bytes by total bytes and rounding
        rate = int(100 * (float(written) / float(total)))

        # Print current upload progress, \r means return to line start, enabling real-time refresh in command line
        # end='' means no line break, allowing next print to overwrite current line
        print(f'\rUpload progress: {rate}% ', end='')

    # Define the data to append
    data1 = b'hello'
    data2 = b' world'

    # First append operation
    result = client.append_object(oss.AppendObjectRequest(
        bucket=args.bucket,  # Specify the bucket name.
        key=args.key,  # Specify the object key name
        position=0,  # The starting position for append, initially 0
        body=data1,  # The data to append
        progress_fn=_progress_fn,  # Set the progress callback function
    ))

    # Print the result of the first append
    print(f'status code: {result.status_code},'
          f' request id: {result.request_id},'
          f' version id: {result.version_id},'
          f' hash crc64: {result.hash_crc64},'
          f' next position: {result.next_position},'
    )

    # Second append operation
    result = client.append_object(oss.AppendObjectRequest(
        bucket=args.bucket,  # Specify the bucket name.
        key=args.key,  # Specify the object key name
        position=result.next_position,  # Start from the next position after the previous append
        body=data2,  # The data to append
        progress_fn=_progress_fn,  # Set the progress callback function
    ))

    # Print the result of the second append
    print(f'status code: {result.status_code},'
          f' request id: {result.request_id},'
          f' version id: {result.version_id},'
          f' hash crc64: {result.hash_crc64},'
          f' next position: {result.next_position},'
    )

# Call the main function when the script is directly run.
if __name__ == "__main__":
    main()

References

For the complete sample code for append upload, see append_file.py and append_object.py.