Object Storage Service (OSS) provides the multipart upload feature that allows you to split a large object into multiple parts to upload. After these parts are uploaded, you can call the CompleteMultipartUpload operation to combine the parts into a complete object.
Notes
The sample code in this topic uses the region ID
cn-hangzhouof the China (Hangzhou) region. By default, the public endpoint is used to access resources in a bucket. If you want to access resources in the bucket by using other Alibaba Cloud services in the same region in which the bucket is located, use the internal endpoint. For more information about OSS regions and endpoints, see Regions and endpoints.In this topic, access credentials are obtained from environment variables. For more information about how to configure the access credentials, see Configure access credentials.
To use multipart upload, you must have the
oss:PutObjectpermission. For more information, see Attach a custom policy to a RAM user.
Process
To upload a local file by using multipart upload, perform the following steps:
Initiate a multipart upload task.
Call the Client.InitiateMultipartUpload method to obtain a unique upload ID in OSS.
Upload parts.
Call the Client.UploadPart method to upload the parts.
NoteFor parts that are uploaded by running a multipart upload task with a specific upload ID, the part numbers identify their relative positions in an object. If you upload a part and reuse its part number to upload another part, the new part overwrites the original part.
OSS includes the MD5 hash of each uploaded part in the ETag header in the response.
OSS calculates the MD5 hash of the uploaded parts and compares the MD5 hash with the MD5 hash that is calculated by OSS SDK for Go. If the two hashes are different, OSS returns the InvalidDigest error code.
Complete the multipart upload task.
After all parts are uploaded, call the Client.CompleteMultipartUpload method to combine these parts into a complete object.
Examples
The following sample code provides an example on how to split a local large file into multiple parts, upload the parts to a bucket, and then combine these parts into a complete object:
package main
import (
"bufio"
"bytes"
"context"
"flag"
"io"
"log"
"os"
"sync"
"github.com/aliyun/alibabacloud-oss-go-sdk-v2/oss"
"github.com/aliyun/alibabacloud-oss-go-sdk-v2/oss/credentials"
)
// Define global variables.
var (
region string // The region in which the bucket is located.
bucketName string // The name of the bucket.
objectName string // The name of the object.
)
// Specify the init function used to initialize command line parameters.
func init() {
flag.StringVar(®ion, "region", "", "The region in which the bucket is located.")
flag.StringVar(&bucketName, "bucket", "", "The name of the source bucket.")
flag.StringVar(&objectName, "object", "", "The name of the source object.")
}
func main() {
// Parse command line parameters.
flag.Parse()
// Specify the upload ID.
var uploadId string
// Check whether the bucket name is specified.
if len(bucketName) == 0 {
flag.PrintDefaults()
log.Fatalf("invalid parameters, source bucket name required")
}
// Check whether the region in which the bucket is located is specified.
if len(region) == 0 {
flag.PrintDefaults()
log.Fatalf("invalid parameters, region required")
}
// Check whether the object name is specified.
if len(objectName) == 0 {
flag.PrintDefaults()
log.Fatalf("invalid parameters, source object name required")
}
// Load the default configurations and specify the credential provider and region.
cfg := oss.LoadDefaultConfig().
WithCredentialsProvider(credentials.NewEnvironmentVariableCredentialsProvider()).
WithRegion(region)
// Create an OSSClient instance.
client := oss.NewClient(cfg)
// Create a request to initiate the multipart upload task.
initRequest := &oss.InitiateMultipartUploadRequest{
Bucket: oss.Ptr(bucketName),
Key: oss.Ptr(objectName),
}
initResult, err := client.InitiateMultipartUpload(context.TODO(), initRequest)
if err != nil {
log.Fatalf("failed to initiate multipart upload %v", err)
}
// Display the result of initiating the multipart upload task.
log.Printf("initiate multipart upload result:%#v\n", *initResult.UploadId)
uploadId = *initResult.UploadId
// Initialize the wait group and mutex.
var wg sync.WaitGroup
var parts []oss.UploadPart
count := 3
var mu sync.Mutex
// Read data from the local file to the memory and replace yourLocalFile with the actual path of the local file that contains the file name.
file, err := os.Open("yourLocalFile")
if err != nil {
log.Fatalf("failed to open local file %v", err)
}
defer file.Close()
bufReader := bufio.NewReader(file)
content, err := io.ReadAll(bufReader)
if err != nil {
log.Fatalf("failed to read local file %v", err)
}
log.Printf("file size: %d\n", len(content))
// Calculate the size of each part.
chunkSize := len(content) / count
if chunkSize == 0 {
chunkSize = 1
}
// Start multiple goroutines for the multipart upload task.
for i := 0; i < count; i++ {
start := i * chunkSize
end := start + chunkSize
if i == count-1 {
end = len(content)
}
wg.Add(1)
go func(partNumber int, start, end int) {
defer wg.Done()
// Create a request to upload a part.
partRequest := &oss.UploadPartRequest{
Bucket: oss.Ptr(bucketName), // The name of the bucket.
Key: oss.Ptr(objectName), // The name of the object.
PartNumber: int32(partNumber), // The part number.
UploadId: oss.Ptr(uploadId), // The upload ID.
Body: bytes.NewReader(content[start:end]), // The content of the part.
}
// Send the request to upload the part.
partResult, err := client.UploadPart(context.TODO(), partRequest)
if err != nil {
log.Fatalf("failed to upload part %d: %v", partNumber, err)
}
// Display the result of the part upload request.
part := oss.UploadPart{
PartNumber: partRequest.PartNumber,
ETag: partResult.ETag,
}
// Use a mutex to protect shared data.
mu.Lock()
parts = append(parts, part)
mu.Unlock()
}(i+1, start, end)
}
// Wait until all goroutines are complete.
wg.Wait()
// Complete the multipart upload task.
request := &oss.CompleteMultipartUploadRequest{
Bucket: oss.Ptr(bucketName),
Key: oss.Ptr(objectName),
UploadId: oss.Ptr(uploadId),
CompleteMultipartUpload: &oss.CompleteMultipartUpload{
Parts: parts,
},
}
result, err := client.CompleteMultipartUpload(context.TODO(), request)
if err != nil {
log.Fatalf("failed to complete multipart upload %v", err)
}
// Display the result of the multipart upload task.
log.Printf("complete multipart upload result:%#v\n", result)
}
Common scenarios
Upload a random string of a specific length by using multipart upload
Cancel a multipart upload task
List the parts that are uploaded in a specific multipart upload task
List the multipart upload tasks of a bucket
Configure upload callbacks for multipart upload tasks
Multipart upload with progress monitoring
References
For the complete sample code that is used to perform multipart upload, visit GitHub.
A multipart upload involves three API operations. For more information about the operations, visit the following topics:
For more information about the API operation that you can call to cancel a multipart upload task, visit AbortMultipartUpload.
For more information about the API operation that you can call to list the uploaded parts, visit NewListPartsPaginator.
For more information about the API operation that you can call to list ongoing multipart upload tasks, visit NewListMultipartUploadsPaginator. Ongoing multipart upload tasks are tasks that are initiated but are not completed or canceled.