High-level commands - Object Storage Service - Alibaba Cloud Documentation Center

This topic describes high-level commands in ossutil.

Command description

Command scope	Supported options
All high-level commands	--encoding-type string: the encoding method of the object name or file name. Valid value: url. --request-payer string: the payer of the request. If pay-by-requester is enabled, specify this parameter. Valid value: requester.
High-level commands that support batch processing	--start-after/--end-with (, ]: specifies the range of object names that you want to query. The range is a left-open, right-closed interval. filter options: You can set filter conditions for object/file names, object/file directories, object/file sizes, object/file times, and object metadata. For more information, see Filter options. --limited-num: limits the number of results in the response. --recursive/-r: specifies that the operation is recursively performed on all files or objects, and all subdirectories in the root directory. --dirs/-d: matches files or objects only in the root directory and ignores subdirectories. Note If the option is applied to objects, a delimiter is used to simulate a directory, and all objects with the specified name prefix are scanned. The amount of time used to scan objects is directly proportional to the number of objects with the specified name prefix. --force/-f: forces the operation without requiring confirmation. --list-objects: lists objects by calling the ListObjects operation.
High-level commands that support filtering by destination object	--update: performs the operation only if the last modified time of the source file or object is later than that of the destination object. --size-only: performs the operation only if the source file or object has a different size from the destination object or file. --checksum: specifies that the operation is performed only when one of the following conditions is met: a. the size of the source object is different from that of the destination object; b. the CRC-64 of the source object is different from that of the destination object, but the size of the source object is the same as that of the destination object. If the CRC-64 does not exist on one side, the objects are considered inconsistent. This option is valid only for object-to-object copy operations. --ignore-existing: skips existing files.
Commands that support single object operations	--version-id string: the version ID of the object.
Commands that support list mode	--list-format: the format of objects in the list file. Valid values: plain and inventory. --list-manifest-from: reads the description information of the list file format from a file. When the list file format is inventory, you must set this parameter.

Local options

Local option	Option type	Description
--encoding-type	string	The encoding type of the object name or file name. Valid value: url.
--request-payer	string	The payer of the request. If pay-by-requester is enabled, specify this option. Valid value: requester.
--start-after	string	Matches objects whose names are lexicographically after the specified value (exclusive).
--end-with	string	Matches objects whose names are lexicographically before or the same as the value of the --end-with option.
--page-size	int	The maximum number of objects to list per page during batch processing (default value: 1000). Valid values: 1 to 1000.
--include	stringArray	The inclusion rule for paths or object names.
--include-from	stringArray	Reads inclusion rules from a file.
--exclude	stringArray	The exclusion rule for paths or object names.
--exclude-from	stringArray	Reads exclusion rules from a file.
--filter	stringArray	The filter rule for paths or object names.
--files-from	stringArray	Reads a list of source objects from a file, with empty lines or comment lines ignored.
--files-from-raw	stringArray	Reads a list of source objects from a file, with empty lines or comment lines ignored.
--max-age	Duration	Excludes objects whose last modified time is earlier than the value of this option. The default unit is s (seconds). You can suffix the numerical value with a unit: ms for milliseconds, s for seconds, m for minutes, h for hours, d for days, w for weeks, M for months, or y for years. Default value: off.
--min-age	Duration	Excludes objects whose last modified time is later than the value of this option. The default unit is s (seconds). You can suffix the numerical value with a unit: ms for milliseconds, s for seconds, m for minutes, h for hours, d for days, w for weeks, M for months, or y for years. Default value: off.
--max-mtime	Time	Excludes objects whose last modified time is later than the value of this option. The time must be in UTC. Default value: off.
--min-mtime	Time	Excludes objects whose last modified time is earlier than the value of this option. The time must be in UTC. Default value: off.
--max-size	SizeSuffix	The maximum size of a file or object that can be transferred. By default, the size is in bytes (B). You can suffix the numerical value with a 1024-based unit: B for byte, K for Kibibyte, M for Mebibyte, G for Gibibyte, T for Tebibyte, or P for Pebibyte.
--min-size	SizeSuffix	The minimum size of a file or object that can be transferred. By default, the size is in bytes (B). You can suffix the numerical value with a 1024-based unit: B for byte, K for Kibibyte, M for Mebibyte, G for Gibibyte, T for Tebibyte, or P for Pebibyte.
--metadata-exclude	stringArray	The exclusion rule for object metadata.
--metadata-include	stringArray	The inclusion rule for object metadata.
--metadata-filter	stringArray	The filter rule for object metadata.
--metadata-filter-from	stringArray	Reads the metadata filter rule from a file.
--checksum	/	Specifies that only when one of the following conditions is met, the source object is copied: a. the size of the source object is different from that of the destination object; b. the CRC-64 of the source object is different from that of the destination object, but the size of the source object is the same as that of the destination object.
--size-only	/	Copies a source object only if it has a different size from the destination object.
-u, --update	/	Copies a source object only if its last modified time is later than that of its destination object.
-r, --recursive	/	Specifies that the command performs the operation recursively. If this parameter is specified, the command takes effect on all objects that meet the conditions in the bucket. Otherwise, the command takes effect on only the objects in the specified path.
-d, --dirs	/	Returns objects and subdirectories in the current directory, instead of recursively displaying all objects in all subdirectories.
-f, --force	/	Forces the operation without requiring user confirmation.
--list-objects	/	Lists objects by calling the ListObjects operation.
--no-progress	/	Hides the progress bar.
-j, --job	int	The number of concurrent jobs for processing multiple files or objects. Default value: 3.
--checkers	int	The number of checkers that can run in parallel. Default value: 16.
--parallel	int	The number of concurrent tasks for internal operations on a single object.
--part-size	SizeSuffix	The part size. By default, ossutil calculates the appropriate part size based on the object size. The part size ranges from 100 KiB to 5 GiB.
--bigfile-threshold	int	The object size threshold for using multipart upload, download, or copy. Default value: 104857600.
--version-id	string	The version ID of the object.
--storage-class	string	The storage class of the object. Valid values: Standard, IA, Archive, ColdArchive, and DeepColdArchive.
--acl	string	The ACL of the object. Valid values: private, public-read, public-read-write, and default.
--cache-control	string	The caching behavior in web browsers when the object is downloaded.
--content-disposition	string	The display form of the object.
--content-encoding	string	The encoding method used to encode the object.
--content-type	string	The content type of the object.
--expires	string	Specifies the absolution expiration time of the cached content.
--copy-props	string	The attributes that you want to synchronize from the source object. Valid values: none, metadata, and default.
--tagging	string	The object tags in the key=value format.
--tagging-directive	string	The method used to configure the tags of the destination object. Valid values: COPY and REPLACE.
--metadata	string	The user metadata of the object in the key=value format.
--metadata-directive	string	The method used to configure the metadata of the destination object. Valid values: COPY and REPLACE.
--list-format	string	The format of objects in the list file. Valid values: plain and inventory.
--list-manifest-from	string	Reads the description information of the list file format from a file. When the list file format is inventory, you must set this parameter

Filter options

Filter options are applicable to commands that support batch processing, such as ls, cp, sync, and rm. Filter options match files or objects by path, file or object name, path list, file list, object list, last modified time, and file or object size. Objects can also be filtered by metadata.

Filter options can be used only in command lines. They cannot be specified in the configuration file.

You can use the ls command to test your filters. You can also use the --dry-run/-n option to perform a test run without making any changes.

Filter files or objects by path or name

When matching file names or file paths, you need to define filter rules to determine which files to include and which files to exclude.

Rules are defined as: --include (include) and --exclude (exclude), such as --include="*.txt", --exclude="*"

You can specify inclusion and exclusion rules directly in command lines or by writing them to a rule file. You can combine the --include and --exclude options based on your requirements and specify them multiple times in a single command. All occurrences of the options define a list of inclusion and exclusion rules, which match files or objects based on the following principles:

The order of the options matters. The rules are applied from the left to the right in command lines and top to bottom in rule files.
A detected file, object, or path is checked immediately for inclusion or exclusion.
Rule match checking occurs in order. If a file, object, or path is a match for an inclusion or exclusion rule, it is no longer checked against the subsequent rules.
If a file, object or path does not match any specified rules, the file, object, or path is included.
If a directory in a local file system is excluded, all subdirectories and files in the directory are excluded.
Objects are flatly stored in a bucket and are checked one by one for inclusion or exclusion.

ossutil uses glob matching, in which the following matching patterns are common:

If a pattern starts with /, the pattern must match the beginning of the path. Otherwise, the pattern can match any part of the path.
If a pattern ends with /, the pattern matches only directories. Otherwise, the pattern can match both directories and files. For example: dir/.
A / in the middle of a pattern represents a path separator. For example: subdir/test.txt.
* matches any character of any length, but does not match /. For example: *.text.
** at the end of a pattern matches any character of any length (including /). For example, dir/** matches the path dir/subdir/xx/test.txt.
? matches any single character except /.
[] matches a single character of a specific type. For example, [a-z] matches a lowercase letter, and [0-9] matches a digit.
By default, a pattern must completely align with the end of the path or file name. For example, the foo pattern matches foo and dir/foo, but does not match dir/foo1 or foo/xx.
A pattern requires a complete match of a file name, object name, or directory name, from the beginning to the end of the pattern. For example, foo does not match xxx/afoo, and abc/foo does not match subdir/aabc/foo.

Note

In Windows, / is still used as the path separator instead of \.

The following table describes filter rule syntax and options that you can use to filter files or objects by path or name:

Item	Description
Filter rule syntax	rule pattern: glob matching pattern rule filename: the path to the file that contains glob matching patterns rule modifiers pattern: instruction glob matching pattern. The instruction `+` indicates an inclusion rule, which means that objects that match the rule are included in the result. `-` indicates an exclusion rule, which means that objects that match the rule are excluded from the result.
Filter options	--include --include-from --exclude --exclude-from --filter --filter-from

The following table describes the filter options:

Option

Description

--include

--exclude

The matching pattern follows the parameter. Each option represents a single rule. You can specify multiple matching patterns in a command by repeating the option.

For example, --include "*.txt" --include "*.js" --exclude "*.jpg" includes all .txt and .js files but excludes all .jpg files.

The following sample command recursively downloads files that match the filter rules from the oss://examplebucket/exampledir/ directory to the local localdir/ directory. The filter rules include all .txt and .js files but exclude all .jpg files.

ossutil cp -r oss://examplebucket/exampledir/ localdir/ --include "*.txt" --include "*.js" --exclude "*.jpg"

--filter

The instruction matching pattern follows the parameter. Each option value represents a single rule. You can specify multiple matching patterns in a command by repeating the option.

For example, --filter "+ *.txt" --filter "+ *.js" --filter "- *.jpg" includes all .txt and .js files but excludes all .jpg files.

+ indicates an inclusion rule, which means that objects that match the rule are included in the result.
- indicates an exclusion rule, which means that objects that match the rule are excluded from the result.

The following sample command recursively downloads files that match the filter rules from the oss://examplebucket/exampledir/ directory to the local localdir/ directory. The filter rules exclude .txt and .js files and include only .png files.

ossutil cp -r oss://examplebucket/exampledir/ localdir/ --filter "- *.txt" --filter "- *.js" --filter "+ *.png"

--include-from

--exclude-from

The path to the rule file follows the parameter. Each line in the rule file represents a glob matching pattern. You can specify multiple rule files in a command by repeating the options.

The rule file that follows the --include-from option contains only inclusion rules. The rule file that follows the --exclude-from option contains only exclusion rules.

For example, --include-from include-file.txt --exclude-from exclude-file.txt includes all .txt and .js files but excludes all .jpg files.

The content of the include-file.txt rule file:

 *.txt
 *.js

The content of the include-file.txt rule file:

 *.jpg

--filter-from

The path to the rule file follows the parameter. Each line in the rule file represents an instruction glob matching pattern.

You can specify multiple rule files in a command by repeating the options.

For example, --filter-from filter-file.txt where the content of the filter-file.txt rule file is:

 # a filter rule file
 + *.txt
 + *.js
 - **

Additional notes:

When reading from a rule file, rules are processed from top to bottom. Empty lines or comment lines that start with # or ; are ignored.
When the file path is -, it indicates reading from standard input (stdin). Only one such file can be configured.
If you use the --include or --include-from rule, a - ** rule is added to the bottom of the rule list.
When using --filter+ pattern, a - ** rule is not added to the bottom of the rule list.

Example: When you need to configure a rule to include only files with the .jpg extension using --filter, you need to set it as --filter "+ *.jpg" --filter "- **".

Note

You cannot use --filter or --filter-from together with --include, --include-from, --exclude, or --exclude-from.

Filter files or objects by using a list

You can use the options described in the following table to specify a file that contains a list of paths, object names, or file names to filter:

Option

Description

--files-from

Reads paths, file names, or object names from the specified list file. You can specify multiple list files in a command by repeating the option.

When processing each line in the list file, leading or trailing spaces are removed, and lines that start with # or ; are treated as comments and ignored.

--files-from-raw

Reads paths, file names, or object names from the specified list file. You can specify multiple list files in a command by repeating the option.

When processing each line in the list file, leading or trailing spaces are not ignored, and lines that start with # or ; are not treated as comments.

Note

If you filter objects or files by using a list of paths, file names, or object names, other types of filters specified in the command are ignored.

Filter files or objects by last modified time

You can filter files or objects by last modified time.

You can specify relative and absolute time filters. The following table describes the options that filter files or objects by last modified time:

Parameter		Description
--min-age	Duration	Processes only files that were modified before the specified time interval. The default unit is seconds. You can use a unit suffix. For example, 1h indicates one hour. Note `--min-age 1h` processes only files that were modified one hour ago or earlier.
--max-age	Duration	Processes only files that were modified within the specified time interval. The default unit is seconds. You can use a unit suffix. For example, 1h indicates one hour. Note `--max-age 1h` processes only files that were modified within the last hour.
--min-mtime	Time	Processes only files that were modified after the specified time. The time format is UTC. For example: 2006-01-02T15:04:05. Note `--min-mtime "2006-01-02T15:04:05"` processes only files that were modified after January 2, 2006, 15:04:05.
--max-mtime	Time	Processes only files that were modified before the specified time. The time format is UTC. For example: 2006-01-02T15:04:05. Note `--max-mtime "2006-01-02T15:04:05"` processes only files that were modified before January 2, 2006, 15:04:05.

Additional notes:

The --min-age and --max-age options allow you to suffix the numerical value with a unit of time: ms for milliseconds, s for seconds, m for minutes, h for hours, d for days, w for weeks, M for months, and y for years.
The numerical value before a unit can be a decimal, as in 1.5d (one and a half days).

The following sample command lists files or objects that were last modified more than three days ago:

ossutil ls oss://bucket/ --min-age 3d

Filter files or objects by data size

You can filter files or objects by their data size to include files or objects whose sizes are within the specified range.

The following table describes the options that you can use to filter files and objects by data size:

Parameter	Description
--min-size	The minimum size of a file or object that can be included. By default, the size is in bytes (B). You can specify a 1024-based size unit after the numeric value, as in 1K (1,024 bytes).
--max-size	The maximum size of a file or object that can be included. By default, the size is in bytes (B). You can specify a 1024-based size unit after the numeric value, as in 1K (1,024 bytes).

Additional notes:

You can specify one of the following short-form size units: B for byte, K for Kibibyte, M for Mebibyte, G for Gibibyte, T for Tebibyte, or P for Pebibyte.
The numerical value before a unit can be a decimal, as in 0.5K.

The following sample command lists files or objects that do not exceed 1 MiB in size:

ossutil ls oss://bucket/ --max-size 1M

Filter objects by metadata

Metadata-based filtering applies only to objects in a bucket. You can filter objects by metadata in a similar way you filter files or objects by path, file name, or object name.

Metadata-based filtering uses glob matching and requires key=value representation of metadata filter rules.

You can use the following options to filter objects by metadata:

--metadata-include
--metadata-exclude
--metadata-filter
--metadata-filter-from

You can filter objects by using the following metadata items:

Storage class. To filter objects by storage class, specify x-oss-storage-class=value, where the value can be Standard, IA, Archive, ColdArchive, or DeepColdArchive.
Object type. To filter objects by object type, specify x-oss-object-type=value, where the value can be Normal, Multipart, Appendable, or Symlink.
Restoration status. To filter objects by restoration status, specify x-oss-restore=value, where the value is the actual restoration status that you want to filter by.
Content type. To filter objects by content type, specify content-type=value, where the value is the actual content type that you want to filter by.
User metadata. To filter objects by user metadata, specify x-oss-meta-aaa=value, where aaa is the custom attribute name in lowercase and value is the metadata value.

The following sample command lists objects in the Archive, Cold Archive, and Deep Cold Archive storage classes in the specified bucket:

ossutil ls oss://bucket/ --metadata-include "x-oss-storage-class=*Archive"