batch-ops
Performing Batch Operations
You can use Amazon S3 batch operations to perform large-scale batch operations on Amazon S3 objects. Amazon S3 batch operations can execute a single operation on lists of Amazon S3 objects that you specify. A single job can perform the specified operation on billions of objects containing exabytes of data. Amazon S3 tracks progress, sends notifications, and stores a detailed completion report of all actions, providing a fully managed, auditable, serverless experience. You can use Amazon S3 batch operations through the AWS Management Console, AWS CLI, AWS SDKs, or REST API.
Use Amazon S3 batch operations to copy objects and set object tags or access control lists (ACLs). You can also initiate object restores from Amazon S3 Glacier or invoke an AWS Lambda function to perform custom actions using your objects. You can perform these operations on a custom list of objects, or you can use an Amazon S3 inventory report to make generating even the largest lists of objects easy. Amazon S3 batch operations use the same Amazon S3 APIs that you already use with Amazon S3, so you'll find the interface familiar.
Topics
- Terminology
- The Basics: Amazon S3 Batch Operations Jobs
- Creating an Amazon S3 Batch Operations Job
- Operations
- Managing Batch Operations Jobs
- Amazon S3 Batch Operations Examples
Terminology
This section uses the terms jobs, operations, and tasks, which are defined as follows:
Job
A job is the basic unit of work for Amazon S3 batch operations. A job contains all of the information necessary to execute the specified operation on the objects listed in the manifest. After you provide this information and request that the job begin, the job executes the operation for each object in the manifest.
Operation
The operation is the type of API action, such as copying objects, that you want the batch operations job to execute. Each job performs a single type of operation across all objects that are specified in the manifest.
Task
A task is the unit of execution for a job. A task represents a single call to an Amazon S3 or AWS Lambda API operation to perform the job's operation on a single object. Over the course of a job's lifetime, Amazon S3 batch operations create one task for each object specified in the manifest.