AWS S3 is the object storage service provided by AWS. It is the most widely used storage service from AWS that can virtually hold an infinite amount of data. It is highly available, durable, and easy to integrate with several other AWS Services. AWS S3 can be used by people with any requirements like mobile/web application storage, big data storage, machine learning data storage, hosting static websites, and many more. If you have been using S3 in your project, you would know that given the vast amount of storage capacity, the management of 100s of buckets and terabytes of data in these buckets can be a demanding job. We have a list of AWS S3 commands with examples that you can use to manage your AWS S3 buckets and data efficiently.
AWS CLI Setup
After you have successfully downloaded and installed the AWS CLI, you need to configure AWS Credentials to be able to access your AWS Account and services. Let us quickly run through how you can configure AWS CLI. The first step is to create a user with programmatic access to AWS Account. Remember to check this box when you create a user for AWS CLI. Give the permissions and create a user. At the final screen after you successfully created this user copy the Access key ID and Secret access key for this user. We will use these credentials to log in via the AWS CLI. Now go to the terminal of your choice and run the following command. Enter the Access key ID and Secret access key when prompted. Select any AWS region of your choice and the command output format. I personally prefer using the JSON format. This is no big deal you can always change these values later. You can now run any AWS CLI Command in the console. Let us now go through the AWS S3 Commands.
cp
The cp command simply copies the data to and from S3 buckets. It can be used to copy files from local to S3, from S3 to local, and between two S3 buckets. There are a lot of other parameters that you can supply with the commands. For example, -dryrun parameter to test the command, –storage-class parameter to specify the storage class of your data in S3, other parameters to set encryption, and much more. The cp command gives you complete control over how you configure your data security in S3.
Usage
Examples
Copy data from local to S3
Copy data from S3 to local
Copy data between S3 buckets
Copy data from local to S3 – IA
Copy all the data from a local folder to S3
ls
The ls command is used to list the buckets or the contents of the buckets. So, if you simply want to view information about your buckets or the data in these buckets you can use the ls command.
Usage:
Examples
List all buckets in the account
This command list all the buckets in your account with the bucket creation date.
List all top-level objects in a bucket
This command list all the top-level objects in an S3 bucket. Note here that the objects with the prefix samplePrefix/ are not shown here only the top-level objects.
List all the objects in a bucket
This command list all the objects in an S3 bucket. Note here that the objects with the prefix samplePrefix/ and all the sub prefixes are also displayed.
mb
The mb command is simply used to create new S3 buckets. This is a fairly simple command but to create new buckets, the name of the new bucket should be unique across all the S3 buckets.
Usage
Example
Create a new bucket in a specific region
mv
The mv command simply moves the data to and from S3 buckets. Just like the cp command, mv command is used to move data from local to S3, S3 to local, or between two S3 buckets. The only difference between the mv and the cp command is that when using the mv command the file is deleted from the source. AWS moves this file to the destination. There are a lot of options that you can specify with the command.
Usage
Examples
Move data from local to S3
Move data from S3 to local
Move data between S3 buckets
Move data from local to S3 – IA
Move all the data from a prefix in S3 to a local folder.
presign
The presign command generates a pre-signed URL for a key in the S3 bucket. You can use this command to generate URLs that can be used by others to access a file in the specified S3 bucket key.
Usage
aws s3 presign <OBJECT_KEY> –expires-in <TIME_IN_SECONDS>
Example
Generate a pre-signed URL that is valid for 1 hour for an object in the bucket.
rb
The rb command is simply used to delete S3 buckets.
Usage
Example
Delete an S3 bucket.
Delete an S3 bucket along with the data in the S3 bucket.
rm
The rm command is simply used to delete the objects in S3 buckets.
Usage
Examples
Delete one file from the S3 bucket.
Delete all files with a specific prefix in an S3 bucket.
Delete all files in an S3 bucket.
sync
The sync command copies and updates files from the source to the destination just like the cp command. It is important that we understand the difference between the cp and the sync command. When you use cp it copies data from source to destination even if the data already exists in the destination. It will also not delete files from the destination if they are deleted from the source. However, sync looks at the destination before copying your data and only copies the new and updated files. The sync command is similar to committing and pushing changes to a remote branch in git. The sync command offers a lot of options to customize the command.
Usage
Examples
Sync local folder to S3
Sync S3 data to a local folder
Sync data between two S3 buckets
Move data between two S3 buckets excluding all .txt files
website
You can use S3 buckets to host static websites. The website command is used to configure the S3 static website hosting for your bucket. You specify the index and the error files and the S3 gives you a URL where you can view the file.
Usage
Example:
Configure static hosting for an S3 bucket and specify the index and error files
Conclusion
I hope the above gives you an idea about some of the frequently used AWS S3 commands to manage buckets. If you are interested in learning more, you may check out AWS certification details.
