How to Use the AWS CLI with S3-Compatible Object Storage: Uploading, Synchronization, and Access Permissions

To connect the AWS CLI to a third-party provider’s S3-compatible Object Storage, you need four key parameters:

• Access Key;
• Secret Key;
• endpoint;
• region.

It is best to create a separate profile right away: aws configure --profile storage

You can test the connection using a safe read command:

   aws s3 ls \
--endpoint-url https://s3.example-provider.com \
--profile storage

Create a bucket:

   aws s3 mb s3://my-bucket \
--endpoint-url https://s3.example-provider.com \
--profile storage

Upload a file:

   aws s3 cp ./file.txt s3://my-bucket/file.txt \
--endpoint-url https://s3.example-provider.com \
--profile storage

Download it again:

   aws s3 cp s3://my-bucket/file.txt ./file.txt \
--endpoint-url https://s3.example-provider.com \
--profile storage

Synchronize a directory:

   aws s3 sync ./data s3://my-bucket/data/ \
--endpoint-url https://s3.example-provider.com \
--profile storage

Before using --delete, first check the result with --dryrun:

   aws s3 sync ./data s3://my-bucket/data/ \
--delete \
--dryrun \
--endpoint-url https://s3.example-provider.com \
--profile storage

If the AWS CLI returns an error, check the following in order: the endpoint, profile and credentials, region, system clock, policy permissions, and addressing style.

For production environments, do not store the Secret Key directly in scripts. Use dedicated service accounts with least-privilege access, and plan key rotation in advance.

What Is S3-Compatible Object Storage, and How Does It Differ from a Regular Disk?

Welcome!

Today, we’ll explore one of those technologies that people use virtually every day but don’t always fully understand, what exactly happens under the hood.

We’re talking about S3-compatible object storage.

At first glance, everything looks very familiar: there are files, something resembling folders, and you can upload data, download it again, and even synchronize entire directories.

This naturally raises the question: “So, is it just a remote disk?”

Not exactly.

Object storage works differently from a conventional file system, network drive, or VPS partition. This distinction is important because it explains almost everything else: why there are buckets, object keys, endpoints, request signatures, and separate access control rules.

In this article, we will use the AWS CLI, but you do not necessarily have to connect to Amazon S3. Our broader goal is to understand how to use an S3-compatible storage service from a third-party provider through a custom endpoint.

By the end, we will know how to:

• Connect to the storage service using the AWS CLI;
• Work with separate profiles;
• Create and view buckets;
• Upload and download objects;
• Synchronize directories;
• Manage access;
• Troubleshoot errors such as AccessDenied and SignatureDoesNotMatch.

But before moving on to the commands, it is worth gaining a proper understanding of the storage model itself.

Why Object Storage Is Not a File System

A conventional file system is built around directories, files, and a hierarchy.

For example: /home/user/photos/cat.jpg

This path contains:

• The /home directory;
• The user subdirectory inside it;
• Then the photos subdirectory;
• And finally, the cat.jpg file.

The file system recognizes that directories exist as separate entities.

Object Storage works differently.

In an S3-like model, data is stored as objects, and each object has a unique key.

For example: photos/cat.jpg may look like a path, but technically it is just a string that serves as the object key.

In other words, object storage does not have to treat photos/ as an actual directory.

It’s more of a prefix.

This is why you may encounter an object backup/2026/09/database.sql, even if no separate backup, 2026, or 09 directories were created beforehand.

The interface may display these as folders because that makes the structure easier for people to understand. At the API level, however, they are usually just keys with common prefixes.

This is one of the key characteristics of Object Storage.

This gives rise to other differences as well.

For example, a conventional disk is well suited for:

• Files that an application accesses constantly;
• Databases;
• System directories;
• Working directories;
• Files that undergo frequent small changes.

Object Storage is more commonly used for:

• Backups;
• Archives;
• Images;
• Videos;
• Documents;
• Static files;
• Logs;
• Large datasets.

Why?

Because the “complete object identified by a key” model is well suited to storing data that needs to be reliably stored, retrieved, and served through an API. As a result, it is easier to build stateless infrastructure in which an application uses an S3 service for data instead of local directories. This facilitates horizontal scaling and simplifies microservice architecture.

However, it does not attempt to behave like a local file system with all its familiar operations.

Can S3 be mounted as a regular file system?

Technically, yes. Tools such as s3fs allow you to mount S3-compatible storage on Linux and access objects through standard file system paths.

However, this does not turn S3 into an actual file system. s3fs uses FUSE to translate file system operations into S3 API requests. As a result, operations that are standard on a local file system may behave differently, run more slowly, or require additional requests to Object Storage.

For this reason, this approach is generally not used as a full replacement for local storage for databases, frequently modified files, or applications that expect a standard POSIX file system. It is more commonly found in legacy scenarios, migrations, and cases where an existing application cannot be quickly adapted to work natively with the S3 API.

For new applications, it is generally more reliable to use Object Storage as an object store, accessing buckets and object keys through the S3 API, AWS CLI, or an SDK.

What Are Buckets, Objects, and Keys?

To avoid confusion later, let’s review these three basic terms.

Bucket

Bucket is a top-level container for objects.

For example: my-backups or site-media

A bucket can be thought of as a separate storage space.

However, a bucket is more than just a regular folder.

The following can be configured at the bucket level:

• Access rules;
• Region;
• Versioning;
• Lifecycle rules;
• Policies;
• Restrictions;
• Public access settings.

The available features depend on the specific S3-compatible provider.

Object

Object is the actual data we uploaded.

For example: report.pdf or database.sql.gz

An object typically includes not only the file contents but also metadata.

In other words, S3 stores more than just a “set of bytes”; it stores an object with additional information.

Key

Key is the unique name of an object within a bucket.

For example: backups/db.sql or images/2026/avatar.png

A key may look like a regular path, but it is still a string.

An object within a bucket is located by its key.

If we have:

   Bucket: my-backups
Key: daily/postgres.sql.gz

then an example object address can be represented as: s3://my-backups/daily/postgres.sql.gz

This notation is commonly used in the AWS CLI.

For example: aws s3 cp backup.sql.gz s3://my-backups/daily/backup.sql.gz

Here, my-backups is the bucket, and daily/backup.sql.gz is the key.

This understanding becomes especially helpful later when using sync, cp --recursive, and prefixes.

What Does “S3-Compatible” Mean?

Now for the most interesting part.

S3 is Amazon Simple Storage Service.

Over time, its API became so widely adopted that many other cloud providers began implementing a compatible interface.

This gave rise to the term: S3-compatible Object Storage.

In other words, a third-party storage service aims to support an API and interaction model compatible with Amazon S3.

This allows the same client to work with different providers.

For example, the AWS CLI was originally designed for AWS, but with the right configuration, it can also send requests to another S3-compatible storage service.

This is convenient because there is no need to learn a separate utility for each provider.

You can use familiar commands:

   aws s3 ls
aws s3 cp
aws s3 sync

and simply specify a different endpoint.

But the term compatible does not mean: “This is a 100% exact copy of Amazon S3, with all its features and behavior.”

The following may vary between providers:

• ACL support;
• Bucket policies;
• Versioning;
• Object Lock;
• Lifecycle rules;
• Regions;
• Naming restrictions;
• How addresses are constructed;
• Certain API features.

Therefore, S3 compatibility generally means compatibility with the core API and common operations rather than a guarantee of complete equivalence.

This is especially important when migrating complex infrastructure between providers.

A simple cp operation usually works without any issues.

However, less common configurations, such as specific bucket policies, may behave differently.

Why a Third-Party Provider Has Its Own Endpoint

Amazon S3 has its own API endpoints.

A third-party provider uses different infrastructure, so requests must be sent to its servers instead.

An endpointis used for this purpose.

An endpoint is the API address that a client connects to.

For example: https://s3.example-provider.com or https://object-storage.example.net

This is where a key difference between using AWS directly and using S3-compatible storage comes into play.

If you run aws s3 ls without any additional configuration, the AWS CLI targets Amazon’s infrastructure by default.

We need to tell it: “Use the same S3 API, but send requests here.”

To do this, use: --endpoint-url

For example:

   aws s3 ls \
--endpoint-url https://s3.example-provider.com

The endpoint determines, where the API request is actually sent.

Meanwhile, the Access Key and Secret Key identify something else— who exactly is sending the request.

It is important to distinguish between these.

In simple terms:

This is why endpoint errors and credential errors look different.

If the endpoint is completely unavailable, the AWS CLI will be unable to establish a connection.

If the endpoint is correct but the credentials are invalid, the request will reach the server, which will then return an authorization error.

Next, we will examine how the AWS CLI constructs these requests, why Access Key and Secret Key are required, what Signature Version 4 is, and why even the server’s system time can unexpectedly affect authorization.

How the AWS CLI Connects to S3-Compatible Storage

On the surface, it looks simple: we enter a command such as aws s3 ls and get a list of buckets.

However, several important steps take place between entering the command and receiving the server’s response. The CLI must determine where to send the request, which credentials to use to sign it, which region to use, and where to obtain all these settings.

This is why, when working with a third-party S3-compatible storage service, issues typically arise not from the cp or sync command itself, but from an incorrect endpoint, keys, or request signature.

Why You Need a Custom Endpoint

By default, the AWS CLI expects to work with Amazon infrastructure unless otherwise specified.

However, a third-party provider uses different infrastructure, a different API address, and possibly a different region.

Therefore, you need to specify the endpoint explicitly.

For example: https://s3.example-provider.com

An endpoint is the address to which the AWS CLI sends S3 requests.

For example, when you run:

   aws s3 ls \
--endpoint-url https://s3.example-provider.com

The CLI does not change the behavior of the S3 command itself. It simply directs the request to a different server.

This is one of the key advantages of S3 compatibility: the client remains the same, and only the endpoint changes.

It is especially important not to confuse the endpoint with the address of a specific bucket.

Typically, the endpoint is the address of the service API, while the bucket is specified separately: s3://my-bucket

Depending on the provider, the request may then take one of the following forms: https://s3.example.com/my-bucket/object.txt or https://my-bucket.s3.example.com/object.txt

We will revisit this distinction when discussing path-style and virtual-hosted-style requests.

How Access Key and Secret Key Work

The endpoint answers the question: Where should the request be sent?

But the storage system also needs to determine who exactly is accessing the API.

Two related credentials are used for this purpose:

• Access Key ID;
• Secret Access Key.

An Access Key can be thought of as an identifier for an account or service user.

It is not a secret in the same sense as a password, but it should still not be shared unnecessarily.

The Secret Key, however, is sensitive information.

It is used to generate the cryptographic signature for the request.

The AWS CLI does not send the Secret Key in plaintext with the request.

Instead, it uses the key as input when calculating the signature.

The server knows the corresponding secret and can verify that the signature is valid.

This is important because the authentication process is not simply a username-and-password scheme in the conventional web sense.

Instead, every API request is signed.

Conceptually, it works like this: “I am the user with Access Key X. Here is the request I am sending, and here is the cryptographic signature proving that the request was generated by the holder of the correct Secret Key.”

This is why losing a Secret Key is critical.

If it is compromised, an attacker may be able to sign requests on behalf of your account until the key is revoked or replaced.

What Is Signature Version 4?

Most modern S3-compatible systems use AWS Signature Version 4, or SigV4 for short.

It is a mechanism for signing API requests.

The AWS CLI uses several request elements:

• HTTP method;
• Path;
• Query parameters;
• Headers;
• Timestamp;
• Region;
• Service name;
• Payload hash.

These elements are used to create a canonical representation of the request.

A signature is then calculated using the Secret Key.

As a result, the server receives not just GET /bucket/file.txt, but a signed request with additional authorization headers.

SigV4 allows the server to verify several things at once:

• Who sent the request;
• Whether the request was modified in transit;
• Whether the signature matches the specific region and service;
• Whether the request is too old.

This is why an error in a seemingly unrelated parameter can result in SignatureDoesNotMatch.

For example, even if the endpoint is correct, the signature may be generated differently if the configured region does not match the one expected by the S3 provider.

The same issue can occur if the client and server interpret the path or host differently.

This is one reason why S3-compatible does not always mean there are no nuances at all.

Why System Time Affects Authentication

This is one of those details that is easy to overlook.

A signed request includes a timestamp.

This is intentional: the server must not accept the same correctly signed request indefinitely.

Therefore, if the system clock is significantly behind or ahead, the server may determine that the request is too old or has arrived “from the future.”

This can result in errors such as RequestTimeTooSkewed or similar signature-related issues.

On a typical VPS, the system clock is usually synchronized automatically via NTP or systemd-timesyncd.

You can check the status by running: timedatectl

Look for lines such as:

   System clock synchronized: yes
NTP service: active

If the system time is significantly off, fix the synchronization before investigating the keys.

This is a particularly useful diagnostic principle: if the keys appear correct, the endpoint responds, but the signature still does not match, check the system time and region.

Where the AWS CLI Stores Configuration and Credentials

The AWS CLI typically stores configuration settings and sensitive data in two separate files in the user’s home directory.

The main paths are ~/.aws/credentials and ~/.aws/config

The credentials file usually contains the Access Key and Secret Key.

For example:

   [storage]
aws_access_key_id = ACCESS_KEY
aws_secret_access_key = SECRET_KEY

The config file contains the profile settings:

   [profile storage]
region = us-east-1
output = json

This separation is useful because credentials can be protected more strictly, while configuration settings can be changed independently.

However, there is an important caveat: the endpoint of a third-party S3 provider is not always saved there automatically after running the standard aws configure command.

In practice, one of two approaches is usually used:

• Pass –endpoint-url each time;
• Or configure the endpoint using the configuration options supported by the specific AWS CLI version.

For this tutorial, we will initially specify --endpoint-url explicitly because this makes it easier to see exactly where the request is sent.

There is one more important point.

The ~/.aws/credentials and ~/.aws/config files are associated with a specific Linux user.

If the AWS CLI is run as another user, for example via sudo, it may look in /root/.aws/ instead of /home/ubuntu/.aws/

This can sometimes lead to a confusing situation:

aws s3 ls

→ works

sudo aws s3 ls

→ credentials not found

The problem is not with S3; the command was run in a different user environment.

Command Quick Reference

At this point, it is clear that connecting to S3 involves more than simply providing a URL and password.

The AWS CLI must bring several components together: the endpoint, credentials, region, system time, and signing rules.

Next, we will install the AWS CLI, create a separate profile for our storage service, and configure it to keep credentials for different services separate.

Installing the AWS CLI and Creating a Separate Profile

It is important not only to install the AWS CLI, but also to configure it from the outset so that access to S3-compatible storage does not use the same credentials as other services.

This is especially useful if the same server or workstation will later be used simultaneously with:

• Amazon S3;
• Third-party S3-compatible storage;
• Separate backup storage;
• Test and production accounts.

Rather than using a single shared configuration for everything, it is more convenient to create a separate profile.

Why a Profile Is More Convenient Than a Global Configuration

The AWS CLI can work with multiple sets of configuration settings.

Each such set is called a profile.

For example:

default

storage

backup

production

Each profile can have its own:

• Access Key;
• Secret Key;
• region;
• output format;
• additional parameters.

Up to €20,000 in EU-Based IaaS Credits

Frankfurt infrastructure, predictable pricing, and expert migration support.

Get

This lets you explicitly tell the AWS CLI to use a specific set of credentials.

For example: aws s3 ls --profile storage or aws s3 ls --profile backup

This makes it much harder to accidentally send a command to the wrong storage service or use the wrong keys.

Without profiles, you typically end up with a single global set of parameters: default

Once you add a second provider, confusion sets in: Which keys are currently active? Which region is selected? Why did the command work yesterday but suddenly return AccessDenied today?

Therefore, we will create a separate profile for our S3-compatible storage from the outset: storage

Parameters to Obtain from Your Provider

Before configuring the AWS CLI, you need to obtain several values from your cloud provider’s control panel.

You will typically need the following:

Field names may vary slightly between provider control panels.

For example, Access Key ID may instead be labeled Access Key or S3 Access Key.

The same applies to the endpoint.

It may look like this: https://s3.example-provider.com, or it may include a region: https://s3.eu.example-provider.com

Be sure to use the values from your provider’s documentation or control panel rather than copying an endpoint from someone else’s article.

The S3 API is compatible across providers, but each service has its own infrastructure address.

Where the credentials and config files are stored

After you configure a profile, the AWS CLI creates or updates the following directory: ~/.aws/

This directory typically contains two files.

First: ~/.aws/credentials

This file stores the keys.

For example:

   [storage]
aws_access_key_id = ACCESS_KEY
aws_secret_access_key = SECRET_KEY

The second: ~/.aws/config

This file contains the remaining profile settings.

For example:

   [profile storage]
region = us-east-1
output = json

Note the slight difference in syntax.

In the credentials file, the profile is specified as [storage].

In the config file, it is specified as [profile storage].

This is the standard AWS CLI format.

You can verify the created files as follows: ls -la ~/.aws/

However, avoid displaying the contents of the credentials file in the terminal unless necessary.

Why Keys Should Not Be Put in Scripts

A Secret Access Key should be treated much like an API password.

If it is exposed, an attacker can use it to sign requests on behalf of your account.

The consequences depend on the permissions granted.

If the key has broad access, it could potentially be used to:

• Read objects;
• Upload new objects;
• Overwrite existing objects;
• Delete data;
• List buckets;
• Change certain settings if permitted by the applicable policies.

The following is a bad practice:

aws configure set aws_secret_access_key VERY_SECRET_KEY

if the command then remains in the shell history, automation logs, or documentation.

Even worse:

   aws s3 sync ./backup s3://my-bucket \
--some-secret-parameter VERY_SECRET_KEY

if the secret is passed directly on the command line.

AWS CLI allows credentials to be stored separately so that they do not have to be included in every command.

Hands-on: Installing the AWS CLI and Creating a Profile

On Ubuntu, first check whether the AWS CLI is installed: aws –version

If the command is not found, you can install the package:

   sudo apt update
sudo apt install -y awscli

After installation, check again: aws –version

Now create a separate profile: aws configure --profile storage

The AWS CLI will prompt you for the following:

   AWS Access Key ID:
AWS Secret Access Key:
Default region name:
Default output format:

For example:

   AWS Access Key ID: ACCESS_KEY
AWS Secret Access Key: SECRET_KEY
Default region name: us-east-1
Default output format: json

We will not specify the endpoint here yet.

In the following commands, we will pass it explicitly using --endpoint-url.

This will make the connection configuration as clear as possible.

After configuration, verify the profile: aws configure list --profile storage

Then list all profiles: aws configure list-profiles

Commands Used in This Chapter

We now have a separate profile with credentials, but we have not yet verified that the endpoint and keys actually work together.

Testing the Connection Through a Custom Endpoint

The profile has already been created, the keys have been saved, and the AWS CLI knows which profile name to use for this set of credentials.

Now it is time to check what matters most: whether the client can actually connect to a specific provider’s S3-compatible storage.

At this point, it is best not to start by uploading files, deleting objects, or creating a bucket. First, run a safe read-only request to verify that the endpoint, profile, and request signature work together correctly.

Why You Should Start with a Safe Read-Only Request

If the configuration has not yet been verified, the first command should avoid making any changes whenever possible.

For example, instead of running aws s3 mb s3://my-bucket, start with a simple listing command: aws s3 ls

Why is this useful?

Because it immediately separates two tasks:

• Connection and authentication;
• Modifying Data.

If the read-only request succeeds, the basic chain is already working:

• the endpoint is reachable;
• DNS resolution works;
• the TLS connection is established;
• the Access Key is recognized;
• the Secret Key is valid for request signing;
• the region and request format do not cause conflicts;
• AWS CLI successfully reads the profile.

If the request fails, we have not created, deleted, or overwritten anything.

This is a sound general principle for infrastructure tools: first verify access using the least risky operation, and only then proceed with changes.

How to verify that the endpoint and keys work

Now let’s add a custom endpoint.

Suppose the provider supplied the following URL:

Let’s check the list of available buckets:

   aws s3 ls \
--endpoint-url https://s3.example-provider.com \
--profile storage

If the account already has buckets, the output may look like this:

   2026-09-15 12:31:42 backups
2026-09-15 12:33:18 site-media

If there are no buckets yet, the command may simply return an empty result without an error.

This also indicates a successful connection.

An empty list means that the AWS CLI reached the storage service and successfully completed the request, but there is nothing to display yet.

Now you can check a specific bucket:

   aws s3 ls s3://my-bucket \
--endpoint-url https://s3.example-provider.com \
--profile storage

If the bucket already contains objects, their names, sizes, and dates will be displayed.

For example:

   2026-09-15 13:02:10      14321 report.pdf
2026-09-15 13:03:54    2849012 backup.sql.gz

If a prefix-based structure is used, the AWS CLI may display entries that resemble directories, such as PRE backups/ and PRE images/

However, as discussed earlier, these are not necessarily actual directories. The CLI simply groups keys by prefix for readability.

Another useful way to check is to use a low-level S3 API command:

   aws s3api list-buckets \
--endpoint-url https://s3.example-provider.com \
--profile storage

Unlike aws s3, the s3api command set maps more closely to direct S3 API operations and typically returns structured JSON.

For example:

   {
"Buckets": [
{
"Name": "backups"
}
]
}

For everyday tasks, aws s3 is more convenient, while s3api is particularly useful for diagnostics and fine-grained configuration.

How a connection error differs from an authorization error

This is where diagnostics become genuinely useful.

Not every AWS CLI error means “incorrect password.”

Suppose you receive: Could not connect to the endpoint URL or EndpointConnectionError

This means that the client was completely unable to establish a proper connection to the endpoint.

The causes may be network-related:

• The URL is incorrect;
• The endpoint does not exist;
• DNS resolution fails;
• The port is unreachable;
• A firewall is blocking the connection;
• The TLS certificate fails validation;
• The service is experiencing a temporary issue.

In other words, the request may fail before the Access Key is even validated.

You can test the address separately:

curl -I https://s3.example-provider.com

You should not necessarily expect a 200 OK response: an S3 endpoint may return a 403, a 400, or an XML error for an unsigned request.

The important thing is to determine whether the server responds at all.

InvalidAccessKeyId is a completely different case

In this case, the endpoint is accessible and the request has reached the S3 service, but the specified Access Key was not accepted.

Possible causes:

• A typo in the key;
• The key has been deleted;
• The key belongs to a different account;
• The key belongs to a different S3 provider;
• The wrong profile is selected.

SignatureDoesNotMatch means that the server received the request and recognized the Access Key, but the calculated signature did not match.

In this case, check the following:

• Secret Key;
• Region;
• System time;
• Endpoint;
• How the host and path are constructed;
• The specific characteristics of the S3-compatible service.

Another error is AccessDenied

This often indicates a more nuanced issue: authentication succeeded, but the user does not have permission to perform the specific action.

For example, a service user may be allowed to read objects from one bucket but prohibited from performing ListAllMyBuckets.

In this case, aws s3 ls may return AccessDenied, while aws s3 ls s3://allowed-bucket succeeds.

And this is an important distinction: an error when listing all buckets does not necessarily mean that the credentials do not work at all.

The permissions may have been intentionally restricted.

This is why, when troubleshooting, it is useful to know which permissions the provider or administrator granted.

Quick Command Reference

At this point, we have confirmed that the profile and endpoint work together.

We can now move from safe requests to making changes and create the first bucket.

Creating a Bucket

We covered the concept of a bucket at the beginning of this article, so we will not revisit how it works here. At this stage, we will focus on the practical aspects: how to create a bucket using the AWS CLI and which parameters can affect the outcome.

The operation may seem straightforward, but creating a bucket often reveals provider-specific nuances in S3-compatible storage—most notably the region and bucket name uniqueness requirements.

How the region affects bucket creation

Region handling may vary across S3-compatible services.

One provider may use a single region for its entire Object Storage service, while another may use multiple regions, each with its own endpoint.

For example:

The region may affect not only data placement but also SigV4 signature generation.

Therefore, if a profile is configured with one value, such as region = us-east-1, but the endpoint expects another, some operations may fail.

This is especially common when creating a bucket.

In some S3 implementations, the following standard command is sufficient:

   aws s3 mb s3://my-bucket \
--endpoint-url https://s3.example-provider.com \
--profile storage

In other cases, the region may need to be specified explicitly using API parameters.

For example:

   aws s3api create-bucket \
--bucket my-bucket \
--create-bucket-configuration LocationConstraint=eu-1 \
--endpoint-url https://s3.example-provider.com \
--profile storage

The exact LocationConstraint value must be taken from the documentation for the specific provider.

In other words, if aws s3 mb returns an error, do not immediately assume that the access keys are the problem. The endpoint and authentication may be working correctly, while the service simply expects a different region parameter.

Why Bucket Name Uniqueness Requirements Depend on the Provider

Another consideration is the scope within which a bucket name must be unique.

In Amazon S3, bucket names exist in the service’s global namespace, so a simple name such as: backup, is almost certainly already taken.

Third-party S3-compatible providers may have different rules.

A bucket name may need to be unique:

• Globally;
• Only within a region;
• Within a project;
• Within an account;
• Within a specific endpoint.

For this reason, it is safer to choose more specific names from the outset.

For example: company-project-prod-backups or site-media-eu-2026 instead of: data or backup

This reduces the risk of naming conflicts and keeps the bucket’s purpose clear even months later.

There is another practical reason not to treat the name as an arbitrary label: it may be part of an object’s URL and used in scripts, access policies, and automation.

It is therefore better to choose a suitable name from the start than to plan a migration later simply because the first bucket was named test123.

Practical Example

Assume we are using:

   Profile: storage
Endpoint: https://s3.example-provider.com
Bucket: my-bucket

Create the bucket:

   aws s3 mb s3://my-bucket \
--endpoint-url https://s3.example-provider.com \
--profile storage

Next, check the list of available buckets:

   aws s3 ls \
--endpoint-url https://s3.example-provider.com \
--profile storage

Then check the contents of the new bucket:

   aws s3 ls s3://my-bucket \
--endpoint-url https://s3.example-provider.com \
--profile storage

Empty output is normal here: the bucket exists, but it does not contain any objects yet.

You can also verify the bucket itself using the S3 API:

   aws s3api head-bucket \
--bucket my-bucket \
--endpoint-url https://s3.example-provider.com \
--profile storage

If the command completes without an error, the bucket exists and the account has access to it.

Command Summary

The storage is now ready to receive data. The next step is to upload the first object, download it again, and see how the AWS CLI handles individual files and entire directories.

Uploading and Downloading Files

At this point, working with the AWS CLI starts to feel much like ordinary file operations: you can use cp, specify a source and destination, and upload individual objects or entire directories.

However, S3 works differently from a conventional file system, particularly when it comes to object keys.

How an object key replaces a conventional file path

Suppose we want to upload a file: report.pdf

to the bucket: my-bucket

and logically place it at: docs/2026/report.pdf

In the AWS CLI, this would look like: s3://my-bucket/docs/2026/report.pdf

However, it is important to remember that docs/2026/report.pdf is not a path in the conventional sense, but an object key.

In other words, the storage system does not need to create the following separately:

docs/

2026/

as actual directories.

It simply stores the object with the key: docs/2026/report.pdf

The AWS CLI or the provider’s web interface can then display these prefixes as folders.

This is particularly convenient because the structure can be defined at upload time.

For example, an object can be stored as backups/postgresql/2026-09-15.sql.gz, even if no backups or postgresql folders existed before.

What Happens When an Object Is Uploaded

When an upload command is executed, the AWS CLI reads the local file and sends it to the S3 API.

The storage service receives:

• the object content;
• the destination bucket;
• the key;
• metadata;
• request parameters.

The object then becomes accessible by its key.

For example, the local file: ./report.pdf can be uploaded as: s3://my-bucket/reports/report.pdf

The local file name and the key do not have to match at all.

For example: ./final-report-v7.pdf can be saved as: s3://my-bucket/reports/report.pdf

In other words, when uploading an object, we effectively assign it a new name in S3.

This is useful in automation workflows, where a temporary local file may have a technical name while being assigned a suitable permanent key in Object Storage.

What Happens When You Upload an Object with the Same Key

This is another important difference between S3 and a “network drive.”

If s3://my-bucket/report.pdf already exists in the bucket and we upload another object with the same key, the old object will usually be replaced with the new content.

In other words, each key must be unique within a bucket.

If versioning is not enabled for the bucket, the previous version usually cannot be retrieved through standard methods after it is overwritten.

If versioning is supported and enabled, previous versions can be retained separately.

Therefore, the command aws s3 cp report.pdf s3://my-bucket/report.pdf … may not simply “upload” a file, but may actually overwrite an existing object.

This is especially important for backup scenarios.

For example, if a backup is saved every day under the same key, backup.sql.gz, only the latest version may remain in the bucket.

To retain a history, it is better to use unique keys:

backups/2026-09-15.sql.gz

backups/2026-09-16.sql.gz

backups/2026-09-17.sql.gz

Alternatively, enable versioning if the provider supports it.

How Uploading a Single File Differs from Recursive Uploads

For a single file, the process is straightforward: the source and destination are specified explicitly.

For example:

file.txt

→ s3://my-bucket/file.txt

However, you often need to upload an entire directory.

Suppose you have:

Full Control Over Your Kubernetes Infrastructure

Deploy production-ready Kubernetes with full control over nodes, storage, and networking on predictable-cost EU infrastructure with direct access to your resources.

Get

When you use recursive copying, the AWS CLI traverses the directory and creates the corresponding object keys.

For example:

index.html

css/style.css

images/logo.png

may become:

s3://my-bucket/site/index.html

s3://my-bucket/site/css/style.css

s3://my-bucket/site/images/logo.png

It is important to understand that cp --recursive simply copies all selected objects.

It does not fully compare the source and destination or try to bring them into the same state.

For that, there is a separate command—sync—which we will cover in the next chapter.

Practical Examples

Assume the following:

   Bucket: my-bucket
Endpoint: https://s3.example-provider.com
Profile: storage

Upload a single file:

   aws s3 cp ./report.pdf s3://my-bucket/report.pdf \
--endpoint-url https://s3.example-provider.com \
--profile storage

Upload it under a different key:

   aws s3 cp ./report.pdf s3://my-bucket/reports/2026/report.pdf \
--endpoint-url https://s3.example-provider.com \
--profile storage

Download the object:

   aws s3 cp s3://my-bucket/report.pdf ./downloaded-report.pdf \
--endpoint-url https://s3.example-provider.com \
--profile storage

List the bucket contents:

   aws s3 ls s3://my-bucket \
--endpoint-url https://s3.example-provider.com \
--profile storage

To upload an entire directory:

   aws s3 cp ./data s3://my-bucket/data/ \
--recursive \
--endpoint-url https://s3.example-provider.com \
--profile storage

Next, as usual, we have a handy collection of commands.

Command Quick Reference

We can now work with both individual files and entire directories.

Synchronizing Directories with aws s3 sync

A standard upload using cp works well when you need to transfer a specific file or simply upload an entire directory.

However, if the local directory and bucket already exist and change regularly, copying everything each time is inconvenient.

For these tasks, AWS CLI provides the following command: aws s3 sync

It compares the source and destination and attempts to transfer only the files that actually need to be updated.

This is why sync is often used for:

• Backups;
• Static websites;
• Archives;
• Uploading logs;
• Synchronizing working directories.

Next, let’s move on to the comparison.

How sync differs from cp –recursive

At first glance, the commands seem similar.

For example, aws s3 cp ./data s3://my-bucket/data/ --recursive and aws s3 sync ./data s3://my-bucket/data/ can both upload an entire directory to S3.

However, they work differently.

cp --recursive works more like a standard copy operation: it traverses the specified directory and copies the objects.

sync attempts to make the source and destination more closely match by comparing their contents and skipping anything it considers up to date.

This is especially apparent when the command is run again.

Suppose we have:

After the first synchronization, all three files were uploaded to the bucket.

We then modified only index.html.

The next time sync runs, the AWS CLI will typically not upload the entire directory again. It will transfer only the modified object.

For large datasets, this saves:

• Time;
• Network traffic;
• Operations;
• Sometimes money as well, if the provider charges for requests or outbound traffic.

Moving on.

How the AWS CLI Identifies Modified Files

There is an important caveat here.

The aws s3 sync command does not perform a full byte-by-byte comparison of every local file with every object before each operation.

That would be too expensive.

The CLI uses metadata—primarily size and modification time—and its exact behavior depends on the direction of synchronization and the options used.

Therefore, sync is best viewed not as a version control system or a cryptographic check that the files are identical, but as a practical synchronization mechanism.

For example, if the local file report.pdf has not changed in size or timestamp, AWS CLI may determine that it does not need to be uploaded again.

This is usually sufficient for typical backup and deployment scenarios.

However, if guaranteed integrity verification is required, additional mechanisms are needed, such as checksums, versioning, or custom application logic.

There is another useful point to note.

When synchronizing from S3 to a local system, the AWS CLI also compares the available metadata and downloads only what it considers new or modified.

This is why the same command works in both directions.

What –delete Does

By default, sync adds and updates the required objects but does not delete extraneous data from the destination.

Suppose the local directory contained:

We synchronized the directory with S3.

Then we deleted old.js locally and ran the regular command again: aws s3 sync ./site s3://my-bucket/site/

The site/old.js object may remain in the bucket.

This behavior is intentional: a regular sync should not delete data unless explicitly instructed to do so.

However, if you add --delete:

The AWS CLI will delete files or objects from the destination if they no longer exist in the source.

In other words: aws s3 sync ./site s3://my-bucket/site/ --delete essentially means: make the destination contents match the source as closely as possible, including deleting extraneous files.

This is very useful for mirroring.

However, it is also significantly more dangerous.

Why You Should Use –dryrun Before –delete

When using the --delete flag, it is best to first review what the command will do.

For this, use --dryrun.

It displays the planned operations without executing them.

For example:

   aws s3 sync ./site s3://my-bucket/site/ \
--delete \
--dryrun

The AWS CLI may produce output like this:

   (dryrun) upload: ./index.html to s3://my-bucket/site/index.html
(dryrun) delete: s3://my-bucket/site/old.js

You can then safely check:

• Whether the correct bucket is selected;
• Whether the correct prefix is specified;
• Whether anything important will be deleted;
• Whether the source and destination have been reversed.

This is especially important in automation.

A mistake such as: s3://my-bucket/ instead of: s3://my-bucket/site/ when combined with --delete, can affect far more objects than expected.

Therefore, a good rule for production workflows is to run sync --dryrun first, followed by the actual sync.

Practical examples

Local directory → S3:

   aws s3 sync ./data s3://my-bucket/data/ \
--endpoint-url https://s3.example-provider.com \
--profile storage

S3 → local directory:

   aws s3 sync s3://my-bucket/data/ ./data \
--endpoint-url https://s3.example-provider.com \
--profile storage

Preview the changes:

   aws s3 sync ./data s3://my-bucket/data/ \
--dryrun \
--endpoint-url https://s3.example-provider.com \
--profile storage

Check what would happen if extra objects were deleted:

   aws s3 sync ./data s3://my-bucket/data/ \
--delete \
--dryrun \
--endpoint-url https://s3.example-provider.com \
--profile storage

Only after checking the changes should you run the actual synchronization:

   aws s3 sync ./data s3://my-bucket/data/ \
--delete \
--endpoint-url https://s3.example-provider.com \
--profile storage

Next, another small table of useful commands.

Command Summary Table

AWS CLI can now do more than copy individual files—it can keep entire directories up to date.

The next logical step is to explore multiple profiles, since AWS CLI can work with several S3-compatible storage services and different sets of access keys at the same time.

Working with multiple storage services using profiles

So far, we have used a single profile: storage

This is sufficient for a single S3-compatible storage service.

In practice, however, a second set of credentials soon appears, followed by a third—and before long, the same AWS CLI needs to work with different providers, regions, and accounts.

For example:

• default
• backup-storage
• archive-storage
• production-storage
• aws

And this is where profiles help ensure you don’t mix up which keys are used and exactly where the command is sent.

Why Use a Separate Profile for Each Storage Service

Suppose we have two S3-compatible services.

The first is used for backups: https://s3.backup-provider.example

The second is used for media: https://s3.media-provider.example

Each has:

• Its own Access Key;
• Its own Secret Key;
• Possibly its own region;
• Its own endpoint;
• Separate permissions.

If everything is stored under the default profile, you have to keep changing its settings.

This is both inconvenient and risky.

It makes much more sense to create two profiles: backup-storage and media-storage

The command itself then becomes much clearer: aws s3 ls --profile backup-storage or aws s3 ls --profile media-storage

In other words, the profile serves as an explicit execution context.

This is especially useful for commands that modify data.

For example, the difference between: aws s3 sync ./backup s3://prod-backups/ --delete and the same command with the following explicitly specified: --profile backup-storage can be quite significant.

The more storage services you use, the more useful it is to avoid implicit behavior.

How the AWS CLI Selects Credentials

The AWS CLI can obtain credentials from multiple sources.

It is important to understand that these sources have an order of precedence.

In simplified terms, the CLI looks for credentials in roughly the following order:

• Explicitly specified parameters and environment variables;
• The selected profile;
• The standard default profile;
• Other available credential mechanisms, if configured.

For routine profile management, we are particularly interested in: ~/.aws/credentials

For example:

   [backup-storage]
aws_access_key_id = BACKUP_ACCESS_KEY
aws_secret_access_key = BACKUP_SECRET_KEY
[media-storage]
aws_access_key_id = MEDIA_ACCESS_KEY
aws_secret_access_key = MEDIA_SECRET_KEY

And in ~/.aws/config you can store the related settings:

   [profile backup-storage]
region = us-east-1
output = json
[profile media-storage]
region = eu-1
output = json

Now, when running aws s3 ls --profile backup-storage , AWS CLI knows which set of credentials to use.

You can check exactly where a profile gets its values by running: aws configure list --profile backup-storage

This is useful when the CLI suddenly appears to be using the wrong keys.

Sometimes the cause is not the credentials file at all, but an environment variable with higher precedence.

When to Use –profile and When to Use Environment Variables

For manual operations, --profile is usually very convenient.

For example:

   aws s3 ls \
--endpoint-url https://s3.example-provider.com \
--profile storage

The advantage is clear: the command itself shows which profile is being used.

This is particularly useful:

• In the terminal;
• When administering multiple storage systems;
• In articles and documentation;
• During manual troubleshooting.

However, profiles are not always suitable for automation.

For example, a CI/CD container may not have a persistent ~/.aws/credentials file at all.

In that case, credentials are often provided through environment variables:

AWS_ACCESS_KEY_ID

AWS_SECRET_ACCESS_KEY

AWS_DEFAULT_REGION

AWS CLI picks them up automatically.

This is convenient for:

• CI/CD;
• Temporary environments;
• Containers;
• systemd services;
• Secrets from Vault or a secret manager.

The choice therefore depends on the use case.

For users: --profile is usually more convenient.

For automated processes, environment variables or a dedicated secret store may be safer and more flexible.

However, there is an important caveat: environment variables can easily be made global within the session by mistake.

For example, if you run:

   export AWS_ACCESS_KEY_ID=...
export AWS_SECRET_ACCESS_KEY=...

and then forget about it, the next command may use those values even when you expect it to use a different profile.

That is why, if you encounter unexpected behavior, you should check: env | grep ^AWS_

If old credentials are still set in the environment, they may affect the result.

Why You Should Not Mix Keys from Different Providers

S3-compatible APIs look alike, which can give the impression that their keys are interchangeable as well.

But they are not.

Access Keys and Secret Keys are created by a specific service and are valid only within that service’s authentication system.

For example, credentials from Provider A should not work with Provider B.

Even if both support AWS S3 and Signature Version 4.

If you accidentally use: Provider B endpoint, with Provider A credentials, you may get InvalidAccessKeyId or another authorization error.

It is even worse when two profiles have similar names and you have to remember which one is for production.

It is therefore better to name profiles according to their purpose instead of using names such as:

• test1
• test2
• new
• storage2

The following names are much clearer:

• prod-backups
• stage-backups
• media-storage
• archive-storage

The same principle applies to bucket names: choosing good names today saves time six months from now.

Command Quick Reference

Profiles are particularly useful when a single AWS CLI installation is used as a general-purpose client for multiple S3-compatible services.

Managing Access to Buckets and Objects

Up to this point, we have mainly discussed how to connect to the storage and how to send commands to it.

Now it is time to address another important question: if an Access Key works, what exactly is it allowed to do?

This is where a separate layer of logic comes into play.

Having a valid Access Key and Secret Key pair does not automatically mean that the user can:

• Read all buckets;
• Upload files anywhere;
• Delete objects;
• Modify policies;
• Make data public.

Credentials verify identity, while access rights determine permitted actions.

Why an Access Key Does Not Grant Full Access

Suppose we have created a dedicated service user for backups.

This user needs only a few permissions:

• List the contents of a specific bucket;
• Upload a new backup;
• Download a backup during recovery.

However, the user does not need to delete the entire bucket, modify its policy, or access other users’ storage.

Therefore, a well-designed access model does not look like this:

Access Key

→ full access to all of Object Storage

Instead, it looks more like this:

Access Key

→ specific user

→ specific policy

→ limited set of actions

This is why two different Access Keys can successfully authenticate against the same endpoint while having completely different permissions.

For example, one user may be able to perform GetObject but receive AccessDenied for DeleteObject.

This is not an authorization error.

On the contrary, authorization has worked correctly—the specific action is simply not permitted.

How Credentials, ACLs, and Bucket Policies Differ

These concepts are easy to confuse because they all relate to access.

However, they operate at different levels.

Credentials

Credentials are authentication details:

• Access Key;
• Secret Key.

They answer the question: Who is sending the request?

In other words, credentials identify a user or service account.

ACL

ACL — Access Control List.

This is an older mechanism for managing access to a bucket or individual objects.

For example, ACLs can be used to grant permissions to specific users or make an object publicly readable.

However, ACL support may be limited or incomplete across different S3-compatible providers.

Some modern S3 use cases favor policies and disable ACLs as the primary mechanism for managing permissions.

Bucket policy

A bucket policy is a policy that applies to a specific bucket.

It allows you to specify:

• Who is granted access;
• Which resources they can access;
• Which actions are allowed;
• Under what conditions.

High-Performance Virtual Machines

Deploy in seconds with full root access, reserved resources, built-in high availability, and predictable monthly pricing.

Get

A policy is much more flexible than a simple allow/deny rule.

For example, you can allow a service account to:

• List only my-backups;
• Upload objects;
• Read objects;
• But not delete them.

Alternatively, you can allow public read access only for a specific prefix.

The distinction can be summarized in the following table:

Now let’s move on to the next chapter.

A public bucket and a public object are not the same thing

This is another important distinction.

When someone says that a bucket is public, they may mean very different things.

For example, a bucket may allow anonymous access to read objects while preventing users from listing all of its objects.

That is, the user will be able to access https://.../images/logo.png, if they know the exact URL, but they will not be able to request a listing of the bucket’s entire contents.

Conversely, access can be configured so that a specific user has ListBucket permission but not GetObject permission.

This is because listing objects and reading a specific object are separate API actions in S3.

Public access should therefore not be viewed as a single “everything is open / everything is closed” switch.

This is especially important with an S3-compatible provider: the control panel may simplify the settings while using its own implementation of policies or ACLs under the hood.

Principle of Least Privilege

One of the fundamental security principles here is least privilege, or the principle of granting only the minimum necessary permissions.

A service should be granted only the permissions it actually needs to perform its task.

For example, a backup script may only need:

1. ListBucket
2. PutObject
3. GetObject

But DeleteObject doesn’t need to be granted at all.

This is particularly useful for protecting against two risks:

• Errors in the script;
• Compromised credentials.

Suppose the script is accidentally run with an incorrect --delete option.

If the account does not have the DeleteObject permission at all, the storage service will simply reject the deletion.

The policy therefore acts as an additional safeguard.

This is why granting every service key full access “so it doesn’t get in the way” is convenient but poor practice.

Access features that may vary across S3-compatible providers

Here again, S3-compatible does not mean complete parity with AWS.

Providers may offer different levels of support for:

• bucket policies;
• object ACLs;
• public access;
• versioning;
• Object Lock;
• lifecycle rules;
• IAM-like users;
• separate service accounts;
• IP-based restrictions;
• temporary credentials;
• presigned URLs.

One provider may allow you to create a complex JSON policy much like one in AWS.

Another may only let you select from a few predefined roles in the control panel:

1. Read Only
2. Read/Write
3. Full Access

A third provider may only allow access management at the project level.

Therefore, before migrating a complex policy from Amazon S3 to a third-party Object Storage service, it is best to first check the documentation for that specific service.

Basic permissions such as read and write are generally highly compatible.

Advanced IAM mechanisms, however, are not always compatible.

Core S3 Actions

Below are several permissions commonly used in access policies:

ListBucket applies to the bucket itself, while GetObject, PutObject, and DeleteObject apply to the objects within it.

Therefore, policies often need to define bucket-level and object-level resources separately.

Conceptually, this might look like the following:

Bucket:

arn:…:my-bucket

Objects:

arn:…:my-bucket/*

The exact format depends on the provider and its implementation of S3 policies.

In practice, the safest approach is as follows:

1. Create a dedicated user or service account.
2. Grant access only to the required bucket.
3. Allow only the required operations.
4. Test each action using the AWS CLI.
5. Only then use the credentials for automation.

This makes it much easier to determine why a particular command returns AccessDenied and is significantly safer than granting full access from the outset.

Why the AWS CLI Returns AccessDenied or SignatureDoesNotMatch

Once the AWS CLI is installed, the profile is configured, and the endpoint is known, it may seem that everything should work automatically.

In practice, however, this is the stage where errors such as AccessDenied, InvalidAccessKeyId, or SignatureDoesNotMatch most commonly occur.

It is important not to treat them as the same issue. They occur at different stages of request processing and therefore indicate different causes.

In simplified terms, the process works as follows: the AWS CLI builds the request, signs it, and sends it to the endpoint. The server then validates the credentials, signature, and user permissions.

Incorrect Access Key or Secret Key

If an incorrect Access Key is specified, the server may not recognize the account at all. This often results in an InvalidAccessKeyId error.

The causes are usually straightforward: a typo, an outdated key, a deleted service account, or a profile from another S3 provider.

The situation is slightly different with the Secret Key. It is used to generate the signature, so an incorrect Secret Key is more likely to result in a SignatureDoesNotMatch error.

The AWS CLI cannot validate the Secret Key in advance. It simply uses the stored value to calculate the signature and sends the result to the server.

Therefore, if the Access Key appears to be correct but the signature does not match, it is worth double-checking the Secret Key.

Environment variables are another important consideration. Even if the ~/.aws/credentials file contains the correct values, active AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables can override them.

Endpoint error

An incorrect endpoint can manifest in different ways.

If the address is completely unreachable, the AWS CLI typically reports a connection problem, such as Could not connect to the endpoint URL.

If the endpoint exists but belongs to a different region or S3 service, the error may appear to be an authentication or signature issue.

For example, a user may accidentally use one provider’s credentials with another provider’s endpoint.

From the AWS CLI’s perspective, everything appears to be correct: the URL exists, the keys are loaded, and the request is signed. However, the server cannot associate the Access Key with its own system and rejects the request.

Therefore, the endpoint must always be checked together with the profile, not separately.

Incorrect Region

The region is part of SigV4, so its value directly affects the signature.

If the service expects one region but the CLI uses another, the server may receive a correctly formed request but calculate a different signature.

This results in a SignatureDoesNotMatch error.

Third-party S3-compatible providers may have specific requirements. Some use a familiar value such as us-east-1 regardless of the storage system’s physical location, while others require a provider-specific region name.

Therefore, it is best to use the region specified in the provider’s documentation rather than infer it from the data center’s geographic location.

To check the current value, use aws configure list --profile storage.

System Clock Skew

SigV4 uses the request timestamp.

If the client system clock is significantly ahead or behind, the server may consider the request invalid.

This is intentional: the timestamp helps limit the replay of previously signed requests.

A significant time difference may result in RequestTimeTooSkewed or other signature- and date-related errors.

This issue is rare on a VPS because the system time is usually synchronized automatically, but it should not be ruled out entirely.

You can check the system clock status using the timedatectl command. If NTP is active and System clock synchronized shows yes, the cause most likely lies elsewhere.

The user’s policy does not allow the required action

AccessDenied usually indicates insufficient permissions rather than an issue with credentials.

For example, a user may have permission to perform GetObject but not PutObject. In that case, downloading an object will work, but uploading one will be denied.

The same applies to ListBucket, DeleteObject, and other actions.

Another possible scenario is that a user has access to a specific bucket but does not have permission to list all buckets in the account.

In that case, aws s3 ls will return AccessDenied, while aws s3 ls s3://my-bucket will succeed with the same profile.

Therefore, when troubleshooting, it is important to consider not only the denial itself but also the specific operation that the AWS CLI attempted to perform.

If the error occurs with only one command while the others work, the issue is almost certainly related to the policy.

Path-style and virtual-hosted-style requests

S3 supports two primary bucket addressing styles.

With path-style requests, the bucket name appears in the path:

https://s3.example-provider.com/my-bucket/object.txt

With virtual-hosted-style requests, it becomes part of the hostname:

https://my-bucket.s3.example-provider.com/object.txt

Virtual-hosted-style addressing has long been the primary option for Amazon S3.

However, support may vary among S3-compatible providers.

For example, a service may work correctly only with path-style addressing or, conversely, require virtual-hosted-style addressing.

The resulting issues can often seem unusual: the endpoint responds and the credentials are correct, but some requests fail with signature, DNS, or bucket access errors.

This is because the request string and host are included in the SigV4 signature calculation. If the client and server construct the bucket address differently, the signature may not match.

In AWS CLI, you can force path-style addressing through the configuration:

aws configure set s3.addressing_style path --profile storage

To restore automatic selection, use the value auto.

This setting is especially useful with older or partially compatible S3 services.

Error Table

The easiest way to diagnose these errors is to proceed methodically: first check the endpoint and connection, then the profile and credentials, followed by the region and system time, and only then investigate the policy and bucket addressing style.

If the checks are performed in exactly this order, most common issues with S3-compatible storage can be identified fairly quickly.

Now that we know not only how to run the commands but also how to identify the causes of errors, we can perform a final end-to-end check: create or select a bucket, upload a test file, download it again, run sync, and delete the test object.

Testing the Entire Storage Setup

Start by listing the available buckets:

   aws s3 ls \
--endpoint-url https://s3.example-provider.com \
--profile storage

If the required bucket already exists, you can use it. Otherwise, create a test bucket:

   aws s3 mb s3://my-bucket \
--endpoint-url https://s3.example-provider.com \
--profile storage

Now create a small local file:

echo "S3 test file" > test.txt

Upload it to Object Storage:

   aws s3 cp ./test.txt s3://my-bucket/test.txt \
--endpoint-url https://s3.example-provider.com \
--profile storage

After the upload, verify that the object appears in the bucket:

   aws s3 ls s3://my-bucket/ \
--endpoint-url https://s3.example-provider.com \
--profile storage

The output should now include test.txt along with its size and modification time.

Next, test the reverse operation by downloading the object under a different local file name:

   aws s3 cp s3://my-bucket/test.txt ./downloaded-test.txt \
--endpoint-url https://s3.example-provider.com \
--profile storage

If desired, compare the contents of the two files:

diff test.txt downloaded-test.txt

If the command produces no output, the files are identical.

Next, test sync. Create a small directory:

   mkdir -p sync-test
echo "First file" > sync-test/first.txt
echo "Second file" > sync-test/second.txt

Sync it to a separate prefix within the bucket:

   aws s3 sync ./sync-test s3://my-bucket/sync-test/ \
--endpoint-url https://s3.example-provider.com \
--profile storage

You can then verify the contents by listing them recursively:

   aws s3 ls s3://my-bucket/sync-test/ \
--recursive \
--endpoint-url https://s3.example-provider.com \
--profile storage

If both files are listed, sync is also working correctly.

The final step is to verify permission to delete objects. Delete the test object created earlier:

   aws s3 rm s3://my-bucket/test.txt \
--endpoint-url https://s3.example-provider.com \
--profile storage

Then list the contents again:

   aws s3 ls s3://my-bucket/ \
--endpoint-url https://s3.example-provider.com \
--profile storage

If test.txt is no longer listed, the full workflow has completed successfully: AWS CLI can connect to a custom endpoint, list bucket contents, upload and download objects, synchronize directories, and delete data.

After testing, you can also delete the test objects under sync-test/:

   aws s3 rm s3://my-bucket/sync-test/ \
--recursive \
--endpoint-url https://s3.example-provider.com \
--profile storage

If the bucket was created solely for this test and is no longer needed, you can delete it after removing its contents using the following command:

   aws s3 rb s3://my-bucket \
--endpoint-url https://s3.example-provider.com \
--profile storage

Of course, there is no need to delete a production bucket for testing purposes.

Once this test is complete, the basic AWS CLI configuration can be considered finished. Security remains to be addressed.

How to Use S3 Credentials Securely in Production

Why You Should Not Store Keys in Scripts

The most obvious, yet still common, anti-pattern is hardcoding the Access Key and Secret Key directly in a shell script.

For example, avoid doing this:

   export AWS_ACCESS_KEY_ID="ACCESS_KEY"
export AWS_SECRET_ACCESS_KEY="SECRET_KEY"
aws s3 sync ./backup s3://my-bucket/

Technically, this will work, but the secrets will be stored in a plain-text file.

That file could then end up:

• in Git;
• in a project archive;
• in a backup;
• in CI logs;
• in someone else’s account;
• in a ticket or chat message during debugging.

Another concern is shell history. If secrets are entered directly in terminal commands, they may be saved in the current user’s command history.

Therefore, it is best to design the production workflow so that the application or script retrieves credentials from a separate secure sourcerather than embedding them in the code.

Depending on the infrastructure, the following options may be used:

• ~/.aws/credentials with appropriate access permissions;
• environment variables passed to the process;
• CI/CD secret storage;
• Vault or another secrets manager;
• a built-in mechanism for temporary credentials, if supported by the provider.

However, using environment variables does not automatically make a system secure. It is important to consider who can read the process environment, how secrets are passed, and whether they end up in logs.

Restricting permissions for a specific service account

In production, it is best to avoid using a single shared key with full access to all of Object Storage.

It is much safer to create a dedicated service account for a specific task.

For example, a backup service may only need ListBucket, PutObject, and GetObject permissions for a single bucket.

If backup deletion is handled by a separate lifecycle policy, the account does not need to be granted DeleteObject permission at all.

This provides several benefits.

First, an accidental error in a script cannot result in actions beyond those permitted.

Second, if the Secret Key is compromised, an attacker will gain only a limited set of capabilities rather than full access to the entire account.

Third, it becomes easier to understand the purpose of each key. Instead of one general-purpose user, you can have separate accounts for:

• backups;
• media uploads;
• CI/CD;
• archiving;
• reading reports.

Each service account should have its own credentials and only the minimum permissions required.

When to Rotate Keys

There is no need to change the Secret Key every Monday simply for the sake of rotation.

What matters far more is being able to replace it quickly when a genuine risk arises.

A key must be revoked immediately and replaced with a new one if it:

• Was committed to Git;
• Was sent in a chat or ticket;
• Appeared in logs;
• Was stored on a compromised server;
• Became accessible to an employee or contractor who no longer needs access.

Periodic rotation is also useful, especially for long-lived production credentials.

A sound rotation process works as follows: first, create a new key; switch the applications over to it; verify connectivity; and only then disable the old key.

This reduces the risk of accidentally stopping a backup or another service by deleting credentials prematurely.

This approach is particularly convenient if the provider supports multiple active keys for a single service account.

Why Object Storage Alone Does Not Eliminate the Need for Backups

Another common mistake is assuming that a file becomes a backup simply because it is stored in Object Storage.

Object Storage does improve storage reliability, but it does not protect against every data loss scenario.

For example, if a script mistakenly deletes an object and its credentials have the DeleteObject permission, the deletion may be propagated to the remote storage.

The same applies to:

• Accidental overwrites;
• Corrupted source data;
• Compromised credentials;
• Administrator error;
• Incorrect use of sync –delete.

A reliable setup therefore typically uses additional safeguards.

If the provider supports versioning, previous versions of objects can be retained after they are overwritten.

Lifecycle rules can automatically move older versions to a different storage class or delete them after a specified period.

Object Lock can protect data from modification and deletion for a specified period.

For especially critical data, it is advisable to maintain an independent copy in another account, region, or even with another provider.

In other words, Object Storage is an excellent platform for backups, but it does not by itself guarantee that a backup cannot be lost.

It is useful to distinguish between two concepts:

• A remote copy of the data;
• A comprehensive backup strategy.

The latter typically includes version history, retention periods, access controls, and regular recovery testing.

Conclusion

S3-compatible Object Storage lets you work with storage services from different providers through the familiar S3 API using the same AWS CLI.

All you need to connect is the endpoint, region, and credentials. However, this simple setup involves several important considerations: SigV4 signing, profiles, access permissions, addressing style, and the specifics of each service’s S3 implementation.

This article covered the entire core workflow: configuring the AWS CLI, creating a dedicated profile, connecting to a custom endpoint, creating a bucket, uploading and downloading objects, synchronizing directories, and troubleshooting common authentication errors.

For production use, follow a few simple rules: do not store the Secret Key in code, use dedicated service accounts, grant only the minimum required permissions, and plan credential rotation in advance.

If Object Storage is used for backups, you should also configure versioning, retention, or another form of protection against accidental overwrites and deletions.

With these measures in place, the AWS CLI becomes more than just a convenient utility for manually uploading files—it becomes a full-fledged tool for backups, automation, CI/CD, and working with large collections of objects.

FAQ

Can AWS CLI be used with S3 storage that is not provided by Amazon?

Yes. If the provider supports an S3-compatible API, you can point AWS CLI to the provider’s endpoint using --endpoint-url. However, specific capabilities—such as policies, ACLs, versioning, Object Lock, and other features—may differ from those available in Amazon S3.

Why does aws s3 ls return AccessDenied even though access to a specific bucket works?

Listing all buckets requires a separate permission. A user may have access to a specific bucket and its objects without having permission to retrieve the full list of buckets in the account.

Therefore, AccessDenied in this case does not mean that the credentials are configured incorrectly.

What happens if I upload a file again using the same object key?

The new object will replace the existing object under the same key. If versioning is not enabled, the previous version will generally no longer be accessible. For backups, it is therefore advisable to use unique keys or enable versioning if the provider supports it.

Can I use a single AWS CLI profile for multiple S3 providers?

Technically, you can keep changing the endpoint and credentials, but this quickly leads to confusion. It is more reliable to create a separate profile for each provider or purpose, such as prod-backups, archive-storage, and media-storage.

Why is sync –delete considered dangerous?

The --delete option deletes objects from the destination if they do not exist in the source. If you specify the wrong bucket, prefix, or synchronization direction, you may delete data that you need.

Before performing this operation, it is advisable to run the same command with --dryrun and review the planned changes.

What is the difference between aws s3 and aws s3api?

aws s3 provides high-level commands such as cp, sync, ls, and rm, making it convenient for everyday file operations.

aws s3api maps more closely to individual S3 API operations and provides more granular control. It is useful, for example, when working with bucket settings, policies, and other features not available through high-level commands.

Do I need separate backups if my data is already stored in Object Storage?

Yes, if the data is truly important. Object Storage alone does not protect against deletion performed with valid credentials, accidental overwrites, or an incorrectly configured sync --delete operation.

For critical data, use versioning, Object Lock, or an independent backup, depending on the capabilities of the provider.

Sources

1. AWS Documentation — Configuration and credential file settings in the AWS CLI
2. Amazon S3 Documentation — Authenticating Requests (AWS Signature Version 4)
3. AWS Documentation — AWS CLI s3 sync command reference
4. Amazon S3 Documentation — Required permissions for Amazon S3 API operations