Synchronizing AWS S3 Buckets Across Accounts

Synchronizing S3 buckets is an easy task thanks to the aws s3 sync command. When synchronizing buckets across accounts, there are a few extra steps you may need to take in order to allow a user from another account to perform the sync operation.

IAM

Whichever user will be performing the sync operation will need the following permissions set for their user on IAM:

s3:PutObjectTagging
s3:PutObject
s3:PutObjectAcl
s3:GetObjectTagging
s3:GetObject
s3:ListBucket

It is easy to forget to set these as you may be more concerned with setting the bucket policy settings for your user, but if the user is missing IAM permissions it does not matter if the bucket policy gives them required permissions; they will not be able to perform the sync. PutObjectAcl permission is required particularly to allow other users to view the objects that you are creating in the destination account.

Bucket Policies

You will need to make sure that you have the following permissions set on your source bucket for the user that will be performing the sync:

{
   "Sid": "syncBucket",
   "Effect": "Allow",
   "Principal": {
       "AWS": "arn:aws:iam::555555555555:user/graham.reid"
   },
   "Action": "s3:ListBucket",
   "Resource": "arn:aws:s3:::source-bucket-name"
},
{
    "Sid": "sync",
    "Effect": "Allow",
    "Principal": {
        "AWS": "arn:aws:iam::555555555555:user/graham.reid"
    },
    "Action": "s3:GetObjectTagging",
    "Resource": "arn:aws:s3:::source-bucket-name/*"
}

On your destination bucket, you will need to set these permissions:

{
   "Sid": "syncBucket",
   "Effect": "Allow",
   "Principal": {
       "AWS": "arn:aws:iam::555555555555:user/graham.reid"
   },
   "Action": "s3:ListBucket",
   "Resource": "arn:aws:s3:::destination-bucket"
},
{
   "Sid": "syncObject",
   "Effect": "Allow",
   "Principal": {
        "AWS": "arn:aws:iam::555555555555:user/graham.reid"
   },
   "Action": [
       "s3:PutObject",
       "s3:PutObjectAcl",
       "s3:GetObject",
       "s3:GetObjectAcl",
       "s3:DeleteObject"
   ],
   "Resource": [
      "arn:aws:s3:::destination-bucket/*",
      "arn:aws:s3:::destination-bucket"
   ]
}

Running the Sync

Once your IAM user’s AWS key and secret have been set in your environment, you can run the sync command to copy files from the source bucket to the destination bucket:

aws s3 sync s3://source-bucket s3://detination-bucket --source-region us-east-2 --region us-east-1 --acl public-read

This command will recurse through folders within the bucket to copy files over from the source bucket to the destination bucket. Using --acl public-read allows other users to view the files after they have been copy. This is important to include when copying files from one AWS account to another because otherwise none of the users in the account will be able to view or alter the copied objects. This is also an important step if the bucket is public because if the objects’ ACLs aren’t public-read, the objects will not actually be publicly accessible despite the bucket being public.

By default, the sync will only copy over objects that don’t already exist in the destination bucket or that have been recently updated on the source bucket. It will not remove objects that have been removed in the source bucket but are still in the destination bucket. Do do this, add --delete to the end of the command.