Aws cli

Installation

Installation for Ubuntu/Debian We use jq as a handy method to filter awscli json output

apt-get install jq
apt-get install awscli

Configure

Configure awscli tool first by running this.

aws configure

This will ask you for your access and secret key, region and output format (default: json)

Bucket is full even after deleting objects

It could possible be that your bucket has versioning enabled. To check this first run the following command

BUCKET_NAME=bucket-name

aws s3api \
      list-objects-versions \
      --bucket $BUCKET_NAME \
      | jq '.Versions \
             | map(select(.IsLatest == false)) \
             | length' \

How to delete s3 objects with deleted marker?

The script below gathers all objects with IsLatest == false and, please keep that in mind, VersionId != “null”

#!/bin/bash

BUCKET_NAME="bucket name"

versions=$(aws s3api list-object-versions --bucket $BUCKET_NAME | jq '.Versions | map(select(.IsLatest == false and .VersionId != "null"))')

for version in $(echo "$versions" | jq -r '.[].VersionId'); do
    echo "Deleting version: $version"
    aws s3api delete-object --bucket $BUCKET_NAME --key "$(echo "$versions" | jq -r --arg version "$version" '.[] | select(.VersionId == $version) | .Key')" --version-id $version
done

Get the total bucket size

Get the total bucket size in Gigabyte for active objects only

aws s3api list-objects
    --bucket $BUCKET_NAME
    --output json | \
    jq '[.Versions[].Size] | add / 1024 / 1024 / 1024 * 100 | round / 100'

Get the real bucket size by listing all objects versions, sum them up and scale them down to Gigabyte

aws s3api list-object-versions
    --bucket $BUCKET_NAME
    --output json | \
    jq '[.Versions[].Size] | add / 1024 / 1024 / 1024 * 100 | round / 100'