Get S3 object securely using Curl and Openssl with SIGV4 Blog Devoteam Rebirth

Temps de lecture : 6 minutes

As I was working on a project for a client, I encountered a challenging situation. I needed to retrieve S3 objects from EC2 instances that were built from an AMI from the AWS Marketplace and deployed in private subnets with no access to the internet. The S3 service is reachable via S3 Gateway endpoints deployed in these private subnets. However, the instance did not have the AWS CLI installed, and the version of Curl that was installed did not have the option to generate Signature v4 automatically to authenticate with the AWS API (–aws-sigv4). This posed a problem for me, as I needed to find a way to authenticate with the AWS API in order to retrieve S3 objects with only Curl and Openssl as tools.

AWS API authentication process using HTTP with SIGV4

To authenticate with AWS API using HTTP, we can use the AWS Signature Version 4 process, known also as SIGV4. This process is detailed in the official AWS documentation here. The main steps of the process are the following:

1) First, we need to create a Canonical Request which involves creating a string that includes the HTTP method, the URL, the query string parameters, and the headers of the request.

GET
/
Action=DescribeInstances&Version=2016-11-15
content-type:application/x-www-form-urlencoded; charset=utf-8
host:ec2.amazonaws.com
x-amz-date:20220830T123600Z

host;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

Example of a Canonical Request to execute the DescribeInstances Action on AWS EC2 API (Source: AWS Official Documentation)

2) Then generate a String to Sign by creating a string that specify the hash algorithm used, the region, the date and time of the request and the hash of the Canonical Request we created in step 1

AWS4-HMAC-SHA256
20220830T123600Z
20220830/us-east-1/ec2/aws4_request
f536975d06c0309214f805bb90ccff089219ecd68b2577efef23edd43b7e1a59

Example of a String to Sign (Source: AWS Official Documentation)

3) Compute the Signature with HMAC-SHA256 (a way of using a cryptographic hash function, SHA-256) and our AWS Secret Access Key to hash the String to Sign that we created in step 2, and then encoding the result in base64.

4) And finally, add the Signature to the HTTP Request in the ‘Authorization’ header. If the Signature has been correctly computed, we should be successfully authenticated to the AWS API

However in my case, I was following these steps from an EC2 instance with an instance profile associated. And as we know an IAM Instance Profile does not provide long-term credentials but temporary ones. And in that case the calculation of the signature is done by taking these temporary credentials.

I found that there was a lack of information on how to form an HTTP request to interact with the AWS S3 API using a Signature Version 4 generated with temporary credentials, and especially to download an object from S3. That is why I decided to write this article to share how to deal with this problem with a bash script.

Retrieve IAM Credentials for an instance profile from instance metadata

First we retrieve IAM Credentials from instance metadata as they are requirements to compute the Signature. Especially the Session Token because in this case we are working with temporary credentials.

# Retrieve IAM credentials from EC2 instance metadata
metadata_ip='169.254.169.254'
INSTANCE_PROFILE=$(curl -s http://$metadata_ip/latest/meta-data/iam/security-credentials/)
METADATA=$(curl -s http://$metadata_ip/latest/meta-data/iam/security-credentials/$INSTANCE_PROFILE)
ACCESS_KEY_ID=$(echo "$METADATA" | grep AccessKeyId | sed -e 's/  "AccessKeyId" : "//' -e 's/",$//')
SECRET_ACCESS_KEY=$(echo "$METADATA" | grep SecretAccessKey | sed -e 's/  "SecretAccessKey" : "//' -e 's/",$//')
SESSION_TOKEN=$(echo "$METADATA" | grep Token | sed -e 's/  "Token" : "//' -e 's/",$//')

Form the Canonical Request

To build the Canonical Request we need several pieces of information like the HTTP method, the URL, the query string parameters, and the headers of the request. In the case to get a S3 object:

HTTP Method will be GET
The URL will be the path of the object on the S3 Bucket
The payload of the request will be empty in that case so we leave an empty line.
Host field will be on the following format: <bucket_name>.s3.amazonaws.com
Header “x-amz-content-sha256” is required for S3 API and as it is the SHA256 hash of the payload, it will be the hash of an empty string.
Header “x-amz-date” should be the timestamp of the request in ISO8601 format
Header “x-amz-security-token” must be defined in this case as we are working with temporary credentials
Signed headers which is the list of headers defined previously separated by semicolons
The hash of the payload which is in this case an empty string hash

Finally we store the completed Canonical Request in a temporary file canonical_request.tmp

# Retrieve AWS Region name where the instance is launched
REGION=$(curl -s http://169.254.169.254/latest/meta-data/placement/availability-zone | sed s/.$//)
AWS_SERVICE="s3"

HTTP_METHOD="GET"
CANONICAL_URI="/$2"

#Retrieve current date in ISO8601 format
DATE_AND_TIME=$(date -u +"%Y%m%dT%H%M%SZ")
DATE=$(date -u +"%Y%m%d")

#Compute payload SHA256 hash, which is an empty string
EMPTY_STRING_HASH=$(echo -n | openssl dgst -sha256 |cut -d ' ' -f 2)

# Store Canonical request
/bin/cat >./canonical_request.tmp <<EOF
$HTTP_METHOD
$CANONICAL_URI

host:$BUCKET_NAME.s3.amazonaws.com
x-amz-content-sha256:$EMPTY_STRING_HASH
x-amz-date:$DATE_AND_TIME
x-amz-security-token:$SESSION_TOKEN

host;x-amz-content-sha256;x-amz-date;x-amz-security-token
$EMPTY_STRING_HASH
EOF

#Remove trailing newline
printf %s "$(cat canonical_request.tmp)" > canonical_request.tmp

Compute the signing key

To compute the signing key that will be used to generate the Signature, we need to perform these fours steps

Concatenate “AWS4” string with the secret access key and generate a SHA256 hash with the Secret Access Key as the key and the Date as the data. We store the result in a variable called DATE_KEY Key. To simplify the calculation, I define a function that generate a SHA256 hash with Key and Data as inputs

# Function to generate sha256 hash
function hmac_sha256 {
  KEY="$1"
  DATA="$2"
  echo -n "$DATA" | openssl dgst -sha256 -mac HMAC -macopt "$KEY" | sed 's/^.* //'
}

DATE_KEY=$(hmac_sha256 key:"AWS4$SECRET_ACCESS_KEY" $DATE)

Generate a SHA256 hash with the DATE_KEY as the key and the Region name as the Data. The result is stored in a variable called DATE_REGION_KEY

DATE_REGION_KEY=$(hmac_sha256 hexkey:$DATE_KEY $REGION)

Generate a SHA256 hash with the DATE_REGION_KEY as the key and the Service name as the Data (here S3). The result is stored in a variable called DATE_REGION_SERVICE_KEY

AWS_SERVICE="s3"
DATE_REGION_SERVICE_KEY=$(hmac_sha256 hexkey:$DATE_REGION_KEY $AWS_SERVICE)

Generate a SHA256 hash with the DATE_REGION_SERVICE_KEY as the key and the string “aws4_request” as the Data. The result is stored in a variable called HEX_KEY that is the Signing Key that will be used to generate the Signature.

HEX_KEY=$(hamc_sha256 hexkey:$DATE_REGION_SERVICE_KEY "aws4_request")

Compute the Signature

In order to compute the Signature we first need to form the String to Sign that specifies the hash algorithm used, the region, the date and time of the request and the hash of the Canonical Request we generated earlier. The String to Sign is stored in a temporary file called string_to_sign.tmp

# Generate canonical request hash
CANONICAL_REQUEST_HASH=$(openssl dgst -sha256 ./canonical_request.tmp | awk -F ' ' '{print $2}')

# Store String to Sign
/bin/cat >./string_to_sign.tmp <<EOF
AWS4-HMAC-SHA256
$DATE_AND_TIME
$DATE/$REGION/$AWS_SERVICE/aws4_request
$CANONICAL_REQUEST_HASH
EOF

printf %s "$(cat string_to_sign.tmp)" > string_to_sign.tmp

We have now the String to Sign and the Signing Key, we can compute the Signature

# Generate signature
SIGNATURE=$(openssl dgst -sha256 -mac HMAC -macopt hexkey:$HEX_KEY string_to_sign.tmp | awk -F ' ' '{print $2})

Make the complete HTTP Request

Now that we have the signature generated, we can perform the HTTP request with the appropriate headers to download an S3 object securely without using the AWS CLI tool !

curl -s https://$BUCKET_NAME.s3.amazonaws.com/$CANONICAL_URI \
  -X $HTTP_METHOD \
  -H "Authorization: AWS4-HMAC-SHA256 \           Credential=$ACCESS_KEY_ID/$DATE/$REGION/$AWS_SERVICE/aws4_request, \      SignedHeaders=host;x-amz-content-sha256;x-amz-date;x-amz-security-token, \
Signature=$SIGNATURE" \
-H "x-amz-content-sha256: $EMPTY_STRING_HASH" \
-H "x-amz-date: $DATE_AND_TIME" \
-H "x-amz-security-token: $SESSION_TOKEN" \
-o "$OUTPUT"

The complete script is available on GitHub here, the prerequisites are the following:

An operating system with a Bash interpreter
The curl program to send HTTP requests
The openssl program to generate hashes and signatures
Access to the EC2 instance metadata interface (http://169.254.169.254/latest/meta-data/)
Valid IAM account credentials and a session token, accessible via the EC2 instance metadata interface.
The necessary permissions to access an object in an S3 bucket, including the name of the bucket and the path to the object.