Using AWS secrets manager for secrets

[craigp318]

Thought I would share in case others are looking for different options for secret management.

I was somewhat reluctant to store my TWS_USERID and TWS_PASSWORD in a .env file, write them down in my dockerfile, or pass them in the command line (all these workflows are discouraged by docker for secret management). Even using docker secrets is a bit strange to me since it is effectively just a flat file stored on docker.

An alternative is to put the secrets in AWS secrets manager and give ec2 (or ecs or ssm) access to the ARN. Then, load the secrets into the environment at runtime only.

Locally, I added a script below, and I source this fn immediately before sourcing common.sh.
Works like a charm if you are already using AWS.

Open to suggestions if anyone has any!
Figured I'd throw this out there in case it's helpful to others.

###############################################################################
#####		Get Secrets
###############################################################################

if [ "$TRADING_MODE" == "live" ]; then
  SECRET_ID="prod/my_secret_name"
else
  SECRET_ID="qa/my_secret_name"
fi

secret_value=$(aws secretsmanager get-secret-value --secret-id $SECRET_ID --query SecretString --output text)

# Check if the secret value is fetched successfully
if [ $? -eq 0 ]; then
  echo "Secret fetched successfully."

  # Parse and export specific keys only
  export TWS_USERID=$(echo $secret_value | jq -r '.TWS_USERID')
  export TWS_PASSWORD=$(echo $secret_value | jq -r '.TWS_PASSWORD')

  # echo "TWS_USERID=$(echo $secret_value | jq -r '.TWS_USERID')"
  # echo "TWS_PASSWORD=$(echo $secret_value | jq -r '.TWS_PASSWORD')"

  # Confirm the environment variables are set
  echo "Environment variables set from AWS Secrets Manager"

else
  echo "Failed to fetch secret value."
fi

echo "Done getting secrets"

[gnzsnz]

Thanks for sharing.

I think that would require to install jq and aws in the image as well.

Probably we could add a directory to run scripts at start-up, by default this directory will be empty, but it can be mapped to the outside with a volume, so any script there can be executed at start-up. It will require to install sudo so any dependency can be installed

Something like this could do the work

/home/ibgateway
                /start-up
                            /start-scripts.d      # before anything else
                           /x-scripts.d           # once X environment is running
                           /ibc-scripts.d         # once IBC is running

at each point if there is a script i those directories it's executed.

This will add flexibility for any secret manager and more cases.

[craigp318]

Yes that's true - I forgot to mention that I added these lines into the Dockerfile underneath "prepare system"

  apt-get update && \
  apt-get install -y python3 python3-pip curl unzip jq && \
  curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip" && \
  unzip awscliv2.zip && \
  ./aws/install && \
  rm -rf awscliv2.zip aws && \

I like that idea for generically letting user specify startup scripts