Protect your deployment pipeline with MFA

Two factor authentication

In this post, I’d like to share how you can secure your deployment pipeline using multiple accounts, profiles, and multi-factor authentication (or MFA). I’ll specifically be talking about Amazon Web Services (AWS) in combination with aws-vault. Let me start off by giving you some context.

For a product I’ve recently worked on, we have different stages of deployment. ‘Development’ for doing our development in (you guessed it) and ‘production’ for the actual production side of things. We don’t really mind the dev environment to break, but we’d strongly prefer to keep prod (as we call it) up and running all the time.

For both profiles, we’ve set up completely different accounts in AWS. For maximum protection, we opted for MFA to secure our accounts. It’s a fairly sane industry practice. Amazon allows “virtual MFA devices” to be used for its MFA, which means you could use Google Authenticator (which is probably installed by default on your Android phone). In addition, we’ve started using aws-vault, which can securely store the Access Key and Secret Access Key you’ll get from Amazon. No need to store your keys in plain text format any more.

Let’s first install aws-vault. You can either simply download the relevant package for your system from their releases page on GitHub (https://github.com/99designs/aws-vault/releases), or install it using Go:

go get github.com/99designs/aws-vault

If you download and install it yourself, copy the executable to a ‘bin’ folder that’s in your PATH and ensure it’s executable by running chmod +x aws-vault inside the directory where you installed it. You may need to rename it when copying. Test your installation by running aws-vault:

sander@sander-Aspire-S7:~$ aws-vault
usage: aws-vault [<flags>] <command> [<args> ...]
A vault for securely storing and accessing AWS credentials in development
environments.
Flags:
--help Show context-sensitive help (also try --help-long and
--help-man).
--version Show application version.
--debug Show debugging output
--backend=file Secret backend to use [file kwallet secret-service]
--prompt=terminal Prompt driver to use [terminal]
Commands:
help [<command>...]
Show help.
add [<flags>] <profile>
Adds credentials, prompts if none provided
list
List profiles
rotate [<flags>] <profile>
Rotates credentials
exec [<flags>] <profile> [<cmd>] [<args>...]
Executes a command with AWS credentials in the environment
remove [<flags>] <profile>
Removes credentials, including sessions
login [<flags>] <profile>
Generate a login link for the AWS Console
server
Run an ec2 instance role server locally

(The latest version of aws-vault at the time of writing is 3.7.1, which is what I am using.)

Next, you need to set up a user in IAM who has the permissions to perform all sorts of operations through CloudFormation. For the sake of clarity, I’ll settle for a user with Administrator Access. In AWS, click on Services and select IAM. (You can sort AWS services alphabetically, which can be a tremendous help for finding them.) Go to Users and click on Add User:

In the next screen, enter a username and select “Programmatic Access”, then click ‘Next’.

You can now select the permissions for the new user. As indicated earlier, we will give this user Administrator Access. Click on “Attach existing policies directly” and from the list select AdministratorAccess:

After clicking next, you can review what you’ve done so far. If you’re happy, simple click ‘Create User’.

You’ve reached the last step of the process, and this is the only opportunity you have to download the new user’s credentials. (If you ever lose those credentials, you cannot retrieve them. However, you can always create new credentials.) We will use these credentials to create a new profile in aws-vault.

We will create a profile called ‘sls-administrator’, which is also the user’s name in IAM.

$ aws-vault add sls-administrator
Enter Access Key ID: AKIAIBCXXXXXX253JEJA
Enter Secret Access Key: U3kE/XXXXxxxxXXXXy/aeLEIJKktDd6ZR+Ce9
Enter passphrase to unlock /home/sander/.awsvault/keys:
Added credentials to profile "sls-administrator" in vault

(The keys have been changed to protect the innocent.)

We have now securely stored the user’s Access Key and Secret Access Key in the vault. All that’s left is to set up MFA. In IAM, click Close if you haven’t yet, and you’ll be sent back to the User section. Click on the newly created ‘sls-administrator’ user, then select Security Credentials.

As you can see, no MFA device has been assigned yet. Click on the pencil icon next to “Assigned MFA device … No”. A popup window will be shown:

For this tutorial, we’ll assume you’ll be using your smartphone, so select “A virtual MFA device” and click ‘Next Step’. You will now be shown a QR code that you must scan with you authenticator app (like Google Authenticator). After scanning the QR code, you should enter two consecutive authentication codes. Click ‘Authenticate’ and you’re good to go.

As a final step, we need to tell aws-vault that we’ll be using MFA. From the security credentials view, copy the ARN that is now shown next to “Assigned MFA Device”. Using your favourite editor, open the file ~/.aws/config. Enter a new profile, indicating the profile expects MFA for authentication purposes. The name should be same as the profile name you selected earlier:

[profile sls-administrator]
mfa_serial = arn:aws:iam::xxx0417xxx80:mfa/sls-administrator

Don’t forget to save the file. (Sorry, couldn’t help myself.)

Now, if you want to deploy, for instance, a serverless service (I mean the Serverless Framework here), you’ll run this command:

$ aws-vault exec -- sls-administrator sls deploy -v -s prod

This command executes (exec) the command sls deploy -v -s prod using the profile sls-administrator. The flag -v means the deployment will be verbose, and the -s option defines the stage. It will ask you for the vault’s passphrase and, because you’ve set up MFA, for an authentication code. Retrieve this code from your phone and enter it.

And that’s it. You have now successfully set up MFA for your deployment pipeline (or at least for one stage). Simply repeat the above steps for different stage you may have, e.g., development, staging, production.