AWS Velocity Series: EC2 based app CI/CD pipeline

In the previous article, you learned how to use CloudFormation to describe a production-ready infrastructure for an EC2 based app. In this article you will...

In the previous article, you learned how to use CloudFormation to describe a production-ready infrastructure for an EC2 based app. In this article you will learn to:

Automate the creation of an AMI that contains the app with Packer
Deploy a CloudFormation stack based infrastructure/ec2.yml with AWS CodePipeline
Run the Acceptance tests on AWS CodeBuild against the infrastructure created in the previous step
Deploy another CloudFormation stack for the production environment

You can follow step by step or get the full source code here: https://github.com/widdix/aws-velocity

The pipeline is based on the CI/CD Pipeline as Code part of this series.

AWS Velocity Series

Most of our clients use AWS to reduce time-to-market following an agile approach. But AWS is only one part of the solution. In this article series, I show you how we help our clients to improve velocity: the time from idea to production. Discover all posts!

Additionally, the EC2 based app pipeline contains:

add a BuildAMI action to the Build stage.
add an Acceptance stage
add a Production stage

Copy the deploy/pipeline.yml file to deploy/pipeline_ec2.yml to get the starting point right. If you don’t have the deploy/pipeline.yml file you can download it from https://github.com/widdix/aws-velocity.

BuildAMI Action

EC2 instances start from an image (AMI) that contains the operating system including all the files that are needed. You can create your own AMI as well. Usually, you take one of the available AMIs like the Amazon Linux, make your modifications, and then create a new image from that. This whole procedure can also be automated with a tool called Packer. You will now see how you can run Packer in CodeBuild to create a new AMI that contains the app.

Packer itself needs configuration files. Create a file infrastructure/packer.json with the following content that build a new AMI based on ami-c51e3eb6 (Amazon Linux) and a Bash script that you will create later:

{
  "variables": {
    "ami_name": "{{env `CODEBUILD_BUILD_ID`}}"
  },
  "builders": [{
    "type": "amazon-ebs",
    "region": "eu-west-1",
    "source_ami": "ami-c51e3eb6",
    "instance_type": "t2.micro",
    "ssh_username": "ec2-user",
    "ami_name": "{{user `ami_name` | clean_ami_name}}",
    "ami_regions": ["REGION"]
  }],
  "provisioners": [{
    "type": "file",
    "source": "app",
    "destination": "/tmp"
  }, {
    "type": "shell",
    "script": "infrastructure/packer.sh"
  }]
}

Packer is configured to run a Bash script to provision the AMI and to upload the app folder. Create a file infrastructure/packer.sh with the following content to:

install the latest patches
install Node.js 6.x
install the CloudWatch Logs agent
install forever, a tool to run Node.js script in the background

infrastructure/packer.shGitHub

#!/bin/bash -ex
# this script runs as ec2-user

sudo yum -y update
curl --silent --location https://rpm.nodesource.com/setup_6.x | sudo bash -
sudo yum -y install nodejs awslogs
sudo npm install -g forever@0.15.3
sudo mv /tmp/app /opt

The script also moves the application files to the right place.

To integrate Packer into the pipeline, add the following resources to the Resources section of deploy/pipeline_ec2.yml to create a CodeBuild project to run Packer with the above configuration. Packer also needs a bunch of IAM permissions which are also added.

deploy/pipeline_ec2.ymlGitHub

# Packer needs a set of access rights as defined in https://www.packer.io/docs/builders/amazon.html
AMICodeBuildRole:
  DependsOn: CloudFormationRole # make sure that CloudFormationRole is deleted last
  Type: 'AWS::IAM::Role'
  Properties:
    AssumeRolePolicyDocument:
      Version: '2012-10-17'
      Statement:
      - Effect: Allow
        Principal:
          Service:
          - 'codebuild.amazonaws.com'
        Action:
        - 'sts:AssumeRole'
    Policies:
    - PolicyName: ServiceRole
      PolicyDocument:
        Version: '2012-10-17'
        Statement:
        - Sid: CloudWatchLogsPolicy
          Effect: Allow
          Action: 
          - 'logs:CreateLogGroup'
          - 'logs:CreateLogStream'
          - 'logs:PutLogEvents'
          Resource: '*'
        - Sid: CodeCommitPolicy
          Effect: Allow
          Action: 'codecommit:GitPull'
          Resource: '*'
        - Sid: S3GetObjectPolicy
          Effect: Allow
          Action: 
          - 's3:GetObject'
          - 's3:GetObjectVersion'
          Resource: '*'
        - Sid: S3PutObjectPolicy
          Effect: 'Allow'
          Action: 's3:PutObject'
          Resource: '*'
    - PolicyName: EC2
      PolicyDocument:
        Version: '2012-10-17'
        Statement:
        - Sid: CloudFormation
          Effect: Allow
          Action: 
          - 'ec2:AttachVolume'
          - 'ec2:AuthorizeSecurityGroupIngress'
          - 'ec2:CopyImage'
          - 'ec2:CreateImage'
          - 'ec2:CreateKeypair'
          - 'ec2:CreateSecurityGroup'
          - 'ec2:CreateSnapshot'
          - 'ec2:CreateTags'
          - 'ec2:CreateVolume'
          - 'ec2:DeleteKeypair'
          - 'ec2:DeleteSecurityGroup'
          - 'ec2:DeleteSnapshot'
          - 'ec2:DeleteVolume'
          - 'ec2:DeregisterImage'
          - 'ec2:DescribeImageAttribute'
          - 'ec2:DescribeImages'
          - 'ec2:DescribeInstances'
          - 'ec2:DescribeRegions'
          - 'ec2:DescribeSecurityGroups'
          - 'ec2:DescribeSnapshots'
          - 'ec2:DescribeSubnets'
          - 'ec2:DescribeTags'
          - 'ec2:DescribeVolumes'
          - 'ec2:DetachVolume'
          - 'ec2:GetPasswordData'
          - 'ec2:ModifyImageAttribute'
          - 'ec2:ModifyInstanceAttribute'
          - 'ec2:ModifySnapshotAttribute'
          - 'ec2:RegisterImage'
          - 'ec2:RunInstances'
          - 'ec2:StopInstances'
          - 'ec2:TerminateInstances'
          Resource: '*'
# This IAM User is only temporarily necessary until https://github.com/mitchellh/packer/pull/4613 is fixed!
AMICodeBuildUser:
  DependsOn: CloudFormationRole # make sure that CloudFormationRole is deleted last
  Type: 'AWS::IAM::User'
  Properties:
    Policies:
    - PolicyName: Packer
      PolicyDocument:
        Version: '2012-10-17'
        Statement:
        - Sid: EC2
          Effect: Allow
          Action:
          - 'ec2:AttachVolume'
          - 'ec2:AuthorizeSecurityGroupIngress'
          - 'ec2:CopyImage'
          - 'ec2:CreateImage'
          - 'ec2:CreateKeypair'
          - 'ec2:CreateSecurityGroup'
          - 'ec2:CreateSnapshot'
          - 'ec2:CreateTags'
          - 'ec2:CreateVolume'
          - 'ec2:DeleteKeypair'
          - 'ec2:DeleteSecurityGroup'
          - 'ec2:DeleteSnapshot'
          - 'ec2:DeleteVolume'
          - 'ec2:DeregisterImage'
          - 'ec2:DescribeImageAttribute'
          - 'ec2:DescribeImages'
          - 'ec2:DescribeInstances'
          - 'ec2:DescribeRegions'
          - 'ec2:DescribeSecurityGroups'
          - 'ec2:DescribeSnapshots'
          - 'ec2:DescribeSubnets'
          - 'ec2:DescribeTags'
          - 'ec2:DescribeVolumes'
          - 'ec2:DetachVolume'
          - 'ec2:GetPasswordData'
          - 'ec2:ModifyImageAttribute'
          - 'ec2:ModifyInstanceAttribute'
          - 'ec2:ModifySnapshotAttribute'
          - 'ec2:RegisterImage'
          - 'ec2:RunInstances'
          - 'ec2:StopInstances'
          - 'ec2:TerminateInstances'
          Resource: '*'
# This IAM Access Key is only temporarily necessary until https://github.com/mitchellh/packer/pull/4613 is fixed!
AMICodeBuildUserAccessKey:
  DependsOn: CloudFormationRole # make sure that CloudFormationRole is deleted last
  Type: 'AWS::IAM::AccessKey'
  Properties:
    UserName: !Ref AMICodeBuildUser
AMIProject:
  DependsOn: CloudFormationRole # make sure that CloudFormationRole is deleted last
  Type: 'AWS::CodeBuild::Project'
  Properties:
    Artifacts:
      Type: CODEPIPELINE
    Environment:
      ComputeType: 'BUILD_GENERAL1_SMALL'
      Image: 'amazonlinux:2016.09'
      Type: 'LINUX_CONTAINER'
      EnvironmentVariables: # pass in the AWS credentials as environment variables is only temporarily necessary until https://github.com/mitchellh/packer/pull/4613 is fixed!
      - Name: 'AWS_ACCESS_KEY_ID'
        Value: !Ref AMICodeBuildUserAccessKey
      - Name: 'AWS_SECRET_ACCESS_KEY'
        Value: !GetAtt 'AMICodeBuildUserAccessKey.SecretAccessKey'
    Name: !Sub '${AWS::StackName}-ami'
    ServiceRole: !GetAtt 'AMICodeBuildRole.Arn'
    Source:
      Type: CODEPIPELINE
      BuildSpec: !Sub |
        version: 0.1
        phases:
          install: # install Packer
            commands:
            - 'yum -y install unzip'
            - 'curl -s -m 60 -o /opt/packer.zip https://releases.hashicorp.com/packer/0.12.3/packer_0.12.3_linux_amd64.zip'
            - 'unzip /opt/packer.zip -d /opt'
          pre_build: # replace the REGION placeholder with the stack's region
            commands:
            - 'sed -i "s:REGION:${AWS::Region}:g" infrastructure/packer.json'
          build:
            commands:
            # run packer
            - '/opt/packer -machine-readable build infrastructure/packer.json | tee infrastructure/packer.txt'
            # extract the AMI id into a JSON file
            - 'echo "{" > infrastructure/ami.json'
            - 'cat infrastructure/packer.txt | grep '',amazon-ebs,artifact,0,id,'' | awk -F'','' ''{print $6}'' | sed ''s/%!(PACKER_COMMA)/''

  ×We are moving to a new platformCertificationTest is being migrated. Some pages may be slow or briefly unavailable; everything will be back to normal within a few hours. If you run into any issue, email support@certificationtest.net.


'n/g'' | awk -F'':'' ''{print ""image": ""$2"","}'' | sed ''$ s/.$//'' >> infrastructure/ami.json'
            - 'echo "}" >> infrastructure/ami.json'
        artifacts:
          files:
          - 'infrastructure/packer.txt'
          - 'infrastructure/ami.json'
    TimeoutInMinutes: 10

You also need to make a small modification to the exiting BuildSpec in the AppProject resource to include the packer files into the App artifact. This is a hack because CodeBuild does not support multiple input artifacts at the moment. Change the artifacts section to:

deploy/pipeline_ec2.ymlGitHub

artifacts:
  files:
  - 'app/**/*'
  - 'infrastructure/packer.json' # this is a hack because we can not pass multiple Arifacts as an input to CodeBuild at the moment
  - 'infrastructure/packer.sh' # this is a hack because we can not pass multiple Arifacts as an input to CodeBuild at the moment

Now the CodeBuild project needs to be called in the pipeline, therefore change the Pipeline resource in the file deploy/pipeline_ec2.yml and add a new build action:

deploy/pipeline_ec2.ymlGitHub

Pipeline:
  Type: 'AWS::CodePipeline::Pipeline'
  Properties:
    # [...]
      - Name: BuildAndTestAcceptance
        ActionTypeId:
          Category: Build
          Owner: AWS
          Provider: CodeBuild
          Version: 1
        Configuration:
          ProjectName: !Ref AcceptanceProject
        InputArtifacts:
        - Name: Source
        OutputArtifacts:
        - Name: Acceptance
        RunOrder: 1
      # NEW STUFF!
      - Name: BuildAMI
        ActionTypeId:
          Category: Build
          Owner: AWS
          Provider: CodeBuild
          Version: 1
        Configuration:
          ProjectName: !Ref AMIProject
        InputArtifacts:
        - Name: App
        OutputArtifacts:
        - Name: AMI
        RunOrder: 2

Now, a new AMI is automatically created whenever the pipeline runs. The AMI will include the app and the latest patches.

It’s time to deploy the app to the acceptance stage and too see if the app works.

Acceptance stage

The acceptance stage consists of a CloudFormation stack based on infrastructure/ec2.yml and the execution of the acceptance tests. To create the CloudFormation stack, you first have to provide a few parameters. Create a file infrastructure/ec2.json with the following content:

infrastructure/ec2.jsonGitHub

{
  "Parameters": {
    "ImageId": {"Fn::GetParam": ["AMI", "infrastructure/ami.json", "image"]},
    "ParentVPCStack": "vpc-2azs",
    "ParentSSHBastionStack": "",
    "KeyName": "",
    "AdminEmail": "your@email.com"
  }
}

If you don’t have a VPC stack based of our Free Templates for AWS CloudFormation (https://github.com/widdix/aws-cf-templates/tree/master/vpc) create a VPC stack first. Make sure to change the ParentVPCStack parameter in the infrastructure/ec2.yml accordingly. Also change the value of the AdminEmail parameter. The other values can be stay as they are. Look at the ImageId parameter value. This is the way of getting a value out of a JSON artifact file in CoePipeline.

To run the acceptance tests, you also need another CodeBuild project, add the following resources to the Resources section of deploy/pipeline_ec2.yml:

deploy/pipeline_ec2.ymlGitHub

ExtendedCodeBuildRole:
  DependsOn: CloudFormationRole # make sure that CloudFormationRole is deleted last
  Type: 'AWS::IAM::Role'
  Properties:
    AssumeRolePolicyDocument:
      Version: '2012-10-17'
      Statement:
      - Effect: Allow
        Principal:
          Service:
          - 'codebuild.amazonaws.com'
        Action:
        - 'sts:AssumeRole'
    Policies:
    - PolicyName: ServiceRole
      PolicyDocument:
        Version: '2012-10-17'
        Statement:
        - Sid: CloudWatchLogsPolicy
          Effect: Allow
          Action: 
          - 'logs:CreateLogGroup'
          - 'logs:CreateLogStream'
          - 'logs:PutLogEvents'
          Resource: '*'
        - Sid: CodeCommitPolicy
          Effect: Allow
          Action: 'codecommit:GitPull'
          Resource: '*'
        - Sid: S3GetObjectPolicy
          Effect: Allow
          Action: 
          - 's3:GetObject'
          - 's3:GetObjectVersion'
          Resource: '*'
        - Sid: S3PutObjectPolicy
          Effect: 'Allow'
          Action: 's3:PutObject'
          Resource: '*'
    - PolicyName: CloudFormation
      PolicyDocument:
        Version: '2012-10-17'
        Statement:
        - Sid: CloudFormation
          Effect: Allow
          Action: 
          - 'cloudformation:DescribeStacks'
          Resource: !Sub 'arn:aws:cloudformation:${AWS::Region}:${AWS::AccountId}:stack/${AWS::StackName}-acceptance/*'
RunAcceptanceProject:
  DependsOn: CloudFormationRole # make sure that CloudFormationRole is deleted last
  Type: 'AWS::CodeBuild::Project'
  Properties:
    Artifacts:
      Type: CODEPIPELINE
    Environment:
      ComputeType: 'BUILD_GENERAL1_SMALL'
      Image: 'aws/codebuild/nodejs:6.3.1'
      Type: 'LINUX_CONTAINER'
    Name: !Sub '${AWS::StackName}-run-acceptance'
    ServiceRole: !GetAtt 'ExtendedCodeBuildRole.Arn'
    Source:
      Type: CODEPIPELINE
      BuildSpec: !Sub |
        version: 0.1
        phases:
          build: # execute acceptance tests against the acceptance stack
            commands:
            - 'cd acceptance/ && ENDPOINT=`aws cloudformation describe-stacks --stack-name ${AWS::StackName}-acceptance --query "Stacks[0].Outputs[?OutputKey==''URL''].OutputValue" --output text` ./node_modules/jasmine-node/bin/jasmine-node .'
    TimeoutInMinutes: 10

Now the CodeBuild project needs to be called in the pipeline, therefore change Pipeline resource in the file deploy/pipeline_ec2.yml to:

deploy the CloudFormation stack suffixed with -acceptance
run the acceptance tests

deploy/pipeline_ec2.ymlGitHub

Pipeline:
  Type: 'AWS::CodePipeline::Pipeline'
  Properties:
    # [...]
      - Name: BuildAMI
        ActionTypeId:
          Category: Build
          Owner: AWS
          Provider: CodeBuild
          Version: 1
        Configuration:
          ProjectName: !Ref AMIProject
        InputArtifacts:
        - Name: App
        OutputArtifacts:
        - Name: AMI
        RunOrder: 2
    # NEW STUFF!
    - Name: Acceptance
      Actions:
      - Name: Deploy
        ActionTypeId:
          Category: Deploy
          Owner: AWS
          Provider: CloudFormation
          Version: 1
        Configuration:
          ActionMode: CREATE_UPDATE
          Capabilities: CAPABILITY_IAM
          RoleArn: !GetAtt 'CloudFormationRole.Arn'
          StackName: !Sub '${AWS::StackName}-acceptance'
          TemplatePath: 'Source::infrastructure/ec2.yml'
          TemplateConfiguration: 'Source::infrastructure/ec2.json'
        InputArtifacts:
        - Name: Source
        - Name: AMI
        RunOrder: 1
      - Name: Test
        ActionTypeId:
          Category: Build
          Owner: AWS
          Provider: CodeBuild
          Version: 1
        Configuration:
          ProjectName: !Ref RunAcceptanceProject
        InputArtifacts:
        - Name: Acceptance
        RunOrder: 2

The acceptance stage is now ready.

Production stage

The production stage is pretty simple, just one CloudFormation stack. Change the Pipeline resource in the file deploy/pipeline_ec2.yml to add a new stage that looks familiar to the acceptance stage:

deploy/pipeline_ec2.ymlGitHub

Pipeline:
  Type: 'AWS::CodePipeline::Pipeline'
  Properties:
    # [...]
      - Name: Test
        ActionTypeId:
          Category: Build
          Owner: AWS
          Provider: CodeBuild
          Version: 1
        Configuration:
          ProjectName: !Ref RunAcceptanceProject
        InputArtifacts:
        - Name: Acceptance
        RunOrder: 2
    # NEW STUFF!  
    - Name: Production
      Actions:
      - Name: Deploy
        ActionTypeId:
          Category: Deploy
          Owner: AWS
          Provider: CloudFormation
          Version: 1
        Configuration:
          ActionMode: CREATE_UPDATE
          Capabilities: CAPABILITY_IAM
          RoleArn: !GetAtt 'CloudFormationRole.Arn'
          StackName: !Sub '${AWS::StackName}-production'
          TemplatePath: 'Source::infrastructure/ec2.yml'
          TemplateConfiguration: 'Source::infrastructure/ec2.json'
        InputArtifacts:
        - Name: Source
        - Name: AMI
        RunOrder: 1

Now the AMI containing the application is deployed to production with confidence and without disturbing the users. Try it and run the pipeline!

Summary

Let’s use my production-ready definition to summarize how each point is implemented:

Highly available: The load balancer (which is HA) sits in front of a fleet of EC2 instances managed by the Auto Scaling Group for maximum availability. In case of an unhealthy instance, the Auto Scaling Group will replace that instance.
Scalable: If the CPU utilization gets over 70%, a Cloud Watch Alarm triggers a Scaling Policy to add new instances automatically.
Frictionless deployment: To deploy a new version of the app, a new AMI is created. This AMI is then rolled out to the acceptance environment by updating the CloudFormation stack with the new ImageId parameter. CloudFormation and the Auto Scaling Group perform a rolling update to avoid the application being down during deployment. If the application can not be started the Rolling Update fails and CloudFormation rolls back.
Secure: During AMI creation, the latest patches are applied. You must ensure that the pipeline runs often enough to keep up with new patches. Besides that, Security Groups control network traffic to the EC2 instances. When following the bastion host approach, you get maximum security. The EC2 instance is only allowed to send logs to CloudWatch Logs by following the least privileges approach.
Operations: All logs are stored in CloudWatch Logs, important metrics are monitored and alarms are defined.

If you now have the impression that running an app on EC2 is a lot of work you are right. In the next two articles, you will learn about other options with fewer responsibilities.

Series

AWS Velocity Cover

Set the assembly line up
Local development environment
CI/CD Pipeline as Code
Running your application
EC2 based app
a. Infrastructure
b. CI/CD Pipeline (you are here)
Containerized ECS based app
a. Infrastructure
b. CI/CD Pipeline
Serverless app
Summary

You can find the source code on GitHub.