Messing around with AWS for LTTKGP!

After hours of minimal success at trying to setup LTTKGP on AWS, I've decided to start over from scratch, with a tiny app and blog my experiments.

Warning: This is my first time using AWS beyond just plain old EC2 and S3 and I apologise for any mistakes. Also, this might get really long. you've been warned :)

Prelude

Last summer, I started working on LTTKGP with a bunch of awesome people. LTTKGP fetches songs posted on the FB Group Listen To This KGP, collects metadata, organizes them into feeds, and has a pretty cool webplayer that lets user play these collections. The project is open source and code can be found on the GitHub org.

LTTKGP has a couple of components, and we use AWS to deploy 2 major services

The other major service is Falcon, the beautiful UI that brings it all together. Falcon is deployed on Netlify and we use Cloudflare to manage DNS for the entire project

Existing AWS Infrastructure

To deploy LTTKGP, Naresh came up with a pretty cool architecture (and daunting looking :P) which we've been using to deploy, for a couple of months now

The present architecture

A top level overview is as follows:

(When this was set up, all I did was stare at the screen in constant awe!)

The major issue with the present infra is that it's pretty expensive mainly because of the NAT Gateway and using Fargate to manage instances, instead of self managing EC2 instances.

Mirroring Existing Infra

Since we could make do without private subnets, by having tighter rules in our security groups, I attempted to replicate the present infra on my personal AWS account and mess around with it till I can get it working again.

The first thing I did was exactly replicate the infra we were using. It took me a couple hours but I set it up and can be accessed at:

Experiments!

I found a couple of cool Medium blogs (One, Two) that descibe how to deploy to ECS using self-hosted EC2 instances but I wasn't able to get C-3PO up and running. I've now decided to start from scratch, with a simple Flask app

Starting from Scratch - Part 1

The first step is creating a new VPC for our use.

The new VPC

I'll create 2 new subnets inside the VPC. To use a elastic load balancer, we'll need atleast 2 subnets in different availability zones:

I also create a route table, with both of these subnets and a internet gateway that is attached to our VPC. Next, I'll setup a new flask application, dockerize it and create an ECS cluster.

NOTE: In your routing table, add a route from 0.0.0.0/0 to the above configured internet gateway. I didn't and it took me some time to figure it out

Creating our test Flask app

To begin, I'll use a simple hello world app. I'll keep iterating if I'm able to successfully deploy this first

Here's the repository: https://github.com/mukul-mehta/Flask-ECS-Adventures

Step 1: Pushing the image to ECR

I've created a new private repository on AWS ECR called flask-ecs, to which I'll push my image. Once I'm done logging in, I build, tag and then push the image to the repository

Pushing image to ECR

Step 2: Create an empty ECS cluster

This should be simple hopefully. Go to ECS, create a new cluster and select template as EC2 Linux + Networking, give it a name and choose empty cluster on the next page

Step 3: Create a launch template for EC2 instances

This is where it started to get tricky for me. Do I need to setup a launch template and a autoscaling group, or can I do with just creating an instance for now :confused:

Let's try creating a launch template for now.

Since we need an instance for ECS, we'll choose a community AMI that it optimized for ECS. The latest one can be found here. The AMI comes with a 30GB root volume by default, and I don't think I'll add more storage for now. I don't specify an instance type in the launch template either. To add this instance to our ECS cluster, I add the following in user data (Advanced -> User data)

#!/bin/bash
echo ECS_CLUSTER=your_cluster_name >> /etc/ecs/ecs.config

I'll create just one instance for now. a t2.micro using the above launch template. I select the subnet as helloworld-public-1 and use the default security group for now. I also need to create a new IAM Role for the instance, as mentioned in this guide. The new IAM role has the follwing permissions:

IAM Role for ECS

Now, we have a t2.micro EC2 instance, running an Amazon Linux AMI that is optimized for ECS, with our custom IAM role and user data. We should now this instance attached to our ECS cluster :crossed_fingers:

IT DIDN'T WORK :neutral_face:

Okay, gotta do this all over again. I need to create an EC2 instance and make sure that it's attached to the ECS cluster. Why is this so hard :disappointed:

AWS's documentation says that the correct IAM role to use is called ecsInstanceRole. There isn't a profile with that name on my dashboard, let me create one using the instructions mentioned here.

Okay, this was my fault. Turns out I forgot to add a route in my routing table for this VPC. In order to use ECS, the instances need to have a route to reach the ECS service endpoints. For this, I need to add a route in my routing table, that routes to my internet gateway. Doing this fixed the problem and I can now see the EC2 instance in my ECS cluster

This is how the routing table should look

Step 4: Create a new service in the ECS cluster

4.1: Create a new task definition

Before this, I need to setup a task definition. The task definition is a recipe for the service and specifies the image and hardware requirements for the task.

In the task definition, I specify the URI of the container image (Which I pushed to ECR), a memory limit of 512MB and 512 CPU Units (Each CPU core is 1024 CPU units). I also need to add a port mapping, from 8000 in host to 8000 in the container, since my flask app is listening on port 8000 inside the container.

Once we're done with creating with task definition, I need to create a service using this task definition

4.2: Create a new load balancer

Before I can create a new service, I need to setup an Application Load Balancer and a Target Group. For now, I'll setup a listener on port 80 instead of configuring a new certificate and setting up a HTTPS listener.

Setting up the ALB should be fairly straightforward. Add a listener on port 80, for HTTP traffic, select the right VPC and subnets and the default security group. Then, setup a new target group that is listening on port 80 and protocol set to HTTP1. On our simple flask app, I setup a health check endpoint /health that on success returns a status code of 200. We need a health check endpoint since the target group needs to know when a target is down. It sends requests to this health check endpoint every couple (5 AFAIK) minutes and if there are 2 consecutive failures (status code != 200), it marks a target as unhealthy. I don't register any targets right now, since we'll do that when we setup our service

Now that the ALB is setup, I'll go back to creating a new service

4.3: Creating a service

Since we're using self-managed EC2 instances, the type of service is EC2. I set the number of tasks to 1. Next, we add the above configured ALB to our service and add the container as a target. For now, I use a very simple autoscaling policy, with minimum tasks set to 1, desired tasks set to 1 and maximum number of tasks as 2, which runs when average CPU utilization for our task crosses 65% I'm sure this won't trigger given our very simple app but nevertheless, we're done!

Step 5: Yaaaay, we have a basic app up and running!

Now that we have our service created, it should've started a task (Which is a running instance of that service). I see a running task :)

ECS Clusters Dashboard

(As expected, CPU utilization is extremely low)

Since our ALB is internet facing, we should be able to access it directly and it should work and it does! Here's the link: http://experiments-helloworld-alb-1556701883.ap-south-1.elb.amazonaws.com

To finish setting up our basic app, I'll just provision a new certificate via Certificate Manager in order to use HTTPS. This would involve adding a new CNAME record. Next, I add a listener to the ALB, listening on 443 and forwarding to our target group. Once this is done, I'll create a CNAME record for hello.metamehta.in pointing to the ALB.

Cool, so we now have our flask application running at https://hello.metamehta.in

Woohoo! So we're done with one milestone! The next job is setting up LTTKGP, instead of our dummy application. We still need to configure a ton of stuff, security groups for example, since for now our default security group allows traffic on all ports which seems like a horrible thing to do

Comments


about · posts · t.i.l. · resume · home · github · twitter · rss