Deploying a Scalable Web Application on AWS

This detailed guide walks you through deploying a scalable web application using AWS services, including EC2, RDS, NGINX, Auto Scaling Groups, and Load Balancers. The example application will be a Flask app running on an EC2 instance, connected to an RDS database.

Table of Contents

1. Part 1: Setting Up the EC2 Instance and Flask Application
2. Part 2: Configuring NGINX as a Reverse Proxy
3. Part 3: Integrating AWS RDS for Database
4. Part 4: Adding Auto-Scaling with Load Balancer
5. Part 5: Monitoring and Scaling with CloudWatch

Part 1: Setting Up the EC2 Instance and Flask Application

Steps:

1. Create an EC2 Instance:
   • Use the AWS Management Console to launch a new EC2 instance.
   • AMI: Choose Ubuntu.
   • Instance Type: Select t2.micro (Free Tier eligible).
   • Configure Security Groups: Allow SSH (port 22) and HTTP (port 80).

1. Connect to the Instance:

   • Use VS Code to connect to the instance via SSH.
   • Update and upgrade the instance:

    sudo apt update && sudo apt upgrade -y
   

2. Install Python and Set Up Virtual Environment:

   • Install Python and pip:

    sudo apt install python3 python3-pip -y
   

• Install virtual environment:
  

   sudo apt install python3-venv -y
  

• Create and activate a virtual environment:
  

   python3 -m venv myenv
   source myenv/bin/activate
  

1. Install Flask and Dependencies:

   • Install Flask, Gunicorn, and psycopg2-binary:

    pip install flask gunicorn psycopg2-binary
   

• Verify the installations:
  

   pip list
  

1. Create the Flask Application:

   • Create a file named application.py:

    nano application.py
   

• Paste the following Flask code into application.py:
  

   from flask import Flask
   app = Flask(__name__)
  
   @app.route('/')
   def home():
       return "Hello, Flask on EC2!"
  
   if __name__ == "__main__":
       app.run(host='0.0.0.0', port=5000)
  

• Ensure the file has the correct permissions:
  

   chmod 644 application.py
  

1. Run Flask with Gunicorn:

   gunicorn -w 3 -b 0.0.0.0:5000 application:app

1. Test the Application:
   • Access the Flask application in your browser: http://your-ec2-public-ip:5000

Part 2: Configuring NGINX as a Reverse Proxy

1. Install NGINX:

   sudo apt install nginx -y

1. Configure NGINX:

   • Open the NGINX configuration file:

    sudo nano /etc/nginx/sites-available/flask-app
   

• Add the following configuration:
  

   server {
       listen 80;
       server_name ;
  
       location / {
           proxy_pass http://127.0.0.1:5000;
           proxy_set_header Host $host;
           proxy_set_header X-Real-IP $remote_addr;
           proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
           proxy_set_header X-Forwarded-Proto $scheme;
       }
   }
  

1. Enable the Configuration:

   sudo ln -s /etc/nginx/sites-available/flask-app /etc/nginx/sites-enabled
   sudo nginx -t
   sudo systemctl restart nginx

1. Test the Application on Port 80:
   • Access your Flask application at: http://your-ec2-public-ip

Part 3: Integrating AWS RDS for Database

1. Create an RDS Instance:
   • Use AWS RDS Free Tier to create a MySQL database.
   • Set it to publicly accessible.

1. Install Database Client Libraries:

   • Install pymysql for MySQL:

    pip install pymysql
   

• Verify installation:
  

   pip show pymysql
  

1. Set Up the Database:

   • Connect to your RDS instance:

    mysql -h  -u  -p
   

• Create a sample database and table:
  

   CREATE DATABASE flaskapp;
   USE flaskapp;
  
   CREATE TABLE messages (
       id INT AUTO_INCREMENT PRIMARY KEY,
       content VARCHAR(255) NOT NULL
   );
  
   INSERT INTO messages (content) VALUES ('Hello from RDS!');
  

1. Update Flask App to Use RDS:

   • Edit application.py to connect to RDS and fetch data:

    from flask import Flask
    import pymysql
   
    app = Flask(__name__)
   
    # RDS Configuration
    DB_HOST = "flask-app-db.xxxxxxx.us-east-1.rds.amazonaws.com"
    DB_USER = "admin"
    DB_PASSWORD = "yourpassword"
    DB_NAME = "flaskapp"
   
    def get_db_connection():
        return pymysql.connect(
            host=DB_HOST,
            user=DB_USER,
            password=DB_PASSWORD,
            database=DB_NAME,
            cursorclass=pymysql.cursors.DictCursor
        )
   
    @app.route("/")
    def home():
        connection = get_db_connection()
        try:
            with connection.cursor() as cursor:
                cursor.execute("SELECT content FROM messages")
                result = cursor.fetchall()
                messages = [row["content"] for row in result]
                return "
   ".join(messages)
        finally:
            connection.close()
   
    if __name__ == "__main__":
        app.run(host="0.0.0.0", port=5000, debug=True)
   

2. Restart Flask Application:

   • Restart the Flask app to apply the changes:

    python application.py
   

3. Test the Application:

   • Access your Flask application at: http://your-ec2-public-ip

Part 4: Adding Auto-Scaling with Load Balancer

1. Create an Amazon Machine Image (AMI):

• Log in to the AWS Management Console and go to EC2 Dashboard.
• Select your running EC2 instance that has the Flask app configured.
• In the Actions dropdown, under Image and templates, select Create image.
• In the Create Image dialog, give your image a name (e.g., flask-app-ami), and optionally provide a description.
• Click Create Image. AWS will begin creating the AMI. It might take a few minutes to complete.
• Once the image is created, you’ll be able to launch new instances based on this AMI.

2. Set Up an Auto Scaling Group:

• Log in to the AWS Management Console and go to the EC2 Dashboard.
• Under Auto Scaling, select Auto Scaling Groups and click Create an Auto Scaling group.
• Select your AMI:
  • Choose Custom AMI and select the AMI you just created.
• Choose an Instance Type:
  • Choose an instance type like t2.micro for testing (or any suitable instance type).
• Configure the Auto Scaling Group:
  • Provide a name for the group (e.g., flask-app-ASG).
  • Choose your desired VPC and Subnets.
  • Set Scaling Policies to scale up at 70% CPU utilization and scale down at 40% CPU utilization.
• Set up Load Balancer:
  • You'll configure a Target Group later for the Elastic Load Balancer.
• Review and Create the ASG:
  • Review the settings and click Create Auto Scaling group.

3. Add an Elastic Load Balancer (ELB):

• Log in to the AWS Management Console and go to EC2 Dashboard.
• Select Load Balancers and click Create Load Balancer.
• Select Application Load Balancer (ALB).
• Provide a name (e.g., flask-app-ALB) and choose internet-facing.
• Select your VPC and the appropriate availability zones (AZs).
• Configure Listeners:
  • By default, ALB listens on HTTP (port 80). You can modify this if needed.
• Create a Target Group:
  • Choose Instance as the target type.
  • For health checks, select HTTP and path / (to check the homepage of your Flask app).
• Register your instances with this target group.
• Assign Load Balancer to Auto Scaling Group:
  • When you create the Auto Scaling Group (ASG), assign this Load Balancer to the ASG so it can distribute traffic across the instances in the ASG.
• Update Security Groups:
  • Ensure that the security group for the load balancer allows inbound traffic on port 80.
  • Ensure your EC2 instances' security group allows inbound traffic from the load balancer.

4. Update NGINX Configuration for Private IP:

• Find the Private IP of the Instance:
  • In your EC2 Dashboard, under Instances, find the instance and copy its private IP.

• Update NGINX Configuration:

  • Update the NGINX config to bind to the private IP:

     server {
         listen 80;
         server_name ;  # Replace with your EC2 public IP or domain name
  
         location / {
             proxy_pass http://127.0.0.1:5000;  # Ensure this matches your Flask app's port
             proxy_set_header Host $host;
             proxy_set_header X-Real-IP $remote_addr;
             proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
             proxy_set_header X-Forwarded-Proto $scheme;
         }
     }
  

• Restart NGINX:
  

   sudo systemctl restart nginx
  

5. Test Load Balancing:

• Simulate load using tools like ApacheBench:
  

   ab -n 1000 -c 10 http:///
  

  • -n 1000: Number of requests.
  • -c 10: Number of concurrent requests.

• Using Locust:

  • Install Locust:

     pip install locust
  

  • Create a locustfile.py:

     from locust import HttpUser, between, task
  
     class LoadTest(HttpUser):
         wait_time = between(1, 3)
  
         @task
         def home(self):
             self.client.get("/")
  

  • Run Locust:

     locust -f locustfile.py --host http://
  

  • Access Locust's web UI (usually at http://localhost:8089) to start the load test.

Part 5: Monitoring and Scaling with CloudWatch

To ensure your application is running smoothly and efficiently, you'll set up monitoring for your EC2 instance, RDS database, and Load Balancer using AWS CloudWatch. Additionally, you'll configure alarms for critical metrics.

Step 1: Log EC2 Application Data

Install the CloudWatch Agent

• Install CloudWatch Agent:
  

   sudo apt update
   sudo apt install -y amazon-cloudwatch-agent
  

• Verify Installation:
  

   amazon-cloudwatch-agent-ctl -a status
  

Configure the CloudWatch Agent

• Create a Configuration File:
  

   sudo nano /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json
  

• Example Configuration:
  

   {
      "logs": {
          "logs_collected": {
              "files": {
                  "collect_list": [
                      {
                          "file_path": "/var/log/nginx/access.log",
                          "log_group_name": "nginx-logs",
                          "log_stream_name": "{instance_id}/access.log"
                      },
                      {
                          "file_path": "/var/log/nginx/error.log",
                          "log_group_name": "nginx-logs",
                          "log_stream_name": "{instance_id}/error.log"
                      },
                      {
                          "file_path": "/home/ubuntu/flask-app.log",
                          "log_group_name": "flask-logs",
                          "log_stream_name": "{instance_id}/flask-app.log"
                      }
                  ]
              }
          }
      }
   }
  

  • Replace /home/ubuntu/flask-app.log with the actual log file path for your Flask app.

Start the Agent

• Start the CloudWatch Agent:
  

   sudo amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -c file:/opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json -s
  

• Verify Logs in CloudWatch:

  • Go to the CloudWatch Console → Log Groups → Verify nginx-logs and flask-logs.

Step 2: Enable Alarms

EC2 Monitoring

• Enable Detailed Monitoring:
  • In the EC2 Console, select your instance → Actions → Monitor and troubleshoot → Manage detailed monitoring.
• Set Up an Alarm for CPU Usage:
  • Go to CloudWatch Console → Alarms → Create alarm.
  • Select the CPUUtilization metric for your EC2 instance.
  • Set the threshold (e.g., >70% for 5 minutes).
  • Configure the action to send an email via SNS (Simple Notification Service).

• Install Memory Monitoring:

  • Add memory usage metrics to the CloudWatch Agent configuration:

     sudo nano /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json
  

  • Add this block under the "metrics" section:

     {
        "metrics": {
            "append_dimensions": {
                "InstanceId": "${aws:InstanceId}"
            },
            "metrics_collected": {
                "mem": {
                    "measurement": [
                        "mem_used_percent"
                    ],
                    "metrics_collection_interval": 60
                }
            }
        }
     }
  

  • Restart the CloudWatch Agent:

     sudo systemctl restart amazon-cloudwatch-agent
  

RDS Monitoring

• Enable Enhanced Monitoring for RDS:
  • Go to the RDS Console, select your database.
  • Modify the database instance and enable Enhanced Monitoring with a granularity of 1 minute.
  • Metrics like CPUUtilization, ReadIOPS, and WriteIOPS will now be available in CloudWatch.
• Create an Alarm for RDS:
  • Go to CloudWatch Console → Alarms → Create alarm.
  • Select an RDS metric (e.g., CPUUtilization).
  • Set a threshold (e.g., >70% for 5 minutes) and configure an SNS action.

Load Balancer Monitoring

• Enable ELB Metrics:
  • In the Load Balancer Console, make sure CloudWatch Metrics are enabled for your ELB.
  • Metrics like RequestCount and HTTPCode_Backend_5XX will appear in CloudWatch.
• Create an Alarm for ELB:
  • Go to CloudWatch Console → Alarms → Create alarm.
  • Select an ELB metric (e.g., UnhealthyHostCount).
  • Set a threshold (e.g., >1 unhealthy instance for 5 minutes) and configure an SNS action.

Step 3: Test Monitoring and Alarms

Simulate High CPU Usage

• Use a stress testing tool like stress to simulate high CPU:
  

   sudo apt install -y stress
   stress --cpu 2 --timeout 300
  

• Check CloudWatch for alerts.

Generate Logs

• Access your Flask app and force some errors (e.g., incorrect database queries) to populate error logs in CloudWatch.

Recap

• CloudWatch Logs: Capturing logs from NGINX and Flask.
• EC2 Monitoring: Alarms for CPU and memory usage.
• RDS Monitoring: Tracking database performance.
• ELB Monitoring: Health checks and request distribution.

With these steps, you can ensure that your application is monitored for performance and reliability. AWS CloudWatch provides the tools to keep an eye on your infrastructure and respond proactively to any issues.

Part 6: Setting Environment Variables

Using environment variables to store sensitive data such as database credentials and application secrets helps in securely managing and accessing these values without hardcoding them into your application. This can also be achieved using AWS Systems Manager Parameter Store, but here we'll focus on environment variables.

Step 1: Set Environment Variables on Your EC2 Instance

1. Edit the .bashrc or .profile File:

   • Open the .bashrc file using a text editor:

    sudo nano ~/.bashrc
   

• Add your credentials as environment variables at the end of the file:
  

   export DB_HOST="database-1.c1u22o08cg4k.eu-north-1.rds.amazonaws.com"
   export DB_USER="admin"
   export DB_PASSWORD="SheepliFe6"
   export DB_NAME="database1"
  

• Save the file and reload it:
  

   source ~/.bashrc
  

1. Verify Environment Variables:

   • Run the following commands to confirm the variables are set:

    echo $DB_HOST
    echo $DB_USER
    echo $DB_PASSWORD
    echo $DB_NAME
   

Step 2: Update Your Flask App

1. Modify Your Flask App to Use Environment Variables:

   • Edit application.py to pull sensitive data from environment variables:

    import os
    from flask import Flask
    import pymysql
   
    app = Flask(__name__)
   
    # RDS Configuration from Environment Variables
    DB_HOST = os.getenv("DB_HOST")
    DB_USER = os.getenv("DB_USER")
    DB_PASSWORD = os.getenv("DB_PASSWORD")
    DB_NAME = os.getenv("DB_NAME")
   
    # Database Connection
    def get_db_connection():
        return pymysql.connect(
            host=DB_HOST,
            user=DB_USER,
            password=DB_PASSWORD,
            database=DB_NAME,
            cursorclass=pymysql.cursors.DictCursor
        )
   
    @app.route("/")
    def home():
        connection = get_db_connection()
        try:
            with connection.cursor() as cursor:
                cursor.execute("SELECT content FROM messages")
                result = cursor.fetchall()
                messages = [row["content"] for row in result]
                return "
   ".join(messages)
        finally:
            connection.close()
   
    if __name__ == "__main__":
        app.run(host="0.0.0.0", port=5000)
   

Why Use These Approaches?

Environment Variables:

• Simple and straightforward for small-scale apps.
• Quick to set up during development.
• Enhances security by avoiding hardcoding sensitive information in the application code.