Table of Content

1. Introduction

   • Brief overview of the problem: Managing web servers and logs at scale
   • Why AWS EC2 + Ansible + Nginx + Fluentd is a powerful combination
   • Why This Stack Works Together

2. Prerequisites

3. Infrastructure Setup

   • Create security group
   • Security Group configuration for web traffic
   • Verify the security group rules
   • Create your key pair
   • Launch EC2 instance using created security group
   • Copy keypair to ssh directory

4. Ansible Configuration

   • Folder Structure
   • Prepare workspace environment
   • Create the host.yaml file
   • Test ansible connection

5. Set Up Roles

6. Common Task

7. Nginx Playbook

8. Security Playbook

9. Log Management FluentD Playbook

10. Deployment and Testing

Introduction

Brief overview of the problem: Managing web servers and logs at scale

In today's cloud environments, managing web servers and logs at scale presents significant challenges.

DevOps teams struggle with manual server configurations, inconsistent deployments, and the overwhelming task of processing massive log data for insights and security analysis.

Our automated solution combines AWS, Ansible, Nginx, and Fluentd to streamline these operations efficiently.

Why AWS EC2 + Ansible + Nginx + Fluentd is a powerful combination

AWS EC2 + Ansible + Nginx + Fluentd creates a robust web infrastructure by combining cloud scalability with automation and efficient logging.

AWS EC2 provides the flexible compute resources, Ansible automates deployment and configuration tasks, Nginx serves as a high-performance web server, and Fluentd handles comprehensive log collection and processing.

Why This Stack Works Together

AWS EC2 (Infrastructure)

• Provides on-demand, scalable computing resources
• Offers multiple instance types to match workload needs
• Integrates seamlessly with other AWS services
• Enables global deployment with multiple regions

Ansible (Automation)

• Automates server configuration and application deployment
• Uses simple YAML syntax for easy maintenance
• Requires no agents on managed servers (agentless)
• Ensures consistent configurations across all servers

Nginx (Web Server)

• Delivers high-performance web serving capabilities
• Handles concurrent connections efficiently
• Provides reverse proxy and load balancing features
• Offers robust security features and SSL/TLS support

Prerequisites

1. Basic Knowledge of AWS & Ansible
2. AWS CLI installation on local machine
3. Configure AWS CLI
4. Install Ansible

Infrastructure Setup

Before we proceed, with setting up infrastructure. It is best we confirm Ansible is installed on our machine and AWS CLI is well configured.

ansible --version

aws sts get-caller-identity

Now we can go ahead with setting up infrastructure.

Create security group

Next step is to create the security group, copy and paste the below code in your terminal

aws ec2 create-security-group \
    --group-name nginx-web-server-sg \
    --description "Security group for Nginx web server and SSH access"

This command would create a security group and output the

• GroupId
• and SecurityGroupArn;

{
    "GroupId": "sg-0ee2b6c700c11e902",
    "SecurityGroupArn": "arn:aws:ec2:us-east-1:910883278292:security-group/sg-0ee2b6c700c11e902"
}

You can login to your AWS console to confirm the creation of the Security Group

Security Group configuration for web traffic

Lets Add the necessary inbound rules for HTTP (80), HTTPS (443), and SSH (22):

• Export the Security Group ID to a variable (from the previous command output; Replace with your actual Security Group ID)

export SG_ID="sg-0ee2b6c700c11e902"

confirm you have exported your SG_ID
echo $SG_ID

• Allow HTTP (port 80)

aws ec2 authorize-security-group-ingress \
    --group-id $SG_ID \
    --protocol tcp \
    --port 80 \
    --cidr 0.0.0.0/0

The above command should output similar results (Remember your security ID is different from mine)

{
    "Return": true,
    "SecurityGroupRules": [
        {
            "SecurityGroupRuleId": "sgr-081264c3c8fe3e58c",
            "GroupId": "sg-0ee2b6c700c11e902",
            "GroupOwnerId": "910883278292",
            "IsEgress": false,
            "IpProtocol": "tcp",
            "FromPort": 80,
            "ToPort": 80,
            "CidrIpv4": "0.0.0.0/0",
            "SecurityGroupRuleArn": "arn:aws:ec2:us-east-1:910883278292:security-group-rule/sgr-081264c3c8fe3e58c"
        }
    ]
}

1. Allow HTTPS (port 443)

aws ec2 authorize-security-group-ingress \
    --group-id $SG_ID \
    --protocol tcp \
    --port 443 \
    --cidr 0.0.0.0/0

Command output should be similar

{
    "Return": true,
    "SecurityGroupRules": [
        {
            "SecurityGroupRuleId": "sgr-06591dee28547a3eb",
            "GroupId": "sg-0ee2b6c700c11e902",
            "GroupOwnerId": "910883278292",
            "IsEgress": false,
            "IpProtocol": "tcp",
            "FromPort": 443,
            "ToPort": 443,
            "CidrIpv4": "0.0.0.0/0",
            "SecurityGroupRuleArn": "arn:aws:ec2:us-east-1:910883278292:security-group-rule/sgr-06591dee28547a3eb"
        }
    ]
}

• Allow SSH (port 22) - Best practice is to limit this to your IP

aws ec2 authorize-security-group-ingress \
    --group-id $SG_ID \
    --protocol tcp \
    --port 22 \
    --cidr 0.0.0.0/0

Note: for a more secure environment, its advisable to only allow your specific IPs instead of using 0.0.0.0/0 as the cidr block

Command output should be similar

{
    "Return": true,
    "SecurityGroupRules": [
        {
            "SecurityGroupRuleId": "sgr-06c1f6c3205247aea",
            "GroupId": "sg-0ee2b6c700c11e902",
            "GroupOwnerId": "910883278292",
            "IsEgress": false,
            "IpProtocol": "tcp",
            "FromPort": 22,
            "ToPort": 22,
            "CidrIpv4": "105.113.64.38/32",
            "SecurityGroupRuleArn": "arn:aws:ec2:us-east-1:910883278292:security-group-rule/sgr-06c1f6c3205247aea"
        }
    ]
}

Verify the security group rules

aws ec2 describe-security-groups --group-ids $SG_ID

Due to the how lengthy the command outputs for the security group verification is, I wont be posting it here.

Create your key pair

This command would create your key pair, and save it on your local machine

aws ec2 create-key-pair --key-name nginx-server-key --query 'KeyMaterial' --output text > nginx-server-key.pem

Confirm the created key
cat nginx-server-key.pem

Launch EC2 instance using created security group

aws ec2 run-instances \
    --image-id ami-0e1bed4f06a3b463d \
    --instance-type t2.micro \
    --key-name nginx-server-key \
    --security-group-ids $SG_ID \
    --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=nginx-fluentd-server}]'

• *Image AMI: * ami-0e1bed4f06a3b463d : installs Ubuntu 22
• *Instance Type: * vCPUs: 1, Memory: 1 GiB
• Key Pair: Key=Name,Value=nginx-fluentd-server : identifies as the name of the server

Copy keypair to ssh directory

cp nginx-server-key.pem ~/.ssh

SSH into EC2 instance

You need this step to confirm you can actually log into the machine.

• Change key pair permissions chmod 400 ~/.ssh/nginx-server-key.pem
• Connect using ssh ssh -i ~/.ssh/nginx-server-key.pem ubuntu@

Once you can log into the instance via ssh, we can move to the next section.

Ansible Configuration

Folder Structure

.
├── hosts.yaml
├── site.yml
└── roles/
    ├── common/
    ├── └── defaults/
    │   |   └── main.yml
    │   └── tasks/
    │   |    └── main.yml
    │   └── handlers/
    │       └── main.yml
    ├── nginx/
    │   └── tasks/
    │       └── main.yml
    ├── security/
    |   └── defaults/
    │   |   └── main.yml
    │   └── tasks/
    │       └── main.yml
    └── fluentd/
        └── defaults/
        |   └── main.yml
        └── tasks/
        |   └── main.yml
        └── files/
        |   └── denylist.txt
        └── templates/
            └── td-agent.conf.j2

Prepare workspace environment

Open your preferred terminal or use vscode terminal, create a new folder called AWS-Nginx-Ansible-FluentD and navigate into that folder

mkdir AWS-Nginx-Ansible-FluentD
cd AWS-Nginx-Ansible-FluentD

Create the host.yaml file

The hosts.yaml file is an Ansible inventory file that defines the target servers (hosts) Ansible will manage.
This file tells Ansible where and how to connect to the managed nodes for executing tasks or running playbooks.

touch host.yaml

add the below code

---
nginx_server:
  hosts:
    nginx-server-1:
      ansible_host: 
      ansible_user: ubuntu
      ansible_ssh_private_key_file: "{{ lookup('env', 'HOME') }}/.ssh/nginx-server-key.pem"
      ansible_ssh_common_args: "-o StrictHostKeyChecking=no"

Note: change the value of ansible_host to your created instance IP

Test ansible connection

Run the below command in the project folder terminal

ansible nginx-server -i hosts.yaml -m ping

If connection is successfull, you should see the below result

nginx-server-1 | SUCCESS => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/bin/python3.10"
    },
    "changed": false,
    "ping": "pong"
}

Set Up Roles

In Ansible, roles are a way to organize and reuse configurations by breaking them into modular, reusable components.
Each role contains specific;

• Tasks,
• Variables,
• Templates,
• Files,
• and handlers required to perform a particular function, such as installing a web server or configuring a database.

By using roles, you can structure your playbooks more efficiently, making them cleaner, more scalable, and easier to maintain

1. Create site.yaml: This file is the root directory for all roles.

touch site.yaml

1. Add the below snippet to the playbook

---
- name: Configure Web Server with Nginx and Fluentd
  hosts: nginx_server
  become: yes

  roles:
    - common
    - nginx
    - security
    - fluentd

Common Task

Create Required directories and files

• In the root directory, create a folder called common

mkdir common

• Create defaults, handler & task folders

mkdir defaults handler tasks

• Create the main files in defaults, handler & tasks

touch defaults/main.yaml handler/main.yaml tasks/main.yaml

Common Playbook

• variable definition: Add the below code to common\defaults\main.yaml The content of the file defines settings for your Ansible tasks. In simple terms, it’s configuring paths and settings for Nginx and Fluentd, as well as setting a rule for how long logs are retained.

---
nginx_html_root: /var/www/html
fluentd_config_dir: /etc/td-agent
log_retention_days: 5

• Handler Task List: Add the below code to common\handler\main.yaml Restart Nginx: Restarts the nginx service to apply any changes or ensure it is running. Restart Fluentd: Restarts the fluentd service to reload its configuration or ensure it is operational.

---
- name: restart nginx
  service:
    name: nginx
    state: restarted

- name: restart fluentd
  service:
    name: fluentd
    state: restarted

• Main Common Task: Add the below code to common\tasks\main.yaml The content of the file updates the package cache on systems using apt (like Debian or Ubuntu).

---
- name: Update apt cache
  apt:
    update_cache: yes
    cache_valid_time: 3600

update_cache: yes: Refreshes the list of available packages from repositories.
cache_valid_time: 3600: Ensures the cache is valid for 3600 seconds (1 hour), skipping updates if the cache is still recent.

Nginx Playbook

• In the root directory, create a folder called nginx

mkdir nginx

• Create nginx task folder

mkdir nginx/tasks

• Create the nginx tasks files

touch nginx/tasks/main.yaml

• Nginx Task playbook: Add the below code to nginx/tasks/main.yaml

---
- name: Install Nginx
  apt:
    name: nginx
    state: present

- name: Create simple HTML page
  copy:
    content: |
      
      
      
      
Hello, World!
      
    dest: "{{ nginx_html_root }}/index.html"
    mode: '0644'
  notify: restart nginx

- name: Enable and start Nginx
  service:
    name: nginx
    state: started
    enabled: yes

- name: Configure logrotate for Nginx
  template:
    src: nginx-logrotate.j2
    dest: /etc/logrotate.d/nginx
    mode: '0644'

This is an Ansible Playbook Task List that sets up and configures Nginx with the following steps:

1. Install Nginx: Ensures the Nginx package is installed on the target machine.
2. Create a Simple HTML Page: Copies a basic “Hello, World!” HTML file to the Nginx web root (nginx_html_root), with proper permissions (0644). It also triggers a restart nginx handler when changes occur.
3. Enable and Start Nginx: Ensures the Nginx service is running (started) and configured to start automatically on system boot (enabled).
4. Configure Logrotate for Nginx: Deploys a log rotation configuration file for Nginx using a Jinja2 template (nginx-logrotate.j2), setting appropriate permissions (0644).

Security Playbook

• In the root directory, create a folder called security

mkdir security

• Create security defaults & task folder

mkdir security/defaults security/tasks 

• Create the security defaults & tasks files

touch security/defaults/main.yaml security/tasks/main.yaml

• Security variable definition: Add the below code to security/defaults/main.yaml

This file sets firewall rules to allow essential traffic and disables unnecessary services to enhance security.

---
ufw_rules:
  - { rule: 'allow', port: '80', proto: 'tcp' }
  - { rule: 'allow', port: '443', proto: 'tcp' }
  - { rule: 'allow', port: '22', proto: 'tcp' }

disabled_services:
  - rpcbind
  - cups
  - avahi-daemon

• Security Task playbook: Add the below code to security/tasks/main.yaml

---
- name: Install UFW
  apt:
    name: ufw
    state: present

- name: Configure UFW rules
  ufw:
    rule: "{{ item.rule }}"
    port: "{{ item.port }}"
    proto: "{{ item.proto }}"
    state: enabled
  with_items: "{{ ufw_rules }}"

- name: Disable unnecessary services
  service:
    name: "{{ item }}"
    state: stopped
    enabled: no
  with_items: "{{ disabled_services }}"
  ignore_errors: yes

These tasks install and configure a firewall for secure access while stopping and disabling unneeded services to improve security and performance.

FluentD Playbook

Fluentd is used to centralize and manage log data efficiently, enabling better monitoring, troubleshooting, and analytics for applications and systems.

• In the root directory, create a folder called security

mkdir fluentd

• Create fluentd defaults, files, task & templates folder

mkdir fluentd/defaults fluentd/files fluentd/tasks  fluentd/templates 

• Create the fluentd defaults, files, task & templates files

touch fluentd/defaults/main.yaml fluentd/files/denylist.txt fluentd/tasks/main.yaml fluentd/templates/td-agent.conf.j2

• fluentd variable definition: Add the below code to fluentd/defaults/main.yaml

fluentd_config_dir: "/etc/fluentd"

This variable tells Ansible (or any script using it) where to find or manage Fluentd’s configuration files, which typically include settings for log inputs, filters, and outputs.

• Fluentd files: To deny a list of IP addresses, you can add them to the fluentd/files/denylist.txtfile.

192.168.1.100
10.0.0.50

• Fluentd Task Playbook: Add the below code to fluentd/tasks/main.yaml

---
- name: Download and run Fluentd installation script
  shell: |
    curl -fsSL https://toolbelt.treasuredata.com/sh/install-ubuntu-jammy-fluent-package5-lts.sh | sh
  args:
    executable: /bin/bash

- name: Verify Fluentd installation
  command: fluentd --version
  register: fluentd_version

- debug:
    var: fluentd_version.stdout

- name: Create Fluentd config directory
  file:
    path: "{{ fluentd_config_dir }}"
    state: directory
    mode: '0755'
  become: yes

- name: Create Fluentd config
  template:
    src: td-agent.conf.j2
    dest: "{{ fluentd_config_dir }}/td-agent.conf"
    mode: '0644'
  notify: restart fluentd

- name: Enable and start Fluentd
  service:
    name: fluentd
    state: started
    enabled: yes

This playbook automates installing Fluentd, setting up its configuration, and ensuring the service is up and running.

• Fluentd Configuration Jinja Templates: Add the below code to fluentd/templates/td-agent.conf.j2

# Input for Nginx access logs

  @type tail
  path /var/log/nginx/access.log
  pos_file /var/log/td-agent/nginx.access.pos
  tag nginx.access
  
    @type nginx
  


# Input for Nginx error logs

  @type tail
  path /var/log/nginx/error.log
  pos_file /var/log/td-agent/nginx.error.pos
  tag nginx.error
  
    @type regexp
    expression /^(?\d{4}/\d{2}/\d{2} \d{2}:\d{2}:\d{2}) \[(?\w+)\] (?\d+).*?: (?.*)$/
    time_format %Y/%m/%d %H:%M:%S
  


# Filter to check IPs against denylist

  @type grep
  
    key remote_addr
    pattern /^(?!#{File.readlines("#{ENV['FLUENT_CONFIG_DIR'] || '/etc/td-agent'}/denied_ips/denylist.txt").map(&:strip).join('|')}).*$/
  


# Route normal logs (non-denied IPs)

  @type file
  path /var/log/td-agent/nginx_access
  append true
  
    timekey 1d
    timekey_use_utc true
    timekey_wait 10m
  
  
    @type json
  


# Route denied IP logs to audit file

  @type copy
  
    @type file
    path /var/log/td-agent/audit/denylist_audit
    append true
    
      timekey 1d
      timekey_use_utc true
      timekey_wait 10m
    
    
      @type json
      include_time_key true
      time_key timestamp
    
  


# Handle error logs

  @type file
  path /var/log/td-agent/nginx_error
  append true
  
    timekey 1d
    timekey_use_utc true
    timekey_wait 10m
  
  
    @type json
  

This configuration collects and processes Nginx access and error logs. It checks access logs against a denylist, routes normal logs and denied IP logs to separate files, and stores error logs in a structured JSON format for easy analysis.

This setup is ideal for monitoring, auditing, and managing log data efficiently.

If you’ve made it this far, well done! Congratulations!

Deployment and Testing

Running the Playbook

Check syntax

ansible-playbook -i hosts.yaml site.yaml --syntax-check

Run playbook

ansible-playbook -i hosts.yaml site.yaml

Ansible would run through all the task listed in the site.yaml file, and run each of the playbooks sequentially.

If you face an error with TASK [security : Disable unnecessary services] theres no need to panic, it just means, you don't have the unnecessary services to disable and you playbook, would continue to the other task

Verification Steps

Check Nginx status

ansible webservers -i hosts.yaml -m shell -a "systemctl status nginx"  

or input the instance public ip on a browser, to get the Hello, World! page

Check Fluentd status

ansible webservers -i hosts.yaml -m shell -a "systemctl status td-agent"

Test log processing
First make a curl request to the server

ansible nginx_server -i hosts.yaml -m shell -a "curl http://localhost"

Then check the tail of the fluentd logs

ansible nginx_server -i hosts.yaml -m shell -a "tail  /var/log/fluent/fluentd.log"

Conclusion

This solution provides a robust, automated approach to deploying and managing web servers on AWS.
By combining Ansible's automation capabilities with AWS services, we create a scalable and maintainable infrastructure that follows DevOps best practices.