Building a Cloud-Based NBA Data Lake with AWS

Welcome to Day 3 of my 30 Days DevOps Challenge! Today, I tackled a fascinating project: creating a Cloud-Based Data Lake for NBA analytics. Using AWS S3, AWS Glue, and Amazon Athena, I built a scalable and efficient solution to store, query, and analyze real-world NBA data fetched from the SportsData.io API.

If you're interested in learning how to combine cloud computing, serverless architecture, and external APIs to create powerful data pipelines, this post is for you.

What Is a Data Lake?

A data lake is a centralized repository that allows you to store structured and unstructured data at any scale. Unlike traditional databases, which require data to be transformed before storage, data lakes let you dump raw data and process it on demand.

For this project, I created a data lake tailored for NBA analytics. The setup includes:

• Raw data storage in Amazon S3.
• Data cataloging and schema creation with AWS Glue.
• On-demand querying using SQL through Amazon Athena.

Project Goals

1. Create an Amazon S3 bucket to store raw and processed NBA data.
2. Upload sample data (retrieved from the SportsData.io API) to S3.
3. Set up AWS Glue to catalog and structure the data for querying.
4. Enable querying with Amazon Athena, allowing SQL-based access to NBA analytics.
5. Automate the entire workflow using a Python script.

Tech Stack

Here’s what I used to bring this project to life:

AWS Services

• Amazon S3: For scalable storage of raw and structured data.
• AWS Glue: To catalog the data and define schemas.
• Amazon Athena: For running SQL queries on the data directly from S3.

Programming Language

• Python: To automate resource creation and data processing.

External API

• SportsData.io: To fetch real-world NBA data in JSON format.

How It Works

The entire workflow can be broken into the following steps:

Step 1: Create an S3 Bucket

Amazon S3 is the backbone of this data lake. It serves as a centralized storage location for all raw and processed data. The bucket was created using Python’s boto3 library, with separate folders for raw data and processed data.

import boto3

def create_s3_bucket(bucket_name):
    s3 = boto3.client('s3')
    s3.create_bucket(Bucket=bucket_name)
    print(f"S3 bucket '{bucket_name}' created successfully.")

Step 2: Fetch Data from SportsData.io

To populate the data lake, I pulled sample NBA player data from the SportsData.io API. This API provides comprehensive information about teams, players, and game statistics.

Here’s the function I wrote to fetch player data:

import requests

def fetch_nba_player_data(api_key, endpoint):
    headers = {"Ocp-Apim-Subscription-Key": api_key}
    response = requests.get(endpoint, headers=headers)
    if response.status_code == 200:
        return response.json()
    else:
        raise Exception("Failed to fetch data from SportsData.io")

Step 3: Upload Data to S3

Once the data was retrieved, it was stored in the raw-data folder of the S3 bucket.

def upload_to_s3(bucket_name, file_name, data):
    s3 = boto3.client('s3')
    s3.put_object(Bucket=bucket_name, Key=file_name, Body=data)
    print(f"Data uploaded to {bucket_name}/{file_name}.")

Step 4: Set Up AWS Glue

AWS Glue makes querying raw JSON data easy by creating a schema and cataloging it. Using Glue, I created a database and table to represent the NBA data.

def create_glue_database(database_name):
    glue = boto3.client('glue')
    glue.create_database(DatabaseInput={'Name': database_name})
    print(f"Glue database '{database_name}' created successfully.")

Querying Data with Athena

With the data cataloged, I used Amazon Athena to query the S3-stored data using SQL. For instance, to fetch all NBA players who play as point guards (PG), I ran the following query:

SELECT FirstName, LastName, Position, Team
FROM nba_players
WHERE Position = 'PG';

Athena directly queries data from S3 without requiring a database or server, making it a powerful tool for analytics.

Automating the Workflow

To tie everything together, I created a Python script that automates the entire setup process. This script:

1. Creates an S3 bucket.
2. Fetches data from SportsData.io.
3. Uploads the data to S3.
4. Configures AWS Glue for cataloging.

Here’s an excerpt:

def main():
    # Step 1: Create S3 bucket
    bucket_name = "sports-analytics-data-lake"
    create_s3_bucket(bucket_name)

    # Step 2: Fetch NBA data
    api_key = "YOUR_SPORTSDATA_API_KEY"
    endpoint = "https://api.sportsdata.io/v3/nba/scores/json/Players"
    nba_data = fetch_nba_player_data(api_key, endpoint)

    # Step 3: Upload data to S3
    upload_to_s3(bucket_name, "raw-data/nba_player_data.json", str(nba_data))

    # Step 4: Create Glue database
    create_glue_database("nba_analytics")

    print("NBA Data Lake setup completed successfully.")

Key Learnings from Day 3

1. Cloud Services Simplify Complex Workflows
   
   AWS services like S3, Glue, and Athena provide a seamless way to store, catalog, and query data without managing servers.

2. Automation Is Essential
   
   Using Python scripts to automate resource creation ensures repeatability and minimizes manual effort.

3. APIs Unlock Real-World Use Cases
   
   Integrating external APIs, like SportsData.io, enriches projects with real-world data and expands functionality.

Challenges Faced

1. Learning Glue and Athena
   
   Understanding how AWS Glue interacts with Athena took some time, but once I grasped the workflow, it became intuitive.

2. Schema Design for Raw Data
   
   Creating a schema for raw JSON data required some trial and error, especially when working with nested structures.

Future Enhancements

While the data lake is functional, there’s always room for improvement. Here are some ideas:

1. Automated Data Ingestion: Use AWS Lambda to automatically ingest new data into the S3 bucket.
2. Data Transformation: Leverage AWS Glue ETL (Extract, Transform, Load) to clean and process raw data.
3. Advanced Analytics: Integrate AWS QuickSight to create dashboards for data visualization.

Conclusion

Day 3 of my 30 Days DevOps Challenge was all about combining AWS cloud services, external APIs, and Python to build a powerful data lake for NBA analytics. This project not only demonstrated the power of serverless architecture but also showcased how data lakes can enable real-time insights for real-world use cases.

If you’re exploring data engineering, cloud computing, or DevOps workflows, I hope this blog inspires you to take on your own challenge. Have questions or suggestions? Let’s connect in the comments below!

Resources

• AWS S3 Documentation
• AWS Glue Documentation
• Amazon Athena Documentation
• SportsData.io API