The project was initiated to address the challenge of efficiently collecting, storing, and analyzing NBA sports data for advanced sports analytics. The organizational pain points included the lack of a centralized, scalable data repository and the absence of an automated pipeline to ingest and query NBA player data. The strategic objective was to build a robust data lake infrastructure on AWS that supports scalable data storage, seamless integration with analytics tools, and cost-effective querying capabilities.
My specific contribution was designing and implementing the entire data lake setup pipeline using AWS services and integrating it with external NBA data sources. I began with an initial assessment of the requirements, which included reliable data ingestion from a third-party API, scalable storage, metadata management, and query capability. My strategic thinking process focused on leveraging AWS managed services like S3, Glue, and Athena to build a serverless, scalable, and cost-efficient solution.
The technical challenge was to ingest NBA player data from an external API into a scalable data lake architecture that supports efficient querying and analytics. Existing infrastructure lacked automated data ingestion, centralized storage, and metadata cataloging, limiting performance and scalability. Constraints included handling large datasets, ensuring data consistency, and enabling performant SQL queries over JSON data.
IAM policy: Set up the necessary policy to create resources.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:CreateBucket",
"s3:PutObject",
"s3:GetObject",
"s3:DeleteObject",
"s3:ListBucket"
],
"Resource": [
"arn:aws:s3:::sports-analytics-data-lake",
"arn:aws:s3:::sports-analytics-data-lake/*"
]
},
{
"Effect": "Allow",
"Action": [
"glue:CreateDatabase",
"glue:DeleteDatabase",
"glue:GetDatabase",
"glue:GetDatabases",
"glue:CreateTable",
"glue:DeleteTable",
"glue:GetTable",
"glue:GetTables",
"glue:UpdateTable"
],
"Resource": [
"arn:aws:glue:*:*:catalog",
"arn:aws:glue:*:*:database/glue_nba_data_lake",
"arn:aws:glue:*:*:table/glue_nba_data_lake/*"
]
},
{
"Effect": "Allow",
"Action": [
"athena:StartQueryExecution",
"athena:GetQueryExecution",
"athena:GetQueryResults"
],
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"s3:PutObject"
],
"Resource": [
"arn:aws:s3:::sports-analytics-data-lake/athena-results/*"
]
}
]
}
Infrastructure Setup: First, I created the core S3 bucket that would serve as the foundation of our data lake:
def create_s3_bucket():
"""Create an S3 bucket for storing sports data."""
try:
if region == "ap-south-1":
s3_client.create_bucket(Bucket=bucket_name)
else:
s3_client.create_bucket(
Bucket=bucket_name,
CreateBucketConfiguration={"LocationConstraint": region},
)
print(f"S3 bucket '{bucket_name}' created successfully.")
except Exception as e:
print(f"Error creating S3 bucket: {e}")
This function handles the region-specific bucket creation syntax required by AWS.
Glue Database Creation: Next, I established a Glue database to serve as the organizational container for our data catalog:
def create_glue_database():
"""Create a Glue database for the data lake."""
try:
glue_client.create_database(
DatabaseInput={
"Name": glue_database_name,
"Description": "Glue database for NBA sports analytics.",
}
)
print(f"Glue database '{glue_database_name}' created successfully.")
except Exception as e:
print(f"Error creating Glue database: {e}")
Data Ingestion Pipeline: The core of the solution is the data extraction and loading process. I implemented an API client that securely retrieves data from SportsData.io:
def fetch_nba_data():
"""Fetch NBA player data from sportsdata.io."""
try:
headers = {"Ocp-Apim-Subscription-Key": api_key}
response = requests.get(nba_endpoint, headers=headers)
response.raise_for_status()
print("Fetched NBA data successfully.")
return response.json()
except Exception as e:
print(f"Error fetching NBA data: {e}")
return []
To ensure Athena compatibility, I implemented a function to convert standard JSON arrays to line-delimited JSON format:
def convert_to_line_delimited_json(data):
"""Convert data to line-delimited JSON format."""
print("Converting data to line-delimited JSON format...")
return "\n".join([json.dumps(record) for record in data])
The upload function then handles writing this properly formatted data to S3:
def upload_data_to_s3(data):
"""Upload NBA data to the S3 bucket."""
try:
line_delimited_data = convert_to_line_delimited_json(data)
file_key = "raw-data/nba_player_data.jsonl"
s3_client.put_object(
Bucket=bucket_name,
Key=file_key,
Body=line_delimited_data
)
print(f"Uploaded data to S3: {file_key}")
except Exception as e:
print(f"Error uploading data to S3: {e}")
Metadata Management: With data in S3, the next step was creating the Glue table definition that would allow Athena to query it:
def create_glue_table():
"""Create a Glue table for the data."""
try:
glue_client.create_table(
DatabaseName=glue_database_name,
TableInput={
"Name": "nba_players",
"StorageDescriptor": {
"Columns": [
{"Name": "PlayerID", "Type": "int"},
{"Name": "FirstName", "Type": "string"},
{"Name": "LastName", "Type": "string"},
{"Name": "Team", "Type": "string"},
{"Name": "Position", "Type": "string"},
{"Name": "Points", "Type": "int"}
],
"Location": f"s3://{bucket_name}/raw-data/",
"InputFormat": "org.apache.hadoop.mapred.TextInputFormat",
"OutputFormat": "org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat",
"SerdeInfo": {
"SerializationLibrary": "org.openx.data.jsonserde.JsonSerDe"
},
},
"TableType": "EXTERNAL_TABLE",
},
)
print(f"Glue table 'nba_players' created successfully.")
except Exception as e:
print(f"Error creating Glue table: {e}")
Note the use of the JsonSerDe serialization library, which is critical for properly parsing the JSON data in Athena.
Query Configuration: Finally, I configured Athena to ensure query results would be stored in a designated S3 location:
def configure_athena():
"""Set up Athena output location."""
try:
athena_client.start_query_execution(
QueryString="CREATE DATABASE IF NOT EXISTS nba_analytics",
QueryExecutionContext={"Database": glue_database_name},
ResultConfiguration={"OutputLocation": athena_output_location},
)
print("Athena output location configured successfully.")
except Exception as e:
print(f"Error configuring Athena: {e}")
Orchestration: The main function ties everything together in the proper sequence:
def main():
print("Setting up data lake for NBA sports analytics...")
create_s3_bucket()
time.sleep(5)
create_glue_database()
nba_data = fetch_nba_data()
if nba_data:
upload_data_to_s3(nba_data)
create_glue_table()
configure_athena()
print("Data lake setup complete.")
This project significantly advanced the organization's capability to perform NBA sports analytics by building a scalable, automated data lake on AWS. Lessons learned around AWS Glue and Athena integration will inform future data engineering projects. Potential future developments include real-time data ingestion and machine learning model integration for predictive analytics.
