This repository contains a powerful realtime climate analysis pipeline leveraging various AWS services and Snowflake, along with Power BI for insightful visualizations. The pipeline is designed to ingest data from a weather API, store it in Dynamo DB, process it through AWS Lambda functions, store it in S3, and finally load it into Snowflake for further analysis. Python scripts are utilized for carrying out advanced analytics such as linear regression on the collected data.
-
Data Ingestion: Utilizes AWS Lambda functions to ingest data from the Climate API into Dynamo DB.
-
Data Storage: Dynamo DB is used as the primary data storage, with Dynamo DB streams used to trigger the ingestion of data into S3.
-
Data Processing: Data stored in S3 is processed and loaded into Snowflake stages, followed by ingestion into Snowflake tables using Snowpipe.
-
-
Insightful Visualizations: Leverages Power BI for building insightful visualizations comparing parameters such as humidity, temperature, pressure, and wind speed across multiple cities.
-
Advanced Analytics: Python scripts are employed for carrying out advanced analytics like linear regression on the collected data.
-
instance prediction of Temprature with XGBoost Model
-
Optimization and Scaling: Provides best practices for optimizing and scaling the data pipeline to enhance performance, ensure data integrity, and accommodate evolving project needs.
-
Troubleshooting and Best Practices: Equips users with troubleshooting skills and industry best practices to overcome common challenges in real-world scenarios, ensuring the reliability and resilience of the data pipeline.
-
Advanced Concepts and Future Trends: Explores advanced concepts and emerging trends in AWS Data Engineering, keeping users updated with the latest technologies and potential impacts on data engineering practices.
- Clone this repository to your local machine.
- Install the required dependencies mentioned in
requirements.txt. - Configure AWS credentials and permissions for accessing AWS services.
- Set up Snowflake credentials for connecting to Snowflake.
- Run the Lambda functions to ingest data from the Climate API into Dynamo DB.
- Monitor Dynamo DB streams for changes and trigger the ingestion of data into S3.
- Process the data stored in S3 and load it into Snowflake stages.
- Ingest the data from Snowflake stages into Snowflake tables using Snowpipe.
- Utilize Power BI for building insightful visualizations and analytics.
- Execute Python scripts for advanced analytics like linear regression & XGboost Models for prediction.
Contributions are welcome! Feel free to open issues or pull requests for any improvements or new features.
- Thanks to Wheather API for providing the weather data.
- This project was inspired by the need for real-time climate analysis and the power of AWS and Snowflake in handling large-scale data pipelines.
For any inquiries or support, please contact https://www.linkedin.com/in/ritikdhame/ .