Job Description:

1. Design, develop, and deploy complex data pipelines using Apache Airflow on AWS infrastructure.
2. Architect, optimize, and automate data workflows to ensure reliability, scalability, and performance.
3. Collaborate with data scientists, analysts, and other stakeholders to understand requirements and translate them into technical solutions.
4. Implement best practices for data ingestion, processing, transformation, and storage.
5. Monitor pipeline performance, troubleshoot issues, and implement enhancements to improve efficiency and reliability.
6. Ensure data quality and integrity throughout the data lifecycle by implementing robust validation and error handling mechanisms.
7. Stay current with emerging technologies and industry trends in data engineering and contribute to continuous improvement initiatives.
8. Mentor junior team members and provide technical guidance and support as needed.

Qualification:

1. 5-7 years of professional experience in data engineering, with a focus on building and managing data pipelines.
2. Extensive hands-on experience with Apache Airflow, including DAG design, scheduling, and orchestration.
3. Strong proficiency in AWS services such as S3, EC2, EMR, Glue, Lambda, and Redshift.
4. Solid understanding of data warehousing concepts, ETL processes, and data modeling techniques.
5. Proficiency in programming languages such as Python, SQL, and Shell scripting.