Senior Azure Data Engineer

Location: Hyderabad, Bangalore and Pune

Experience: 3-7 Years

Work Mode: Hybrid

Mandatory Skills: Python, Spark, SQL, Hadoop, Hive, Spark, Azure Databricks, Azure DataFactory, Azure Synapse, DLTs, Unity Catalog, Workflows, Infrastructure as Code (IaC), Orchestration-Airflow, ETL & Data Pipelines.

Role Overview:

We are looking for a Senior Data Engineer with a strong background in Python, Spark, Databricks, and Azure to join our growing team. The ideal candidate will have extensive experience in data processing, working with cloud platforms (specifically Azure), and be able to handle complex data engineering tasks. 

Key Responsibilities:

• Design, build, and optimize scalable data pipelines for large-scale data processing using Python and Apache Spark.
• Leverage Azure Databricks to perform big data processing, including working with PySpark to process and analyze large datasets.
• Utilize Azure Data Factory for seamless data integration and orchestration across cloud environments.
• Implement and maintain solutions using Azure Data Lake Storage for big data analytics and transformation.
• Write efficient and reusable Python code, utilizing libraries such as Pandas and NumPy, to process and manipulate data.
• Work with core Azure data services like Azure SQL Database, Azure Synapse Analytics, Azure Cosmos DB, and Azure Blob Storage.
• Should have worked with popular tools for Infrastructure as Code (IaC) like Terraform or Biceps in conjunction with CI/CD pipelines for the purpose of provisioning new Azure Infra resources.
• Architectural experience is plus
• Design and maintain complex SQL queries for managing and analyzing relational databases.
• Ensure high-quality data management, integrity, and security across all stages of data processing & analysis.
• Collaborate with cross-functional teams to understand data requirements and build solutions to meet business needs.
• Provide mentorship and guidance to junior team members.

Key Skills & Qualifications:

• Programming: Proficiency in Python, including knowledge of Pythonic principles, data structures, and algorithms. Hands-on experience with libraries such as Pandas and NumPy.
• Cloud Technologies: In-depth knowledge of Azure, including Azure SQL Database, Azure Synapse Analytics, Azure Cosmos DB, Azure Data Factory, Azure Blob Storage, and Azure Data Lake Storage.
• Data Processing: Strong experience with Apache Spark using Azure Databricks and PySpark for large-scale data processing.
• SQL: Strong skills in SQL for querying and managing relational databases. Ability to write complex and optimized SQL queries.
• Communication: Excellent communication skills, with the ability to collaborate effectively across teams and present technical solutions to non-technical stakeholders.
• Problem-Solving: Strong analytical and problem-solving skills, with the ability to troubleshoot and resolve complex data challenges

Education:  B.Tech in any branch or specialization.