Lead Data Engineer (Data Fabric) || (10-15 Y) || Remote || Quick Joiner ||

This role is critical and requires resources to have adequate hands on exp in MS Fabric in Projects and should be reflective in the resume too.

Scope of Work:

Design, develop, and maintain ETL pipelines using PySpark Notebooks and Microsoft Fabric.

Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and deliver efficient data solutions.

Migrate and integrate data from legacy SQL Server environments into modern data platforms.

Optimize data pipelines and workflows for scalability, efficiency, and reliability.

Provide technical leadership and mentorship to junior developers and other team members.

Troubleshoot and resolve complex data engineering issues related to performance, data quality, and system scalability.

Develop, maintain, and enforce data engineering best practices, coding standards, and documentation.

Conduct code reviews and provide constructive feedback to improve team productivity and code quality.

Support data-driven decision-making processes by ensuring data integrity, availability, and consistency across different platforms.

Requirements

Bachelor’s or master’s degree in computer science, Data Science, Engineering, or a related field.

Experience with Microsoft Fabric or similar cloud-based data integration platforms - any (SAP HANA Cloud / Databricks DIP / Vertex AI / SnowFlake / Dataiku / IBM Watson Studio / Dataloop, etc.) is a must.

10+ years of experience in data engineering, with a strong focus on ETL development using PySpark or other Spark-based tools.

Proficiency in SQL with extensive experience in complex queries, performance tuning, and data modeling.

Strong knowledge of data warehousing concepts, ETL frameworks, and big data processing.

Familiarity with other data processing technologies (e.g., Hadoop, Hive, Kafka) is an advantage.

Experience working with both structured and unstructured data sources.

Excellent problem-solving skills and the ability to troubleshoot complex data engineering issues.

Proven ability to work independently, as part of a team, and in leadership roles.

Strong communication skills with the ability to translate complex technical concepts into business terms.

Highly Desired Skills:

Experience with Azure Data Services, including Azure Data Factory, Azure Synapse, or similar tools.

Knowledge of scripting languages (e.g., Python, Scala) for data manipulation and automation.

Familiarity with DevOps practices, CI/CD pipelines, and containerization (Docker, Kubernetes) is a plus