Job Name : Senior Databricks Data Engineer

Location : Chennai - Work from Office

Job Type : Full-time

Years of Experience : 6+ years

Senior Databricks Data Engineer

JOB TITLE : SENIOR DATABRICKS DATA ENGINEER (6-9 yrs)

About Client : Our client is a market-leading Content and Data Technology company providing data services, subject matter expertise, & technology solutions to multiple domains. Data Analytics & Al Solutions, Data Al Powered Operations, and Education & Learning form the core pillars of the company's long-term vision. The company is a specialized solutions provider to business information providers in finance, insurance, legal, real estate, life sciences, and logistics. Client continues to be the leading content services provider to research and education publishers. Data Analytics & Al Services : Our Data Solutions business has become critical to our client's success. We use technology and Al with human experts in the loop to create data assets that our clients use to power their data products and their end customers" workflows. As our clients expect us to become their future-fit Analytics and Al partner, they look to us for help building data analytics and Al enterprise capabilities. With a client base scoping 30 countries worldwide, Client" s multi-geographical resource pool is strategically located in eight countries - India, Philippines, USA, Nicaragua, Vietnam, United Kingdom Job Title : Senior Databricks Data Engineer Job Summary : We are seeking an experienced Senior Databricks Data Engineer with a very good understanding of Databricks, Lakehouse concepts, with architecture experience, and a sound knowledge of Azure and its services. The Senior Databricks Data Engineer will play a pivotal role in designing, developing, and maintaining data pipelines and data solutions on the Databricks platform, with a focus on building a Lakehouse architecture. This role requires a deep knowledge of data engineering fundamentals, a strong grasp of Databricks and Lakehouse principles, the ability to architect data solutions for scalability and efficiency while leveraging Azure services, and expertise in data migration strategies. Understanding Hadoop eco-system will add brownie points. Key Responsibilities : - Architect Lakehouse Solutions : Design, implement, and architect Lakehouse solutions on Databricks, leveraging Delta Lake, and utilize Azure services to enhance data storage and processing. - In-depth Databricks Expertise : Demonstrate a deep understanding of Databricks platform features, including Spark SQL, Delta Lake, and Databricks notebooks, to optimize data engineering processes. - Data Transformation : Implement advanced data transformations and quality checks within the Lakehouse architecture to ensure data accuracy, completeness, and consistency. - Performance Optimization : Architect data pipelines for performance and scalability, making efficient use of Databricks clusters and Azure resources. - Data Modelling : Create and implement advanced data models and schemas on Databricks, aligning with Lakehouse principles for analytical and reporting needs. - Data Integration : Seamlessly integrate data from diverse sources, aligning with Lakehouse principles for data ingestion and storage, leveraging Azure Data Lake Storage and Azure SQL Data Warehouse. - Lakehouse Architecture Experience : Apply architecture principles to build and maintain a scalable, efficient, and resilient Lakehouse architecture on Databricks while utilizing Azure for enhanced data processing and storage. - Data Security : Implement and maintain comprehensive data security and access controls within the Lakehouse architecture, utilizing Azure security features to safeguard sensitive data. - Data Migration Expertise : Possess a deep understanding of data migration strategies and practices, particularly in Azure environments, for seamless and secure data transfers. Experience on Hadoop ecosystem will be given advantage. - Collaboration : Collaborate closely with data scientists, data analysts, and cross-functional teams to understand data requirements and deliver Lakehouse-based solutions that meet their needs, incorporating Azure-based solutions and data migration strategies as needed. - Documentation : Create detailed documentation for data pipelines, data models, Lakehouse architecture configurations, Azure integrations, and data migration plans. - Troubleshooting : Proactively identify and resolve complex data pipeline and architecture issues, ensuring data integrity and availability, with a focus on Azure monitoring and diagnostics. - Performance Monitoring : Employ advanced monitoring techniques and Azure-based tools to maintain the performance and health of the Lakehouse architecture, taking proactive measures to address potential bottlenecks or issues. Should also have understanding of cluster monitoring and sizing. - Data Governance : Ensure strict adherence to data governance and data management best practices within the Lakehouse architecture, utilizing Azure-based data governance solutions. - Continuous Learning : Stay updated with the latest trends and developments in Databricks, Lakehouse concepts, Azure, data migration strategies, and data engineering best practices, and incorporate them into projects. Qualifications and Skills : - Bachelor's or higher degree in computer science, data engineering, or a related field. - Proven experience as a data engineer with a strong focus on Databricks, Lakehouse architecture, advanced data engineering concepts, Azure services, and data migration strategies. - Exceptional understanding of data engineering fundamentals, including data modeling, ETL, and data warehousing. - Proficiency in programming languages such as Python, Scala, or Java. - Extensive experience with Databricks platform features, advanced Delta Lake concepts, and a strong working knowledge of Azure services. - Strong problem-solving skills and the ability to work in a fast-paced, collaborative environment. - Excellent communication and teamwork skills. - Relevant certifications in Databricks, Lakehouse, Azure services, and data migration are highly desirable

INTERESTED REACH OUT TO aravindhisys@gmail.com/8838410245