Azure Databricks

Role Overview:

We are seeking a highly skilled Azure Databricks Developer with relevant years of experience in Azure Cloud services and hands-on expertise with Databricks, Azure data factory,ADB, Python, and PySpark. The ideal candidate will be responsible for designing, implementing, and optimizing data processing pipelines and will play a critical role in our data engineering projects.

Key Responsibilities:

Data Engineering:

• Design, develop, and optimize large-scale data processing pipelines using Azure Databricks, Azure data factory,etc.
• Implement complex data transformations and processing workflows with PySpark.
• Develop and maintain scalable data architecture and pipelines to support various data projects.

Azure Cloud Services:

• Utilize various Azure services such as Azure Data Factory, Azure Data Lake Storage, and Azure Synapse Analytics to build end-to-end data solutions.
• Monitor, troubleshoot, and optimize Azure cloud resources and Databricks clusters for performance and cost-efficiency.

Data Integration:

• Integrate data from multiple sources and ensure data consistency and integrity.
• Develop and implement data validation and quality checks.

Collaboration:

• Work closely with data scientists, analysts, and other stakeholders to understand data requirements and deliver solutions.
• Participate in agile development processes and contribute to continuous improvement practices.

Documentation and Reporting:

• Document data workflows, architecture, and processes for knowledge sharing and maintenance.
• Generate reports and visualizations to communicate insights and progress to stakeholders.

Required Skills and Experience:

Technical Skills:

• Strong experience with Azure Databricks , Azure data factory,etc and PySpark.
• Proficiency in Python programming, with a focus on data processing and manipulation.
• Experience with Azure services such as Azure Data Factory, Azure Data Lake Storage, and Azure Synapse Analytics.
• Solid understanding of data warehousing concepts and ETL processes.
• Familiarity with CI/CD practices and tools for data pipelines.

Professional Experience:

• Rel years of hands-on experience in data engineering and cloud-based data processing.
• Proven track record of working with large-scale data systems and distributed computing.
• Experience in performance tuning and optimization of data processing jobs.

Soft Skills:

• Excellent problem-solving skills and attention to detail.
• Strong communication skills to collaborate effectively with cross-functional teams.
• Ability to work independently and manage multiple tasks in a dynamic environment.