Description

We are seeking a skilled and experienced Data Engineer to join our dynamic team. The ideal candidate will be responsible for designing, building, and maintaining our data infrastructure, ensuring efficient and reliable data processing to support our in-house enterprise data platform. This platform will feature an intuitive UI for managing data ETL/ELT pipelines, running ML jobs, and their lifecycle.

Roles & Responsibilities

●       Lead the design, development, and implementation of data-driven applications using Java/Python technologies, ensuring reliability, scalability, and performance.

●       Provide support and guidance to team members on data-related issues and best practices.

●       Design, develop, and maintain scalable data pipelines to process large volumes of data from various sources. Using technologies such as Apache Spark, Apache Flink, or Hadoop, to extract insights from large datasets.

●       Ensure data is collected, cleaned, transformed, and stored efficiently to support analytics and business intelligence.

●       Integrate data from multiple sources, including databases, APIs, and flat files.

●       Collaborate with data scientists and analysts to understand data requirements and ensure data availability and quality.

●       Design and manage data storage solutions, including relational databases, NoSQL databases, and data warehouses.

●       Optimize database performance and ensure data security and integrity.

●       Monitor and troubleshoot ETL jobs to ensure timely and accurate data processing.

●       Implement data quality checks and validation procedures to ensure data accuracy and consistency.

●       Maintain data documentation and metadata to support data governance practices.

●       Architect and implement event-driven solutions, leveraging technologies like Apache Kafka, RabbitMQ, or Pub-Sub to enable real-time data processing and event streaming.

●       Design and develop data processing pipelines and analytics solutions using technologies such as Apache Spark, Apache Flink, or Hadoop, to extract insights from large datasets.

●       Manage data storage and retrieval, including working with both relational (e.g., MySQL, PostgreSQL) and NoSQL databases (e.g., MongoDB, Cassandra), designing efficient schemas, and optimizing queries.

●       Integrate real-time data streams with backend systems, ensuring smooth data flow and synchronization between different components of the application.

●       Identify and address performance bottlenecks in data processing and event-driven applications, optimizing performance for scalability and efficiency.

●       Implement fault-tolerant and reliable systems, handling data inconsistencies and failures gracefully to ensure data integrity and system availability.

●       Implement stream processing solutions using frameworks like Apache Storm, Apache Beam, or Spring Cloud Stream, for real-time data processing and analysis.

●       Develop APIs for data access and manipulation, ensuring security, performance, and adherence to RESTful principles.

●       Implement monitoring and logging solutions to track system health, performance metrics, and event processing, using tools like Prometheus, Grafana, or ELK stack.

●       Implement security measures to protect data integrity and confidentiality, ensuring compliance with industry standards and regulations (e.g., GDPR, HIPAA).

●       Coordinate with project managers to estimate tasks, prioritize work, and meet project deadlines effectively.

●       Work in Agile/Scrum methodologies, participate in sprint planning, backlog grooming, and sprint reviews, and contribute to continuous improvement of development processes.

Skillset and experience

●       5+ years of professional experience in Java development

●       Proficiency in Java/Python programming language

●       Experience with data processing frameworks like Apache Spark, Apache Flink, or Hadoop for batch processing.

●       Familiarity with stream processing frameworks like Apache Kafka Streams, Apache Storm, or Spring Cloud Stream for real-time data processing.

●       Expertise in working with message brokers like Apache Kafka, RabbitMQ, or PubSub for event-driven architectures.

●       Strong understanding of both relational databases (e.g., MySQL, PostgreSQL) and NoSQL databases (e.g., MongoDB, Cassandra).

●       Proficiency in designing efficient database schemas, optimizing queries, and ensuring data consistency.

●       Experience in designing and developing RESTful APIs using JAVA/Python web frameworks like Spring Boot/Flask.

●       Familiarity with working in Kubernetes environments is advantageous.

●       Strong analytical and problem-solving skills, with the ability to troubleshoot complex issues and propose effective solutions.

●       Communication: Excellent written and verbal communication skills, with the ability to communicate technical concepts effectively to both technical and non-technical stakeholders.

●       Deep understanding of software engineering practices, Design Patterns, Data Structures, Algorithms

●       Strong sense of ownership, drive and ability to deliver results

●       2+ years of experience in product driven organization is must

●       A Bachelors or Masters degree in engineering from a reputed institute (preferably IITs, NITs, or other top engineering institutes)