Key Responsibilities  

Job Specific Responsibilities  

● Define, design, and develop services and solutions around large data ingestion, storage, and management such as with RDBMS, No SQL DBs, Log Files, and Events. 

● Define, design, and run robust data pipelines/batch jobs in a production environment. 

● Develop highly scalable, highly concurrent, and low-latency systems. 

● Strong analytic skills related to working with unstructured datasets. 

● Work with third-party and other internal provider services to support a variety of integrations. 

● Design and implement workflows and data pipelines using tools like Apache Nifi. 

● Build and manage messaging and streaming systems using Kafka. 

● Create batch processing pipelines using Spark and real-time stream processing solutions with Spark Streaming or Flink. 

● Work with product teams on a range of tools and services, improving products to meet user needs. 

● Participate in sprint planning to work with developers and project teams to ensure projects are deployable and able to monitor from the outside. 

 Education, Experience and Required Skills  

Educational Qualification Degree in Computer Science, Software Engineering or related preferred Experience Minimum 3 - 5 years experience as a Big Data Engineer 

Essential Skills  

▪ Experience with Apache Nifi for designing workflows and managing data pipelines. 

 ▪ Proficiency in working with Kafka for messaging and streaming use cases.  

▪ Strong understanding of Spark for batch processing and Flink for real-time stream processing. 

▪ Familiarity with Debezium for change data capture (CDC) and real-time data synchronization.  

▪ Good understanding of data architecture principles. 

▪ Experience of big data environments (also advising best practices/new technologies to Analytics team) ▪ Experience of Storing Data in systems such as S3(minio), Kafka. 

▪ Familiarity with the basic principles of distributed computing and data modeling 

 ▪ Working knowledge of message queuing, streaming processing, and highly scalable data stores 

 ▪ Scripting or programming skills. 

 ▪ In-depth knowledge of Spark, Kafka, Airflow, Apache Avro, Parquet, ORC. 

▪ Good understanding of OLAP data models design 

▪ Experience working in an agile environment. 

▪ Knowledge of the use of version control systems such as git. 

 Desirable Skills 

 ▪ Experience in large-scale Analytics, Insight, Decision Making, and Reporting solutions, based on Big Data technology 

 ▪ Good understanding of data architecture principles 

 ▪ Understanding of infrastructure (including hosting, container based deployments and storage architectures) would be advantageous