Role and Responsibilities

·        Identify, design, and implement internal process improvements: automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability, etc.

·        Working knowledge of message queuing, stream processing, and highly scalable big data data stores (Kafka, Kinesis, Storm) and ETL orchestration tools such as Airflow.

·        Working SQL knowledge and experience working with relational databases, query authoring (SQL) as well as working familiarity with a variety of databases (SQL Server, Oracle, Athena, RDS, Redshift) AWS services (EC2, EMR, Glue, Lambda, etc.

·        Experience with JSON and XML schemas and documents

·         Preference for deep understanding of Spark, Databricks and Delta Lake, and applying them to solve data science and machine learning business problems.

·        Build, and test an optimal data pipeline architecture (preferably on any cloud environments –AWS experience is a must)

Experience & Skillset Required

·        10-15 years of overall IT experience

·        6 Years of Data Ingestion engineer, using Python, Lambda, other ingestion tools

·        3-4 years of Experience with Spark, Scala, Kafka

·        Experience in Python data structures and data engineering libraries is mandatory

AWS/Azure Devops - CI/CD.

·        Experience in Python data structures and data engineering libraries is mandatory

AWS/Azure Devops - CI/CD.