Job Title: Data Engineer

Requirements:

• 3+ years experience with big data tools and technologies
• Proficiency in using Apache Spark for large-scale data processing
• Hands-on experience with either Scala or Python for Spark application development
• Experience with Spark SQL, Spark Streaming, and Spark MLlib
• Knowledge of cluster computing frameworks like Hadoop/YARN
• Experience with data pipelines, ETL, and data modeling
• Familiarity with relational and NoSQL databases
• Strong analytical skills and data intuition
• Excellent communication and collaboration skills

Responsibilities:

• Build and optimize Spark batch and streaming data pipelines
• Develop Scala/Python applications to process and analyze large datasets
• Employ Spark SQL and Spark MLlib for data analysis and machine learning
• Integrate Spark with relational and NoSQL databases like Cassandra, MongoDB
• Implement best practices for performance tuning, debugging, and troubleshooting Spark jobs
• Collaborate with cross-functional teams to understand business needs and implement data solutions
• Monitor and maintain big data infrastructure and clusters
• Keep up-to-date with latest developments in big data analytics and data engineering space