Sr Big Data Engineer (GCP)- Airflow and Oozie

We are seeking a highly skilled and experienced Senior Big Data Engineer to join our dynamic team. The ideal candidate will have a strong background in developing batch processing systems, with extensive experience in Oozie, the Apache Hadoop ecosystem, Airflow, and a solid understanding of public cloud technologies, especially GCP. This role involves working in a remote environment, requiring excellent communication skills and the ability to solve complex problems independently and creatively.

Key responsibilities include:

• Developing scalable and robust code for batch processing systems using technologies like Hadoop, Oozie, Pig, Hive, Map Reduce, Spark (Java), Python, and Hbase.
• Developing, managing, and optimizing data workflows using Oozie and Airflow within the Apache Hadoop ecosystem.
• Leveraging GCP for scalable big data processing and storage solutions.
• Implementing automation/DevOps best practices for CI/CD and Infrastructure as Code.

Requirements:

• Experience with GCP managed services and understanding of cloud-based batch processing systems.
• Proficiency in Oozie, Airflow, Map Reduce, and Java.
• Strong programming skills with Java (specifically Spark), Python, Pig, and SQL.
• Expertise in public cloud services, particularly in GCP.
• Proficiency in the Apache Hadoop ecosystem with Oozie, Pig, Hive, and Map Reduce.
• Familiarity with BigTable and Redis.
• Experience in Infrastructure and Applied DevOps principles, including CI/CD and IaC tools like Terraform.
• Ability to tackle complex challenges and devise effective solutions.
• Proven experience in working effectively in a remote setting with strong communication skills.
• Hands-on experience in engineering batch processing systems at scale.

This role offers an exciting opportunity to work with cutting-edge big data technologies in a cloud environment, contributing to scalable and efficient data processing solutions.