Data Engineer Spark

Global innovation consultancy and Agile software development company dedicated to helping organizations embrace tech as a force for good.
Data
Senior Software Engineer
Hybrid
501 - 1,000 Employees
4+ years of experience
Enterprise SaaS · AI

Description For Data Engineer Spark

PALO IT is a global innovation consultancy and Agile software development company working across 10 offices and five continents. As a B Corp-certified company and World Economic Forum New Champion, we're dedicated to helping organizations embrace technology as a force for good.

As a Data Engineer Spark, you'll be at the forefront of designing and implementing robust, scalable data pipelines using Apache Spark. You'll work with cutting-edge big data technologies, collaborating with data scientists and analysts to deliver actionable insights. The role involves working with cloud platforms like AWS, Azure, or Google Cloud, ensuring data security, and optimizing data processing for performance and scalability.

We offer a unique environment focused on innovation and impact, with opportunities for international mobility, internal R&D projects, and knowledge sharing. Our company is committed to becoming climate net-zero and achieving 50% revenue from positive impact projects by 2025. You'll join a community of innovators working with Fortune 1000s, SMEs, and startups addressing the world's most complex challenges.

The ideal candidate brings 4+ years of data engineering experience, strong programming skills in Python, Scala, or Java, and expertise in distributed data processing frameworks. You'll work in an Agile environment where problem-solving skills and attention to detail are valued, alongside a commitment to doing the right thing and contributing to the team's continuous improvement.

Last updated 2 months ago

Responsibilities For Data Engineer Spark

  • Design, develop, and optimize large-scale data pipelines using Apache Spark
  • Implement ETL processes to support data transformation and integration
  • Collaborate with data scientists, analysts, and engineers to understand data requirements
  • Develop data models, schemas, and storage solutions
  • Optimize data processing for performance, scalability, and cost-effectiveness
  • Work with cloud platforms to manage and deploy data workflows
  • Ensure data security, quality, and governance
  • Monitor and troubleshoot production pipelines
  • Stay updated on advancements in big data technologies

Requirements For Data Engineer Spark

Python
Java
  • 4+ years of experience in data engineering, with a strong focus on big data technologies
  • Proficiency in Apache Spark and distributed data processing frameworks
  • Experience in programming languages such as Python, Scala, or Java
  • Strong understanding of data lakes, warehouses, and databases
  • Hands-on experience with cloud platforms (AWS, Azure, or Google Cloud)
  • Familiarity with workflow orchestration tools like Apache Airflow
  • Deep knowledge of data formats such as Parquet, Avro, and ORC
  • Experience with CI/CD pipelines and version control
  • Solid understanding of data governance and security principles
  • Excellent problem-solving skills and ability to work in an Agile environment

Benefits For Data Engineer Spark

Education Budget
  • Stimulating working environments
  • Unique career path
  • International mobility
  • Internal R&D projects
  • Knowledge sharing
  • Personalized training
  • Entrepreneurship & intrapreneurship opportunities

Interested in this job?

Jobs Related To PALO IT Data Engineer Spark

Senior Data Engineer

Senior Data Engineer role at PALO IT, building and maintaining enterprise data solutions while contributing to positive global impact.

People Insights Manager

Senior People Insights Manager role at McDonald's, leveraging data analytics to drive strategic HR decisions globally.

Senior Software Engineer - Trading Data Fabric

Senior Software Engineer position at Belvedere Trading, focusing on building and managing data and research platforms for high-volume trading operations using cloud technologies.

LLM Engineer (Data Platform)

Senior LLM Engineer position at 42dot focusing on petabyte-scale data platform development for AI model training, requiring expertise in distributed systems and data engineering.

Senior Data Engineer

Senior Data Engineer position at Titan Wealth's Cape Town Tech Hub, focusing on Azure data solutions with hybrid work options and comprehensive benefits.