Data Engineer Spark

Global innovation consultancy and Agile software development company dedicated to helping organizations embrace tech as a force for good.
Data
Senior Software Engineer
Hybrid
501 - 1,000 Employees
4+ years of experience
Enterprise SaaS · AI

Description For Data Engineer Spark

PALO IT is a global innovation consultancy and Agile software development company working across 10 offices and five continents. As a B Corp-certified company and World Economic Forum New Champion, we're dedicated to helping organizations embrace technology as a force for good.

As a Data Engineer Spark, you'll be at the forefront of designing and implementing robust, scalable data pipelines using Apache Spark. You'll work with cutting-edge big data technologies, collaborating with data scientists and analysts to deliver actionable insights. The role involves working with cloud platforms like AWS, Azure, or Google Cloud, ensuring data security, and optimizing data processing for performance and scalability.

We offer a unique environment focused on innovation and impact, with opportunities for international mobility, internal R&D projects, and knowledge sharing. Our company is committed to becoming climate net-zero and achieving 50% revenue from positive impact projects by 2025. You'll join a community of innovators working with Fortune 1000s, SMEs, and startups addressing the world's most complex challenges.

The ideal candidate brings 4+ years of data engineering experience, strong programming skills in Python, Scala, or Java, and expertise in distributed data processing frameworks. You'll work in an Agile environment where problem-solving skills and attention to detail are valued, alongside a commitment to doing the right thing and contributing to the team's continuous improvement.

Last updated 3 months ago

Responsibilities For Data Engineer Spark

  • Design, develop, and optimize large-scale data pipelines using Apache Spark
  • Implement ETL processes to support data transformation and integration
  • Collaborate with data scientists, analysts, and engineers to understand data requirements
  • Develop data models, schemas, and storage solutions
  • Optimize data processing for performance, scalability, and cost-effectiveness
  • Work with cloud platforms to manage and deploy data workflows
  • Ensure data security, quality, and governance
  • Monitor and troubleshoot production pipelines
  • Stay updated on advancements in big data technologies

Requirements For Data Engineer Spark

Python
Java
  • 4+ years of experience in data engineering, with a strong focus on big data technologies
  • Proficiency in Apache Spark and distributed data processing frameworks
  • Experience in programming languages such as Python, Scala, or Java
  • Strong understanding of data lakes, warehouses, and databases
  • Hands-on experience with cloud platforms (AWS, Azure, or Google Cloud)
  • Familiarity with workflow orchestration tools like Apache Airflow
  • Deep knowledge of data formats such as Parquet, Avro, and ORC
  • Experience with CI/CD pipelines and version control
  • Solid understanding of data governance and security principles
  • Excellent problem-solving skills and ability to work in an Agile environment

Benefits For Data Engineer Spark

Education Budget
  • Stimulating working environments
  • Unique career path
  • International mobility
  • Internal R&D projects
  • Knowledge sharing
  • Personalized training
  • Entrepreneurship & intrapreneurship opportunities

Interested in this job?

Jobs Related To PALO IT Data Engineer Spark

Senior Data Engineer

Senior Data Engineer role at PALO IT, building and maintaining enterprise data solutions while contributing to positive global impact.

Senior Data Scientist, Research, Ads Metrics

Senior Data Scientist position at Google focusing on ads metrics research, offering competitive compensation and the opportunity to shape advertising products.

Senior Data Scientist, Research, Workspace

Senior Data Scientist position at Google focusing on Workspace products, requiring 5+ years of analytics experience and expertise in Python, R, and SQL.

Senior Data Scientist, Product, Google Store

Senior Data Scientist position at Google Store focusing on product analytics, requiring 5+ years of experience in data analysis, statistics, and programming.

Senior Data Scientist, Google Play

Senior Data Scientist position at Google Play focusing on analytics, product decisions, and strategic insights using Python, R, and SQL.