Data Engineer Spark
Description For Data Engineer Spark
PALO IT is a global innovation consultancy and Agile software development company working across 10 offices and five continents. As a B Corp-certified company and World Economic Forum New Champion, we're dedicated to helping organizations embrace technology as a force for good.
As a Data Engineer Spark, you'll be at the forefront of designing and implementing robust, scalable data pipelines using Apache Spark. You'll work with cutting-edge big data technologies, collaborating with data scientists and analysts to deliver actionable insights. The role involves working with cloud platforms like AWS, Azure, or Google Cloud, ensuring data security, and optimizing data processing for performance and scalability.
We offer a unique environment focused on innovation and impact, with opportunities for international mobility, internal R&D projects, and knowledge sharing. Our company is committed to becoming climate net-zero and achieving 50% revenue from positive impact projects by 2025. You'll join a community of innovators working with Fortune 1000s, SMEs, and startups addressing the world's most complex challenges.
The ideal candidate brings 4+ years of data engineering experience, strong programming skills in Python, Scala, or Java, and expertise in distributed data processing frameworks. You'll work in an Agile environment where problem-solving skills and attention to detail are valued, alongside a commitment to doing the right thing and contributing to the team's continuous improvement.
Responsibilities For Data Engineer Spark
- Design, develop, and optimize large-scale data pipelines using Apache Spark
- Implement ETL processes to support data transformation and integration
- Collaborate with data scientists, analysts, and engineers to understand data requirements
- Develop data models, schemas, and storage solutions
- Optimize data processing for performance, scalability, and cost-effectiveness
- Work with cloud platforms to manage and deploy data workflows
- Ensure data security, quality, and governance
- Monitor and troubleshoot production pipelines
- Stay updated on advancements in big data technologies
Requirements For Data Engineer Spark
- 4+ years of experience in data engineering, with a strong focus on big data technologies
- Proficiency in Apache Spark and distributed data processing frameworks
- Experience in programming languages such as Python, Scala, or Java
- Strong understanding of data lakes, warehouses, and databases
- Hands-on experience with cloud platforms (AWS, Azure, or Google Cloud)
- Familiarity with workflow orchestration tools like Apache Airflow
- Deep knowledge of data formats such as Parquet, Avro, and ORC
- Experience with CI/CD pipelines and version control
- Solid understanding of data governance and security principles
- Excellent problem-solving skills and ability to work in an Agile environment
Benefits For Data Engineer Spark
- Stimulating working environments
- Unique career path
- International mobility
- Internal R&D projects
- Knowledge sharing
- Personalized training
- Entrepreneurship & intrapreneurship opportunities
