Data Engineer
Description For Data Engineer
PayPay, a rapidly growing FinTech company with over 65M users, is seeking a Data Engineer for their DaaS team. The role involves designing, implementing, and operating a robust Data Engineering Platform using cutting-edge technologies like Spark, Hudi, Delta Lake, Scala, and AWS suite of data tools. The ideal candidate will have 5+ years of experience as a Data Engineer, with expertise in Apache Hudi, Delta Lake, Spark, and Scala. They should be proficient in AWS services, data orchestration tools, and infrastructure as code (Terraform). The position offers a dynamic work environment with a diverse team from over 50 countries, flexible work arrangements (Work From Anywhere at Anytime), and comprehensive benefits. PayPay values innovation, risk-taking, and constant challenges, aiming to create a future beyond imagination. Join PayPay to grow, reach new dimensions, and contribute to a vision that no one else can imagine.
Responsibilities For Data Engineer
- Design, develop, and maintain scalable data ingestion pipelines using AWS Glue, Step Functions, Lambda, and Terraform
- Optimize and manage large scale data pipelines to ensure high performance, reliability, and efficiency
- Implement data processing workflows using Hudi, Delta Lake, Spark, and Scala
- Maintain and enhance Lakeformation and Glue Data Catalog for effective data management and discovery
- Collaborate with cross-functional teams to ensure seamless data flow and integration across the organization
- Implement best practices for observability, data governance, security, and compliance
Requirements For Data Engineer
- 5+ years experience as a Data Engineer or in a similar role
- Hands-on experience with Apache Hudi, Delta Lake, Spark, and Scala
- Experience designing, building, and operating a DataLake or Data Warehouse
- Knowledge of Data Orchestration tools such as Airflow, Dagster, Prefect
- Strong expertise in AWS services, including Glue, Step Functions, Lambda, and EMR
- Familiarity with change data capture tools like Canal, Debezium, and Maxwell
- Experience with data warehousing tools like AWS Athena, BigQuery, Databricks
- Experience in at least one primary language (e.g. Scala, Python, Java) and SQL (any variant)
- Experience with data cataloging and metadata management using AWS Glue Data Catalog, Lakeformation, or Unity Catalog
- Proficiency in Terraform for infrastructure as code (IaC)
- Strong problem-solving skills and ability to troubleshoot complex data issues
- Excellent communication and collaboration skills
- Ability to work in a fast-paced, dynamic environment and manage multiple tasks simultaneously
Benefits For Data Engineer
- Social Insurance (health insurance, employee pension, employment insurance and compensation insurance)
- 401K
- Translation/Interpretation support
- VISA sponsor + Relocation support
- Annual leave (up to 14 days in the first year)
- Personal leave (5 days each year)
- Late overtime allowance
- Work from anywhere allowance (JPY100,000)
- Super Flex Time (No Core Time)
Jobs Related To PayPay Data Engineer
