Data Engineer
Description For Data Engineer
PayPay is a rapidly growing FinTech company that has amassed over 65M users since its 2018 launch. As part of the DaaS (Data as a Service) team, you'll be at the forefront of designing, implementing, and operating a robust Data Engineering Platform to support PayPay's expanding business needs.
The role involves working with cutting-edge technologies including Spark, Hudi, Delta Lake, Scala, and AWS suite of data tools. You'll be responsible for building and maintaining scalable data ingestion pipelines, optimizing data workflows, and ensuring high performance and reliability of data systems.
Key technical aspects include:
- Working with AWS services (Glue, Step Functions, Lambda, EMR)
- Implementing data processing workflows using Hudi, Delta Lake, Spark, and Scala
- Managing Lakeformation and Glue Data Catalog
- Using infrastructure as code with Terraform
- Working with data orchestration tools and change data capture systems
The position offers a flexible work environment with their WFA (Work From Anywhere at Anytime) policy, allowing you to work remotely from anywhere in Japan. PayPay provides comprehensive benefits including social insurance, 401K, visa sponsorship, and relocation support.
You'll be joining a diverse, international team that values innovation and professional growth. The company culture emphasizes taking risks and challenging oneself, providing daily opportunities to grow and reach new dimensions. PayPay seeks professionals who can embrace challenges and contribute to the rapid evolution of their products with passion and expertise.
The work schedule is super flexible with no core time (nominally 10:00am-6:45pm), and you'll receive generous paid leave including annual leave and special personal leave. The compensation package includes a competitive annual salary paid monthly, with annual reviews and special incentives based on performance.
Responsibilities For Data Engineer
- Design, develop, and maintain scalable data ingestion pipelines using AWS Glue, Step Functions, Lambda, and Terraform
- Optimize and manage large scale data pipelines to ensure high performance, reliability, and efficiency
- Implement data processing workflows using Hudi, Delta Lake, Spark, and Scala
- Maintain and enhance Lakeformation and Glue Data Catalog for effective data management and discovery
- Collaborate with cross-functional teams to ensure seamless data flow and integration
- Implement best practices for observability, data governance, security, and compliance
Requirements For Data Engineer
- 5+ years experience as a Data Engineer or in a similar role
- Hands-on experience with Apache Hudi, Delta Lake, Spark, and Scala
- Experience designing, building, and operating a DataLake or Data Warehouse
- Knowledge of Data Orchestration tools such as Airflow, Dagster, Prefect
- Strong expertise in AWS services, including Glue, Step Functions, Lambda, and EMR
- Familiarity with change data capture tools like Canal, Debezium, and Maxwell
- Experience with data warehousing tools like AWS Athena, BigQuery, Databricks
- Experience in at least one primary language (e.g. Scala, Python, Java) and SQL
- Experience with data cataloging and metadata management
- Proficiency in Terraform for infrastructure as code
- Strong problem-solving skills and ability to troubleshoot complex data issues
- Excellent communication and collaboration skills
Benefits For Data Engineer
- Super Flex Time with no core hours
- Annual leave (up to 14 days in first year)
- Personal leave (5 days per year)
- Social Insurance
- 401K
- Translation/Interpretation support
- Visa sponsorship
- Relocation support
- Work from anywhere allowance (JPY100,000)
- Annual salary review
- Special incentive based on performance
Jobs Related To PayPay Data Engineer
