LLM Engineer (Data Platform)
Description For LLM Engineer (Data Platform)
42dot is seeking a Senior LLM Engineer for their Data Platform team to develop systems managing petabyte-scale text, image, and video data for Generative Model training. The role focuses on building efficient data management systems that integrate with ML training pipelines to reliably supply necessary data, thereby improving service quality. The position offers opportunities to work with cutting-edge AI technologies and data engineering tools, while developing technical leadership in LLM training data design and optimization. The ideal candidate will have strong experience in distributed systems, cloud platforms, and data engineering, with the ability to handle large-scale data processing and optimization. The role combines technical expertise in data engineering with AI/ML knowledge, offering a unique opportunity to impact large-scale AI model development.
Responsibilities For LLM Engineer (Data Platform)
- Design and develop data collection, processing, storage, and utilization pipeline for petabyte-scale text, image, and video data to improve model performance
- Generate and manage large-scale synthetic data to contribute to model training quality improvement
- Define data quality metrics and design/build automated quality verification and monitoring systems
- Maximize data preprocessing efficiency using industrial standard formats and tools like Parquet, WebDataset, TorchData, TFRecord, datatrove
- Automate version management and labeling processes for continuously changing datasets
- Develop storage and transmission technologies considering data integrity and security, comply with regulations and internal security policies
Requirements For LLM Engineer (Data Platform)
- 5+ years of experience in software/data engineering
- Experience with large-scale distributed processing environments (Spark, Hadoop)
- Development experience in cloud environments (AWS, GCP, Azure)
- Ability to use cloud-based storage and distributed processing platforms (S3, EMR, DataProc)
- Experience in optimizing large datasets through compression, indexing, and sharding
- Strong software engineering capabilities with high proficiency in programming languages including Python and C++
- Understanding of model training, preprocessing, optimization processes and collaboration skills
Jobs Related To 42dot LLM Engineer (Data Platform)
