Databricks Engineer
Description For Databricks Engineer
ShyftLabs, a dynamic data product company established in 2020, is seeking a skilled Databricks Engineer to join their team. The company has built a strong reputation working with Fortune 500 clients, delivering innovative digital solutions that drive business growth.
The role focuses on designing, developing, and optimizing big data solutions using the Databricks Unified Analytics Platform. As a Databricks Engineer, you'll be at the forefront of implementing scalable data architectures and ETL workflows, leveraging technologies like Apache Spark, SQL, and Python. The position requires expertise in cloud platforms (AWS/Azure/GCP) and offers the opportunity to work with cutting-edge data technologies.
The ideal candidate will bring 5+ years of hands-on experience with Databricks and Apache Spark, along with strong proficiency in data processing and analysis. You'll be responsible for implementing data governance, optimizing data lakehouse architectures, and ensuring high-performance data solutions. The role offers a collaborative environment where you'll work alongside data scientists and analysts to enable advanced AI/ML workflows.
ShyftLabs offers a competitive compensation package, including comprehensive insurance benefits and substantial learning and development opportunities. The hybrid work environment provides flexibility while maintaining collaborative team dynamics. This position is perfect for someone who is passionate about big data, enjoys solving complex technical challenges, and wants to be part of a growing company that's making an impact in the data solutions space.
Responsibilities For Databricks Engineer
- Design, implement, and optimize big data pipelines in Databricks
- Develop scalable ETL workflows to process large datasets
- Leverage Apache Spark for distributed data processing and real-time analytics
- Implement data governance, security policies, and compliance standards
- Optimize data lakehouse architectures for performance and cost-efficiency
- Collaborate with data scientists, analysts, and engineers to enable advanced AI/ML workflows
- Monitor and troubleshoot Databricks clusters, jobs, and performance bottlenecks
- Automate workflows using CI/CD pipelines and infrastructure-as-code practices
- Ensure data integrity, quality, and reliability in all pipelines
Requirements For Databricks Engineer
- Bachelor's or Master's degree in Computer Science, Data Engineering, or related field
- 5+ years of hands-on experience with Databricks and Apache Spark
- Proficiency in SQL, Python, or Scala for data processing and analysis
- Experience with cloud platforms (AWS, Azure, or GCP) for data engineering
- Strong knowledge of ETL frameworks, data lakes, and Delta Lake architecture
- Experience with CI/CD tools and DevOps best practices
- Familiarity with data security, compliance, and governance best practices
- Strong problem-solving and analytical skills
Benefits For Databricks Engineer
- Competitive salary
- Strong insurance package
- Learning and development resources
Jobs Related To ShyftLabs Databricks Engineer
