Bizoforce Hiring – Tech Lead – Software Engineer (Senior Databricks Engineer) | Santa Clara, California, USA

Bizoforce is hiring an experienced Senior Databricks Engineer with strong expertise in Databricks, Python, PySpark, and SQL to design, develop, and optimize large-scale data processing solutions. The ideal candidate will have hands-on experience building scalable data pipelines, implementing ETL/ELT processes, and working with cloud-based data platforms to support analytics and business intelligence initiatives. This role focuses on developing reliable data engineering solutions, optimizing Spark workloads, implementing data quality frameworks, and collaborating with data architects, analysts, and business stakeholders to deliver efficient, high-performance data solutions.

Experience (in Years)

• Required Experience: 8–15 years.
• Relevant Experience: 8+ years in data engineering, with strong hands-on experience in Databricks, PySpark, Python, and SQL.
• Technical Expertise: Experience developing scalable data pipelines, optimizing Spark jobs, and implementing cloud-based data processing solutions.

Required Skills

• Databricks: Strong hands-on experience with the Databricks platform.
• Programming: Expertise in Python, PySpark, and SQL.
• Apache Spark: Experience with Spark optimization techniques and performance tuning.
• ETL/ELT: Experience designing, developing, and maintaining data integration and transformation pipelines.
• Data Processing: Experience handling large volumes of structured and unstructured data.
• Delta Lake: Knowledge of Delta Lake, Spark SQL, and Databricks workflows.
• Data Modeling: Strong understanding of data warehousing concepts and dimensional modeling.
• Data Quality: Experience implementing validation, monitoring, and data quality frameworks.
• Version Control and Deployment: Familiarity with Git, CI/CD pipelines, and deployment best practices.
• Agile Methodologies: Understanding of Agile development practices.
• Problem-Solving: Strong analytical, troubleshooting, and performance optimization skills.
• Communication: Excellent communication and stakeholder management skills.

Preferred Skills / Good-to-Have Technical Experience

• Experience with cloud platforms such as Microsoft Azure, AWS, or Google Cloud Platform (GCP).
• Knowledge of Azure Data Factory (ADF), Apache Airflow, or other orchestration tools.
• Experience with streaming technologies such as Apache Kafka or Spark Structured Streaming.
• Exposure to Power BI, Tableau, or other business intelligence and analytics tools.
• Databricks or cloud platform certifications.
• Experience with cloud data lakes, data warehouses, and modern data engineering architectures.

Qualifications

• Bachelor's or Master's degree in Computer Science, Information Technology, or a related field.
• At least 8 years of experience in data engineering.
• Demonstrated expertise in Databricks, PySpark, Python, and SQL.
• Proven ability to develop and optimize scalable ETL/ELT pipelines.
• Strong understanding of data transformations, data modeling, and cloud-based data platforms.
• Ability to troubleshoot production issues and deliver timely resolutions.
• Excellent analytical, communication, collaboration, and stakeholder management skills.

Key Responsibilities

Data Pipeline Development

• Design, develop, and maintain scalable data pipelines using Databricks, PySpark, Python, and SQL.
• Build and optimize ETL/ELT frameworks for processing large volumes of structured and unstructured data.
• Develop data ingestion solutions from multiple source systems into cloud data lakes and warehouses.
• Implement efficient data transformation processes to support reporting, analytics, and business intelligence.

Performance Optimization

• Optimize Spark jobs for performance, scalability, and cost efficiency.
• Apply Spark optimization techniques to improve processing speed and resource utilization.
• Monitor pipeline performance and identify opportunities for improvement.
• Use Delta Lake, Spark SQL, and Databricks workflows to support reliable data processing.

Data Quality and Reliability

• Implement data quality checks, validation processes, and monitoring frameworks.
• Ensure data accuracy, consistency, and reliability across pipelines.
• Troubleshoot production issues and provide timely resolutions.
• Follow best practices for coding, testing, version control, and deployment.

Collaboration and Stakeholder Management

• Collaborate with data architects, analysts, and business stakeholders to understand data requirements.
• Translate business and analytical requirements into effective data engineering solutions.
• Coordinate with cross-functional teams to support data platform initiatives.
• Communicate technical findings, issues, and recommendations clearly.

Continuous Improvement

• Identify opportunities to improve data processing efficiency and pipeline reliability.
• Apply appropriate cloud services and orchestration tools to streamline workflows.
• Follow Agile methodologies and contribute to continuous improvement initiatives.
• Stay updated on emerging data engineering technologies and best practices.

Assignment Details

• Job Location: Santa Clara, California, USA
• Work Mode: On-site — customer location attendance required from Day 1
• Job Type: Full Time
• Open Positions: Multiple
• Working Hours: 9:00 AM – 5:00 PM
• Hours per Week: 40 hours
• Timesheet Cycle: Weekly
• Interview Process: In-person interview after clearing the first-level interview
• Job Status: Active