Databricks Developer

About the role

StataAI is seeking a skilled Databricks Developer to drive cloud‑based data engineering initiatives within the healthcare sector. You will design, build, and maintain robust data pipelines that transform raw data into high‑quality datasets for clinical operations, analytics, and reporting.

Key responsibilities

• Develop, test, and maintain ETL/ELT pipelines using Azure Databricks, PySpark, Spark SQL, and Databricks notebooks.
• Build and optimise ingestion and transformation workflows with Azure Data Factory and Databricks Jobs.
• Implement Delta Lake tables and Medallion architecture (Bronze, Silver, Gold).
• Integrate data from SQL databases, REST APIs, SFTP, and file systems into Azure Data Lake Storage (ADLS Gen2).
• Support Enterprise Data Warehouse, lakehouse, and data mart development.
• Perform data profiling, cleansing, validation, and quality checks.
• Apply data governance, metadata management, and lineage practices.
• Write and optimise SQL queries for pipeline performance.
• Collaborate with technical, clinical, and business teams to translate requirements into solutions.
• Participate in code reviews, unit/integration testing, and CI/CD processes.

Required profile

• Minimum 2 years of hands‑on experience with Azure Databricks, PySpark, and Spark SQL.
• Experience building data pipelines in Azure Data Factory and managing Delta Lake.
• Familiarity with healthcare data standards and regulatory requirements such as HIPAA and GDPR.

Required skills

• Azure Databricks
• PySpark
• Spark SQL
• Azure Data Factory
• Delta Lake & Medallion architecture
• SQL
• Azure Data Lake Storage (ADLS Gen2)
• REST APIs
• SFTP