Databricks Developer – Azure Cloud Data Engineer (Healthcare)

About the role

The Cloud Data Engineer (Azure Databricks) will design, build and maintain data pipelines that support clinical, operational and analytical teams in the healthcare domain. Working under senior data‑engineering leadership, the role focuses on delivering reliable, high‑quality data through the Enterprise Data Warehouse and Lakehouse architecture.

Key responsibilities

• Develop and operate batch and streaming ETL/ELT pipelines on Azure Databricks using PySpark, Spark SQL and notebooks.
• Implement Delta Lake best practices (OPTIMIZE, Z‑ORDER, VACUUM, schema evolution) and manage Medallion architecture layers.
• Orchestrate workflows with Databricks Jobs and integrate them with Azure Data Factory for end‑to‑end pipeline execution.
• Ingest data from databases, SFTP, file systems and REST APIs into ADLS Gen2 and Azure Blob Storage.
• Collaborate on data modeling, data quality testing and documentation of data lineage.

Required profile

• Hands‑on experience with Azure Databricks, PySpark and Delta Lake.
• Solid understanding of SQL and Azure data services (SQL Database, Synapse Analytics, ADLS Gen2).
• Familiarity with data‑engineering concepts such as Medallion architecture, data modeling and governance tools (Unity Catalog).
• Ability to work in a healthcare environment and follow coding standards and testing practices.

Required skills

• Azure Databricks
• PySpark & Spark SQL
• Databricks notebooks & Workflows
• Delta Lake
• Azure Data Factory
• Medallion architecture (bronze/silver/gold)
• Unity Catalog
• SQL (Azure SQL Database, Synapse, Databricks SQL)
• Azure Data Lake Storage (ADLS Gen2) & Azure Blob Storage
• SFTP & REST API integration
• Data modeling fundamentals
• Azure Machine Learning basics
• Data normalization/denormalization
• ER/Studio or Erwin (optional)
• Healthcare vocabularies (SNOMED CT, LOINC, ICD‑10, RxNorm – optional)