Senior Data Engineer

About the role

We are seeking a Senior Data Engineer to design, develop, and optimise large‑scale data solutions on the Databricks platform. The role involves leading migration projects, building robust ETL pipelines, and collaborating closely with clients to translate business needs into technical implementations.

Key responsibilities

• Design and implement scalable data engineering solutions using Databricks, Apache Spark, PySpark, and SQL.
• Lead legacy‑to‑Databricks migration projects, ensuring smooth transition and minimal disruption.
• Develop, optimise, and maintain ETL/ELT workflows and data processing frameworks.
• Apply Databricks architecture best practices, clustering techniques, performance tuning, and workload optimisation.
• Work with cloud platforms such as Azure, AWS, or GCP to provision and manage resources.
• Implement CI/CD pipelines for data engineering workloads and maintain DevOps standards.
• Collaborate with clients and stakeholders to gather requirements and deliver technical solutions.
• Provide technical guidance, architecture recommendations, and mentorship to junior engineers.
• Evaluate and improve existing data platforms, pipelines, and processing frameworks.
• Explore emerging GenAI capabilities within Databricks, including Databricks Genie and Apps where relevant.

Required profile

• 7+ years of data engineering experience, including 3+ years in a consulting environment.
• Proven delivery of at least 5 Databricks projects.
• Strong client‑facing, communication, problem‑solving, and leadership abilities.
• Ability to independently own and deliver complex projects.

Required skills

• Databricks
• Apache Spark
• PySpark
• SQL
• Legacy‑to‑Databricks migration
• Databricks architecture and performance optimisation
• Clustering and workload tuning
• Cloud platforms – Azure, AWS, GCP
• CI/CD and DevOps practices