Data Engineer – Batch & Streaming Pipelines

About the role

We are looking for a Data Engineer to design, develop and maintain scalable batch and streaming data pipelines. The role focuses on building production‑ready solutions using Python and Apache Spark while integrating with cloud data sources such as Amazon S3.

Key responsibilities

• Design, develop and optimise batch and streaming pipelines with Python, Apache Spark and Spark SQL.
• Write clean, modular, well‑tested code and implement CI/CD workflows for data‑engineering projects.
• Integrate data platforms with external APIs, cloud services and storage (e.g., Amazon S3).
• Apply data validation, testing, monitoring and quality practices throughout the pipeline lifecycle.
• Collaborate with engineers, stakeholders and cross‑functional teams to translate business requirements into reliable data solutions.
• Leverage AI‑assisted development tools to improve productivity and code quality.

Required profile

• Strong hands‑on experience with Apache Spark / Spark SQL.
• Proficient Python programming with a focus on production‑ready code.
• Experience building both batch and streaming data pipelines.
• Familiarity with CI/CD pipelines, data QA, validation and monitoring.
• Ability to work with APIs and cloud‑based data sources such as Amazon S3.
• Excellent problem‑solving, communication and collaboration skills.

Required skills

• Python
• Apache Spark
• Spark SQL
• SQL
• Amazon S3
• APIs
• Git
• CI/CD
• Databricks Lakehouse Platform (preferred)
• Delta Lake, Unity Catalog, DLT/Lakeflow (preferred)
• Databricks Asset Bundles (DAB) and YAML deployment configurations (preferred)
• AI‑assisted development tools (preferred)