Back to jobs

Senior Databricks Engineer

Job description

Senior Data Engineer - Databricks & Data Vault

My client is expanding its data engineering capability and is seeking a Senior Data Platform Engineer to design and deliver modern analytical solutions built on Databricks and Data Vault 2.0 principles.

This role sits at the intersection of architecture, engineering, and delivery. You will shape scalable data platforms, develop enterprise-grade pipelines, and help establish best-practice frameworks that support analytics, reporting, and advanced data use cases across multiple business domains.

You will collaborate with architects, platform specialists, and delivery teams to turn complex business needs into reliable, governed, and high-performance data products. Prior exposure to general insurance or financial services environments is advantageous.

What You'll Do

Enterprise Data Modelling

  • Design and build Data Vault 2.0 structures including Hubs, Links, and Satellites across Raw and Business Vault layers.

  • Define and apply modelling standards for keys, hashing, historisation, and late-arriving data.

  • Contribute to reference architectures and modelling guidelines.

Databricks Engineering

  • Develop batch and streaming pipelines using Spark (PySpark/Scala) on Databricks.

  • Implement ingestion, transformation, and curated (gold/semantic) layers using Delta Lake.

  • Tune workloads for performance, reliability, and cost efficiency.

Platform & Delivery Practices

  • Implement CI/CD pipelines and source control integration for notebooks and code.

  • Promote production-grade engineering with testing, monitoring, and automated deployments.

  • Ensure secure handling of sensitive data and appropriate access controls.

Data Quality & Governance

  • Embed automated data validation and reconciliation checks.

  • Support lineage, auditing, and metadata management.

  • Contribute to governance frameworks and controlled data sharing practices.

Leadership & Collaboration

  • Provide technical guidance to engineers through reviews, design sessions, and mentoring.

  • Partner with architects and stakeholders to shape target-state solutions.

  • Champion engineering standards and reusable patterns.

What You'll Bring

Essential

  • Extensive experience in data engineering with multiple years delivering solutions on Databricks.

  • Proven hands-on delivery of Data Vault 2.0 implementations.

  • Strong Spark and SQL skills with experience tuning large-scale workloads.

  • Solid understanding of modern data integration and modelling concepts.

  • Experience building reliable, well-tested, and observable data pipelines.

Desirable

  • Exposure to Unity Catalog or enterprise metadata/catalog tooling.

  • Experience with metadata-driven or automated Data Vault loading frameworks.

  • Familiarity with at least one major cloud platform (Azure, AWS, or GCP).

  • Experience with orchestration tools, dbt, and event-streaming technologies.

  • Background in regulated or highly governed industries.

Preferred Certifications

  • Databricks Data Engineer certification (or equivalent).

  • Cloud platform certifications.

What Success Looks Like

  • A robust, scalable Data Vault foundation supporting enterprise analytics.

  • Standardised, reusable pipeline patterns with strong observability.

  • Clear documentation, engineering standards, and operational readiness.