Data Engineer

Data Engineer (Demographic & Geospatial Data) Location: Remote working Type: Contract or Permanent Day Rate: Up to £550 per day Inside IR35 Salary: Up to £90,000 package Security Clearance: SC Eligible (Mandatory) / Active SC Preferred Start Date: Immediate or up to 2 weeks' notice The Opportunity We are supporting a major programme seeking an experienced Data Engineer with strong Databricks and Geospatial Data expertise. You will play a key role in designing and delivering scalable data pipelines that combine demographic datasets with geographic reference data to support population analysis, reporting, service planning, and operational decision-making.

This is an excellent opportunity for a Data Engineer who enjoys working with large-scale data platforms, geospatial processing, and modern cloud-based technologies in a highly collaborative environment. Key Responsibilities

Design, build and support data pipelines using Databricks, Python, PySpark and SQL

Develop reusable ingestion, transformation, validation and enrichment processes

Ingest demographic datasets, postcode data, geographic boundaries and reference datasets

Build and maintain Delta Lake datasets and manage data lineage

Perform postcode enrichment and geographic assignment processing

Implement spatial joins, point-in-polygon matching and location-based calculations

Manage geographic reference data versions and historical reproducibility

Develop automated data quality controls and reconciliation processes

Optimise Spark workloads and pipeline performance

Support production releases, monitoring and troubleshooting activities

Work closely with analysts, data owners and platform teams to define business rules and acceptance criteria

Essential Skills & Experience

Strong commercial experience with

Databricks

Python

PySpark

SQL

Proven experience building and supporting production-grade data pipelines

Strong knowledge of Delta Lake, schema management, merge logic and incremental processing

Experience handling large datasets within cloud-based environments

Practical experience working with geospatial data and spatial processing

Understanding of coordinate reference systems, including

British National Grid (BNG)

WGS84

Experience with spatial joins, geometry validation and geographic aggregation

Strong data quality, reconciliation and testing experience

Experience with Git, CI/CD and automated deployment pipelines

Desirable Experience

Apache Sedona or Databricks Spatial Functions

GeoPandas, Shapely, pyproj or QGIS

NHS or healthcare data

NHS Postcode Directory (NHSPD)

ONS Postcode Directory (ONSPD)

National Statistics Postcode Lookup (NSPL)

Experience working with

LSOAs

MSOAs

Local Authorities

Integrated Care Boards (ICBs)

GeoJSON, GeoParquet, Shapefiles, WKT/WKB formats

Population, census or demographic datasets

Experience working within regulated or secure environments

What We're Looking For The ideal candidate will have a strong Data Engineering background combined with experience working with geographic and location-based datasets. You will be comfortable building scalable Databricks solutions, implementing automated processes, and ensuring the accuracy and reliability of demographic and geospatial data.

Candidates must be eligible to obtain SC Clearance, with active clearance highly advantageous. What's on OfferContract

Up to £550 per day

Inside IR35

Predominantly remote working

Permanent

Up to £90,000 total package

Base salary up to approximately £81,000

Performance bonus

Predominantly remote working