Data Engineer

ESSENTIAL ROLES & RESPONSIBILITIES

Identify and understand customer data-centric use cases within regulated financial services environments

Design and implement data ingestion, processing, and transformation pipelines on Azure

Build and maintain data pipelines for cleaning, normalisation, enrichment, and preparation

Apply appropriate data modelling techniques and architecture patterns, with a strong focus on medallion architecture

Orchestrate, monitor, and optimise Azure Databricks jobs and Azure Data Factory pipelines across development, UAT, and production environments

Configure platforms, clusters, and compute resources to optimise performance, cost, and reliability

Use automated CI/CD pipelines to manage, deploy, and version data artefacts and pipelines

Operationalise workflows developed by analysts and data scientists

Support customers in adopting Azure data, analytics, and machine learning services

Ensure secure storage, processing, and quality of customer data

Ensure networking and security best practices are applied when designing and operating data solutions

Design solutions for processing large volumes of data using batch and streaming approaches

Collaborate with analytics teams on data visualisation best practices and reporting enablement

Ensure all solutions are well-documented, including pipelines, schemas, transformations, and operational runbooks

GOVERNANCE & REPORTING

Maintain accurate documentation of data pipelines, schemas, transformations, and deployment processes

Support data governance initiatives including lineage, metadata management, and access control

Contribute to service reporting, risk tracking, and continuous improvement actions

Ensure data environments are audit-ready and aligned with governance standards

TECHNOLOGY STACK (AZURE)

Cloud Platform

Microsoft Azure

Data Engineering & Analytics

Azure Databricks (development, UAT, and production)

Azure Data Factory

Azure Synapse Analytics (where applicable)

Machine Learning & AI

Azure Machine Learning (limited non-production usage)

Azure Document Intelligence

Databases

Microsoft SQL Server / Azure SQL Database (primary platforms)

PostgreSQL (limited use)

MySQL (limited use)

Data Processing

Batch and streaming data pipelines

Security & Governance

Role-based access control (RBAC)

Data encryption and key management

Audit logging and monitoring

DevOps

CI/CD pipelines for data artefacts and infrastructure

BEHAVIOURAL COMPETENCIES – ORGANISATIONAL & BEHAVIOURAL FIT

Positive mindset and enthusiasm for learning new technologies

Collaborative and supportive team player

Strong sense of ownership and accountability

Methodical, analytical approach to problem-solving

Strong understanding of ethical data usage in regulated environments

CRITICAL COMPETENCIES – TECHNICAL FIT

Essential

Strong SQL skills

Programming experience with Python and/or Scala

Hands-on experience with Azure-based data platforms

Experience designing, building, and maintaining data pipelines

Strong understanding of data modelling (relational and analytical), including medallion architecture

Experience orchestrating and optimising Databricks and Data Factory workloads

Experience using CI/CD pipelines for data and analytics solutions

Strong awareness of security, networking best practices, GDPR, and PII handling

Desirable

Experience with Azure Databricks in production environments

Familiarity with Azure Machine Learning and AI services

Exposure to data visualisation tools (e.g. Power BI)

Experience with big data frameworks (Spark, Kafka)

Knowledge of data governance, lineage, and metadata tooling

SHIFT & WORKING PATTERN

Standard business hours, with participation in an on-call rota as required

Occasional weekend engineering coverage will be required, typically limited to a small number of planned weekends per year to support business continuity, resilience testing, or disaster recovery activities