Lead Data Engineer

Our client, a leading technology firm, is currently seeking an experienced Lead Data Engineer to join their team in London. This is a permanent, full-time position. About the Opportunity

The successful candidate will architect, build, and scale the data infrastructure powering an advanced AI-driven platform. This hands-on leadership role involves designing scalable systems from the ground up, playing a critical role in building the data foundations that support AI products, analytics capabilities, machine learning workflows, and large-scale operational systems. You will work closely with AI engineers, backend developers, and product teams to ensure data is processed efficiently, reliably, and securely, while helping to define the future direction of the organisation's data architecture. As a senior member of the team, you will establish best practices, influence technical strategy, and contribute to the growth of the data function. Key Responsibilities:

Architect and build scalable data pipelines and infrastructure to support AI and product systems.

Design and maintain data ingestion, transformation, and storage architectures for operational and AI workloads.

Develop and manage batch and real-time data processing pipelines.

Build and optimise systems for vector search, data retrieval, and machine learning workflows.

Ensure data reliability, security, governance, and compliance across the platform.

Collaborate closely with AI and backend engineering teams to support model training, inference, and product development.

Implement monitoring, observability, and data quality frameworks.

Optimise the performance of large-scale datasets, data warehouses, and query systems.

Contribute to technical architecture decisions and long-term data strategy.

Mentor engineers and help establish engineering standards, processes, and best practices.

Job Requirements:Core Experience

Experience in Data Engineering, Software Engineering, or Backend Engineering.

Proven experience designing and building scalable data pipelines and distributed data systems.

Strong experience with relational databases, preferably PostgreSQL.

Experience working with NoSQL databases.

Experience with vector databases utilised in modern AI applications.

Strong Python development skills.

Data & Infrastructure

Experience with some of the following technologies is highly desirable: Frameworks & Infrastructure:

Apache Spark

Apache Airflow

Kafka

Elasticsearch / OpenSearch

Databases

PostgreSQL

MongoDB

Vector databases such as Qdrant, Milvus, or pgvector

Python Ecosystem

Pandas

Polars

Engineering Skills

Experience building highly scalable backend systems.

Strong understanding of data modelling and storage architecture.

Deep knowledge of data processing, optimisation, and performance tuning.

Experience implementing reliable and maintainable data engineering practices.

Nice to Have

Experience with JavaScript and Node.js.

Experience working on AI, machine learning, or generative AI platforms.

Familiarity with stream-processing and event-driven architectures.

Experience with cloud platforms such as AWS, Azure, or Google Cloud.

Previous experience within a high-growth startup or scale-up environment.

What You'll Gain

Opportunity to shape the data architecture of a growing AI-driven platform.

Significant ownership and influence over technical direction.

Exposure to modern AI, machine learning, and data engineering technologies.

Collaborative environment where innovation and impact are highly valued.

Strong career growth potential within an ambitious and scaling business.

If you are an experienced Lead Data Engineer looking for a new challenge and the opportunity to have a significant impact, we would love to hear from you. Apply now to join our client's dynamic and innovative team in London.