AI Evaluation Engineer

AI Evaluation Engineer Rate / Duration / Location £700-£750/day Inside IR35 | Duration: 4 months | Clearance: BPSS + SC eligible | Location: London, Bristol or Manchester | Hybrid: 2 days onsite per week. Overview We're looking for an experienced AI Evaluation Engineer to join a specialist Public Sector Technology team. This hands-on role focuses on building evaluation frameworks, tooling and harnesses for AI systems, particularly LLM and agentic AI solutions. The role requires ownership, experimentation and rapid delivery. Key Responsibilities

Design evaluation frameworks and tooling

Develop evaluation harnesses

Evaluate model and agentic AI systems

Define metrics and methodologies

Build repeatable evaluation processes

Investigate AI failures

Work directly with government teams

Prototype and test solutions

Present findings and recommendations

Contribute to strategic direction.

What We Are Looking For

Strong software engineering experience

Experience building and evaluating AI systems

Understanding of LLMs and agentic AI

Experience with AI evaluation frameworks

Ability to code independently

Strong problem-solving and communication skills

Comfortable working autonomously.

Technical Experience AI/LLM Evaluation, RAG Evaluation, Ragas, Agentic AI Evaluation, Evaluation Harnesses, LLM Testing and Benchmarking, Prompt Engineering, Python Development, Model and Agent Integration. Ideal Background AI Evaluation Engineer, Applied AI Engineer, AI Engineer, LLM Engineer, Harness Engineer, Prompt Engineer or Software Engineer specialising in AI. Team Small Public Sector Technology team of approximately 2 to 6 people within a wider organisation. What Will Make Someone Stand Out Experience building evaluation tooling, evaluating LLMs, using Ragas, identifying AI failure modes, combining engineering with strategic thinking, and thriving in fast-moving environments. Summary Hands-on AI Evaluation Engineer role supporting government AI initiatives through evaluation, testing, benchmarking and framework development.