AI Automation Tester

Overview

We are looking for an AI Automation Tester to support IBM on a major programme of work

The role will focus on designing and executing automated testing across an AI-enabled solution, covering traditional functional and integration testing alongside the additional challenges involved in validating AI-generated responses and behaviours.

You'll work as part of the wider delivery team to ensure the solution is reliable, performs as expected and continues to meet requirements as it develops.

Key Responsibilities

Design, build and maintain automated tests across the AI solution

Execute functional, integration and regression testing

Develop and maintain automated test suites

Perform API testing across integrated services

Validate AI-generated responses and system behaviours

Design appropriate test scenarios for AI functionality

Manage and maintain test data required across different testing activities

Test integrations between the AI solution and wider systems/services

Support performance testing and identify potential issues or bottlenecks

Analyse test results, identify defects and work with technical teams through resolution

Support continuous testing throughout the delivery lifecycle

Ensure appropriate regression coverage as the solution evolves

Work closely with developers, engineers and wider delivery stakeholders

Key Skills & Experience

We're looking for candidates with strong experience across

Test Automation

API Testing

Functional Testing

Integration Testing

Regression Testing

Performance Testing

Automated test frameworks and scripting

Test data management

Continuous testing / CI/CD environments

Defect identification, investigation and management

Experience testing AI, Generative AI or LLM-based solutions would be particularly valuable, including experience evaluating the quality, consistency and accuracy of AI responses.

AI Testing

Alongside traditional automation testing, an important part of the role will be understanding how to effectively test an AI-enabled solution.

Relevant experience could include

AI response evaluation and validation

Testing non-deterministic AI outputs

Developing test scenarios and evaluation criteria for AI responses

Testing AI APIs and integrations

Automated evaluation of AI functionality

Identifying unexpected, inconsistent or inappropriate model responses

Building AI-specific testing into wider automated test suites

Candidates don't necessarily need to come exclusively from an "AI Tester" background, but should have strong automation testing fundamentals and ideally some practical exposure to testing AI/LLM-enabled applications.

Reasonable Adjustments

Respect and equality are core values to us. We are proud of the diverse and inclusive community we have built, and we welcome applications from people of all backgrounds and perspectives. Our success is driven by our people, united by the spirit of partnership to deliver the best resourcing solutions for our clients.

If you need any help or adjustments during the recruitment process for any reason, please let us know when you apply or talk to the recruiters directly so we can support you.