We are seeking an experienced AI & Innovation Engineer to join our Technology Delivery team supporting a strategic AI-powered system.
This role plays a critical part in ensuring the quality, reliability, and performance of modern Machine Learning, Large Language Model (LLM), and Generative AI solutions. You will design and implement scalable automation frameworks across the full technology stack - from React-based user interfaces through to backend services, AI models, search platforms, and data orchestration pipelines.
You will lead testing innovation for AI-driven applications, validating contextual accuracy, factual correctness, and ongoing performance of LLM models while enabling continuous delivery of platform enhancements.
Key Responsibilities
* Design and implement enterprise-grade test automation frameworks across UI, API, backend, and AI services
* Develop automated testing strategies for ML, LLM, and Generative AI applications
* Validate AI outputs for accuracy, relevance, and response quality across multiple models
* Integrate automation into CI/CD pipelines to support continuous delivery
* Build end-to-end regression and performance test suites
* Monitor automation metrics and provide actionable quality insights
* Lead test planning, coverage strategy, and risk-based testing activities
* Mentor junior engineers and promote automation best practices
* Collaborate closely with developers, product teams, and stakeholders
* Drive continuous improvement of testing frameworks and DevOps practices
Technical Environment
* Languages & Tools: Python, pytest, FastAPI
* Automation: Playwright, Selenium, Appium, REST Assured
* AI & Data: Azure OpenAI, AWS Bedrock, Databricks, Elasticsearch, Vector Search
* Frontend & Backend: React / Next.js, APIs, distributed services
* Cloud & DevOps: AWS, Azure, Git, CI/CD pipelines
Skills & Experience
* Strong experience designing scalable test automation frameworks
* Hands‐on testing of AI, NLP, or LLM-based applications
* Advanced Python automation development
* Experience testing complex distributed systems, APIs, and data platforms
* Deep understanding of Agile delivery and DevOps practices
* Experience with cloud-based testing environments
* Ability to produce meaningful quality metrics and reporting
What You'll Bring
* Technical leadership in automation and quality engineering
* Strong analytical and problem‐solving capability
* Passion for emerging AI technologies and innovation
* Ability to work autonomously while contributing to high‐performing teams
Desirable Certifications
* ISTQB Advanced Test Automation Engineering (CTAL‐TAE)
* AI/ML or Data Science certifications
* Python or SQL certifications
Desired Skills and Experience
* Advanced Python programming with expertise in Fast API, pytest, and data science libraries (Pandas, NumPy)
* Strong experience building backend test automation frameworks using Python, preferably with Fast API
* Experience testing Large Language Models, natural language processing systems, or AI‐driven applications.
* Demonstrated experience testing complex, multi‐component systems including APIs, databases, and search technologies.
* Proven ability to design and build scalable, reusable test automation frameworks from the ground up.
* Experience with LLM observation and evaluation tools, such as MLflow or Opik
* Experience with cloud platforms: AWS services (S3, Aurora, Bedrock, Transfer), Azure services (OpenAI, Document Intelligence), and modern data platforms like Databricks
* Experience testing Elasticsearch, vector databases, or similar search technologies.
* Experience integrating test automation into CI/CD pipelines using Git and collaborative development workflows.
* Significant hands‐on experience with UI test automation (such as Playwright), particularly with React/Next.js applications
Why Join Us?
This is an opportunity to work at the forefront of AI and Generative AI technology, shaping testing practices for next‐generation intelligent platforms while delivering meaningful enterprise impact.
For more information or a confidential discussion about this role, please contact Ronald at ronald.tran@talentinternational.com
#J-18808-Ljbffr