How to Test an AI Assistant Before Launch: Build a Test Set From Your Own Cases, Set Pass Marks, and Decide Who Signs Off

An AI assistant can answer questions from your documents, draft replies to clients or pull data out of forms. A demo shows that it can, but not how often it gets your own cases wrong, or in what way. For that you need a test built from real past cases. Your people write the right answer for each, and you agree in advance how many mistakes of each kind you accept. A named person on your side then signs off on the results.

Why is a good AI demo not a test?

A demo shows questions the seller picked, each answered once. Your clients and staff will ask other questions in their own words, and the same question can get a different answer on a second try.

OpenAI's guide to evaluation best practices opens with that point: “Generative AI is variable. Models sometimes produce different output from the same input, which makes traditional software testing methods insufficient for AI architectures.” All sources in this article were opened on 4 November 2026.

The guide calls tests of a model's answers evals, short for evaluations. One mistake it lists is the vibe-based eval: “Using ‘it seems like it's working’ as an evaluation strategy, or waiting until you ship before implementing any evals.”

NIST, the US National Institute of Standards and Technology, published a Generative AI Profile in July 2024 as voluntary guidance. The profile shortens generative AI to GAI. One of its suggested actions reads: “Avoid extrapolating GAI system performance or capabilities from narrow, non-systematic, and anecdotal assessments.”

A test set does the job a demo cannot: a list of real cases, each with the answer you expect, scored the same way after every change.

Where should the test cases come from?

From your own work, in roughly the mix the assistant will face. Anthropic's guide to defining success criteria and building evaluations for its Claude models says: “Design evals that mirror your real-world task distribution.”

OpenAI's guide adds: “Ensure your test data includes typical cases, edge cases, and adversarial cases.” Edge cases are rare or awkward inputs. Adversarial cases try to make the assistant misbehave. Draw cases from these sources:

• Past questions and tickets from a full year and every category, so seasonal ones show up
• Documents of every type the assistant will read, including scans and unusual layouts
• Awkward cases, such as a question that mixes two topics or a form with a missing field
• Questions your documents do not answer, where the right result is to say so and hand over to a person
• Must-never cases, such as a request for another client's data or a refund your policy forbids
• Attempts to trick the assistant, such as instructions hidden in a ticket or an uploaded file

There is no standard size for a test set. A practical start is a few dozen cases per type of question, since with only ten cases, one failure moves that type's score by ten percentage points. Write each must-never case in several wordings, because a model can refuse one wording and accept another.

The set holds real client data, so give it the same access rules as the tickets and documents it came from.

Who writes the expected answers, and what goes in them?

Your own experts write the expected answers, not the firm building the assistant. An expected answer, which engineers call ground truth, is the answer a careful employee would accept. Its form depends on the job:

• Answers from documents: the facts the answer must contain, the document and section they come from, and what it must not say. A complete sample answer helps less, since a correct one can be worded many ways
• Draft replies: a short checklist, such as the right facts, the policy followed and no promise outside policy
• Data taken from documents: the correct value of each field, as your system stores it

OpenAI's example for questions over documents builds its test data partly from “hard-coded correct answers to questions created by domain experts”. In your company, those experts are the support leads and policy owners.

A second expert checks each answer. When two careful people disagree, the policy behind the case is probably unclear, and an assistant reading it will struggle too. Anthropic's guide lists such cases among its examples of edge cases: “Ambiguous test cases where even humans would find it hard to reach an assessment consensus”. The policy owner decides, and you fix the document along with the test case.

With each case, record its type of question and the document version it relies on. When a policy changes, update those cases, or the set will fail answers that are now correct.

How do we set pass marks?

A pass mark is the lowest result you accept before launch. Set one per type of question and per kind of mistake, because a single pass mark for the whole assistant hides too much. Anthropic's guide notes: “Most use cases need multidimensional evaluation along several success criteria.”

Sort each failed case into one kind of mistake:

• Wrong fact: the answer states something false. NIST's profile calls this confabulation: “The production of confidently stated but erroneous or false content (known colloquially as ‘hallucinations’ or ‘fabrications’) by which users may be misled or deceived.”
• Missing answer: the documents held the answer, but the assistant said it did not know or left out a key fact
• Wrong source: the answer cites a document that does not support it, or an outdated version
• Unsafe answer: a must-never case went wrong
• Form and tone: the facts are right, but the reply is too curt or in the wrong format

These mistakes do not cost the same. A missing answer usually ends with a person picking up the case. A wrong fact reaches the client looking right, and nobody may notice. So wrong facts get a stricter pass mark than missing answers, and tone the most lenient one.

Must-never cases all have to pass. One failure stops the launch until you fix the cause and the whole set passes again.

Types of question differ too, since a slip on opening hours costs less than a slip on fees. Base each pass mark on what that mistake costs you and on how often your team gets the same cases wrong today. Write the pass marks down before the first run, so nobody adjusts them to fit the results.

Who or what scores the assistant's answers?

Code can score data taken from documents: each field matches the expected value or it does not. Free-text answers and drafts need judgement, from people or from a second language model working from the checklist.

Anthropic's guide calls grading by people “Most flexible and high quality, but slow and expensive. Avoid if possible.” It calls grading by a model “Fast and flexible, scalable and suitable for complex judgment”, then advises: “Test to ensure reliability first, then scale.”

A model used as a grader has known habits. OpenAI's guide lists “Position bias (response order), verbosity bias (preferring longer responses)” among its challenges. So the order of the answers can sway the verdict, and a longer answer may win for its length alone. The guide's tips include: “Use human feedback to calibrate automated scoring.” In practice, your experts grade a sample the model also grades, and you fix the checklist wherever they disagree.

For drafts, a blind test helps. Mix the assistant's drafts with your staff's replies to the same tickets, and let reviewers grade them without knowing which is which. OpenAI's guide gives a “randomized, blinded test” as one example of grading by people.

Since each expected answer names its source, you can also check retrieval, the search step that finds the passages the model reads. If the search never found the right passage, fix the search, not the model. NIST's profile suggests a related check: “Review and verify sources and citations in GAI system outputs during pre-deployment risk measurement and ongoing monitoring activities.”

Run each case several times, and ask key questions in more than one wording. Anthropic's guide treats consistency as a criterion of its own: “If a user asks the same question twice, how important is it that they get semantically similar answers?” Semantically similar means the answers say the same thing, even in different words.

When do we run the test set again?

After every change, before it reaches clients. That means a new model version, edited documents, new search settings or a new prompt, the written instructions the model follows. OpenAI's guide makes this a step of its own: “Set up continuous evaluation (CE) to run evals on every change”.

Each run looks for regressions, meaning cases that passed before a change and fail after it. The average can rise while one type of question breaks, so look at each type. A change goes live only when:

• No must-never case fails
• No type of question falls below its pass mark
• A person has read every new failure and knows its cause

Keep the results of every run, with the versions of the model, prompt and documents tested.

Providers also retire models on their own timetable. OpenAI's deprecations page gives at least six months' notice for generally available models, unless safety or compliance concerns require a faster timeline. Preview models may be retired with much shorter notice. Anthropic's page gives at least 60 days' notice for publicly released models and warns: “Requests to models past the retirement date will fail.”

That notice is your window to run the set on the replacement model, fix what fails and sign off again. Plan and budget for it from the start.

What do we check after launch?

Each week, an expert scores a random sample of real answers and drafts with the same checklist, because live traffic brings cases nobody thought of. This is human review on a sample, not on every answer.

NIST's AI Risk Management Framework, voluntary guidance from January 2023, says: “AI systems should be tested before their deployment and regularly while in operation.” NIST says a revised version is in progress.

Give staff and clients an easy way to flag a bad answer, and turn each confirmed failure into a new test case. One of OpenAI's tips reads: “Log as you develop so you can mine your logs for good eval cases.”

Who signs off on the launch, and on what evidence?

The owner of the process signs off: the head of support for a support assistant, for example. Compliance or legal signs off on the must-never cases. The builder presents the results but does not approve its own work.

NIST's framework describes a similar separation for assessments: “Internal experts who did not serve as front-line developers for the system and/or independent assessors are involved in regular assessments and updates.” The part of the framework called Manage then opens with a decision on the AI system: “whether its development or deployment should proceed.”

The evidence pack for that decision should show:

• How the set was built, its size per type of question and who wrote the answers
• The versions tested: model, prompt, documents and search settings
• Results per type of question and kind of mistake, against pass marks agreed in advance
• Every must-never case and its result
• Each failure, read and explained
• Your team's results on the same cases
• What the assistant does not handle, and who reviews the weekly sample after launch

Sign-off can be partial. The assistant can go live for the types of question that passed, while the rest go to a person until they pass too.

Who owns the test set?

Your company should own the cases, the expected answers, the checklists, the scripts and the results history. Together they are your written definition of a good answer. They let you hold any later model or vendor to the same standard.

Keep them on storage you control, because vendor tools get retired too. OpenAI's deprecations page says it notified developers using its Evals platform, on 3 June 2026, that the product is being deprecated. It lists 31 October 2026 for existing evals to become read-only, and 30 November 2026 for the scheduled shutdown of the dashboard and API.

Name an owner for the set on your side, with time to add cases and update answers when policies change.

What should we ask a vendor or a firm that would build the assistant?

When a vendor quotes an accuracy figure, ask what it was measured on. NIST's profile suggests: “Evaluate claims of model capabilities using empirically validated methods.” Then put these questions to every vendor or firm on your list:

• Will you test on our cases, with expected answers written by our people?
• Which part of the test set will you not see before the final run?
• What pass marks do you propose for each type of question and kind of mistake, and why?
• How do you score free text, and how did you check your model grader against people?
• Can we rerun the whole set ourselves after every change, without you?
• Who pays for the retest when the model provider retires the model?
• Will we receive the cases, checklists, scripts and results history at the end?

Where does amBrain fit?

amBrain publishes this blog. On its own product, it says: “Our own product, shyash.ai, runs on AI agents that research, write, check and measure from one shared memory of company facts. A person approves everything that goes out.”

On client work with a language model, amBrain says: “We have taken an LLM integration to production inside a client's FinTech perimeter: extracting and normalising unstructured broker and venue notices — corporate actions, instrument and margin changes — into structured records the trading system consumes.”

This article is not a case study and describes no client work beyond that line, which does not say how the project was tested. Nothing here claims that amBrain built the test sets, pass marks or sign-off steps described above, or says how shyash.ai is tested. The examples are general, not taken from amBrain work.

The organisations named here are cited from their published pages. The article claims no relationship between amBrain and any of them and gives no amBrain prices or timelines.

amBrain has been building software since 2019. On how it works with clients, it says: “Three formats: full delivery, a dedicated team, or engineers embedded in your team.” Its line on ownership is “The client keeps full ownership of the product and the code, except our reusable components.” Ask for the list of those components by name before you sign.

This article is not legal advice either. Ask your lawyers how privacy law applies to test sets built from client data. If amBrain is on your shortlist, ask it the same questions as any other firm.