Calibration Dataset for Legal AI

Short version: a calibration dataset (a set of data for testing) consists of test situations with no real people in them. For each of them, a lawyer has recorded the correct answer in advance. The service receives these situations as ordinary requests, and its answers are compared with the recorded ones.

An everyday example: an exam with question cards. The teacher has an answer key, and the grade follows the key whatever the examiner’s mood.

This article is about a lawyer’s work. It covers what goes into such a set, where the situations come from and how to decide whether a LegalTech service (an online tool that automates part of legal services) is ready to launch.

What the series has covered so far:

the article on the technical specification for a LegalTech service names the set as the last, sixth step;
Triage Logic gives seven test situations for a labor dispute;
Red Flags describes eight situations for testing red flags.

The main points of this article:

the correct answers are recorded by a lawyer;
every example has a basis (a provision of law, a paragraph of a Supreme Court clarification or a court ruling) and a date it was checked;
the set contains invented or rewritten situations in which a person cannot be recognized;
missing a red flag (a condition under which the service stops and hands the case to a lawyer) is unacceptable, while an unnecessary handoff to a lawyer costs time and slows the answer;
the set is extended after launch and reviewed whenever a provision changes.

The provisions are given as of October 26, 2026.

Why Does a Legal Service Need a Calibration Dataset?

A developer checks that the program starts and responds. Whether the answer is correct under the law, the developer usually cannot judge. A lawyer is needed here.

The correct answers are recorded by a lawyer, and this work is called data labeling.

The set answers two questions:

Does each rule work?
Has anything got worse after a change to the service?

The neural network, its settings or the model version get updated, and the service’s answers may change. Running the set shows exactly what broke.

What Does a Lawyer Record in Each Example?

A card has eight fields:

Request: the text as the user would write it, with no legal terms.
Facts: what the service knows from the person’s answers to its questions.
Expected route: answer on its own, ask for more data or hand to a lawyer (Triage Logic).
Requirements for the answer: what it must contain and what it cannot contain.
Basis: a provision of law, a paragraph of a Supreme Court clarification or a court ruling, with the edition.
Date checked: when the lawyer last opened the text of the provision or ruling.
Type of example: one of six (the list is below).
Who labeled it and who reviewed it.

What it looks like in practice. Take the situation from the Triage Logic article: an employee was handed no dismissal order and no work record book.

Field	Entry
Request	”I was fired, but I never received the order or my work record book, and I did not refuse to take them. Several months have passed. Can I sue?”
Facts	The order was not handed over, the work record book was not issued, there was no refusal to receive the documents
Type	Trap: by the calendar the deadline passed long ago, under the Supreme Court’s position it has not
Route	Hand to a lawyer
Must be in the answer	The deadline is not declared expired, the case is handed to a lawyer
Must not be in the answer	A statement that the deadline has expired, in any words
Basis	Art. 392(1) of the Labor Code (a dismissal dispute can be filed within one month of receiving the order or the work record book), Supreme Court Ruling of December 7, 2020, No. 5-KG20-101
Date checked	October 26, 2026

Traps are tested most strictly. A service that counts the deadline by the calendar will answer “the deadline has been missed” and take away the person’s chance to go to court.

Where Do the Examples Come From?

Four sources:

Invented by the lawyer. The lawyer makes up a situation for each rule of the service and for each red flag. There are no real people or anyone else’s data in such situations.
Cases from practice, written anew. The substance stays, while names, dates, amounts, cities and names of organizations are changed. The risk is higher than with invented situations: the client’s information is still used for a different purpose (the reasons are below). I advise using it if the contract with the client allows it or the client agrees. In other cases it is better to invent the situations.
Court acts. Court decisions are not protected by copyright (Art. 1259(6), item 1 of the Civil Code), so you can take a situation from them. The people in the act are still replaced:
- courts publish acts for the sake of open justice, and this is not counted as the people’s consent to program testing;
- in acts of courts of general jurisdiction the surnames, first names and patronymics of the parties remain, while in arbitration acts there may be more information about individuals (Art. 15(2), (3) and (4) of Law No. 262-FZ on Access to Information on the Activities of Courts).
The decision log after launch. A working service records every request: what the person wrote, which rule fired and what the service decided (answer on its own, ask for more data or hand to a lawyer). If a lawyer sees that the service got it wrong, the case becomes a new example. It is rewritten so that the person cannot be recognized, and the correct answer is recorded.

What stays out of the set: clients’ documents and correspondence as they are.

There are two reasons:

Personal data. The client shared personal information so that the lawyer could handle the case. The law does not allow using this information for another purpose that is incompatible with the first (Art. 5(2) of Law No. 152-FZ on Personal Data). Testing a program is not part of the client’s case. It needs the client’s separate consent, drawn up as a separate document (Art. 9(1) of the same law). Invented situations need no such consent.
Confidentiality. For an advocate, any information connected with providing legal assistance to the principal is advocate-client privilege (Art. 8(1) of Advocates Law No. 63-FZ). A lawyer without advocate status is usually bound to keep silent by the contract with the client, so look at your contract before using a case.

Research purposes. The law permits processing for research if the data is anonymized (Art. 6(1), item 9). Whether testing a program counts as research remains unclear. I found no clarifications or court practice on this as of October 25, 2026. I searched open sources, the Arbitration Courts Decisions Bank (I read about 130 acts) and acts of courts of general jurisdiction on sudact.ru (about 50), and none of them fit. It is safer not to take real data at all.

Replacing the surname is not enough. The law treats as personal data any information about a person who can be identified directly or indirectly (Art. 3, item 1 of 152-FZ). A place of work, a rare job title or unusual circumstances of the case give a person away even without a surname. Anonymization means that it is impossible to tell whom the data concerns without additional information (Art. 3, item 9). The law counts anonymization as processing (Art. 3, item 3), so it does not remove the question of purpose. If a client’s data is anonymized after all, Roskomnadzor Order No. 140 of June 19, 2025 applies (since September 1, 2025): you must assess whether the chosen method is sufficient, keep the original and anonymized data separately and record these actions.

An invented situation with no one else’s data is simpler and safer: it does not need protecting, handing to the developer under a contract or deleting when the work ends. Invented examples, however, come out too tidy, while real people write in a muddle, make mistakes and mix up terms (“a receipt” instead of “a contract”). Each situation is written in several versions: calm, muddled, emotional and with mistakes.

What Kinds of Examples Are There?

Six types:

Typical. An ordinary situation, and the service answers on its own.
Borderline. A value right at the edge of a rule: the number of days left until the deadline equals the number from which a case counts as urgent.
Trap. The situation looks simple, but the law makes an exception (the example in the card above).
Incomplete data. The person does not know or does not answer. The service asks for a document or hands the case to a lawyer but does not guess the answer.
Control. There are no flag signs, and the example checks that the service does not stop without a reason.
Checking references. The answer names a provision or a case. The act must exist, and its content must match the service’s summary. The check of references exists because of the case described in the article AI in Court: A ₽50,000 Fine.

The minimum for a set is one example for each rule and each red flag. After that, add examples for disputed points and for every error from the log.

How Do You Know the Service Has Passed?

Errors come in two kinds:

A miss. The service answered on its own where it should have stopped and handed the case to a lawyer. This error is the most dangerous.
An unnecessary handoff. The service sent a lawyer a situation it could have closed itself. The lawyer’s time is spent and the person waits longer, but a specialist gives the answer. When a deadline is short, a delay can also do harm, so the number of unnecessary handoffs is capped.

For red flags I suggest a strict rule: one miss means “not accepted”. It works when an ordinary rule in the program performs the stop on a flag. For the same situation a neural network may answer differently, so it is better not to give it that job (Red Flags).

The client decides how many unnecessary handoffs are acceptable and writes the number into the technical specification before launch.

How to compare an answer with the recorded one. A neural network can express one idea in ten ways: “the deadline has expired”, “you missed the time allowed”. The wording does not matter, the meaning does. A lawyer compares the meaning. A second neural network can serve as an assistant, but its assessments are checked by hand on a selective basis.

How to avoid fitting the service to the set (my recommendations):

Keep part of the examples hidden from the developer and from those who write prompts (text instructions for a neural network). Judge the delivered work on that hidden part. When all the examples are known, it is easier to tune the service to them, and it may make mistakes on new requests.
If the service fails a hidden example, the developer is told the kind of error (“the one-month deadline in a dismissal dispute, Art. 392(1) of the Labor Code, was miscounted when the order had not been handed over”). The example itself is not shown. If an example has been revealed, it is replaced with a new one.
A second lawyer reviews the labeling. If two lawyers recorded different answers, the rule is unclear: first clarify the rule, then the example.

How Does the Set Live After Launch?

The set is reviewed in three cases:

An entry in the decision log. An error that was found turns into a new example with the correct answer, rewritten without anyone else’s data.
A provision changed. The examples built on it are reread, and the date checked is updated. For regular monitoring of provisions, the service page offers a support format (Legal Maintenance).
The neural network was updated or the prompt text changed. The run is repeated before the new version starts answering users.

The set is kept in a table with a change history: every edit is written as a separate row, and the earlier rows stay.

What Goes Into the Technical Specification?

The section on the calibration dataset contains:

The composition of the set: one example for each rule and red flag, then examples for disputed points.
The mandatory types of examples.
The acceptance rule for red flags.
The acceptable number of unnecessary handoffs to a lawyer.
The part of the set closed to the developer.
The names of those who label and those who review the labeling.
The procedure for extending and reviewing the set.
Where it is stored and who has access.

Which Six Mistakes Should You Avoid With a Calibration Dataset?

The developer writes the examples. The developer understands how the code works but does not know where the law makes an exception.
Only typical situations. The service passes the check and gives a wrong answer at the first trap.
An expected answer with no basis. A year later, nobody remembers where it came from.
Real cases with a surname swap. The person can still be recognized, and nobody has protected the confidentiality or the personal data.
The set is known to everyone. The service is fitted to these examples, and its fidelity to the law stays unchecked.
The set is not extended. Errors from the log repeat on new requests.

Frequently Asked Questions

How many examples are needed? There is no single number. The minimum is one example for each rule and each red flag. After that the set grows by examples for disputed points and for every error from the log.

Who labels the examples: a lawyer or a developer? A lawyer. The correct answer under the law is recorded by the person responsible for the legal side. A second lawyer rechecks the labeling.

Can real client requests be used? Not without the client’s separate consent: the purpose of processing the client’s data is different (Art. 5(2) of 152-FZ), and for an advocate it is also advocate-client privilege. I consider a rewritten case in which the person cannot be recognized acceptable if the contract with the client allows it or the client has agreed, but swapping the surname is not enough for that. An advocate is safer using invented situations.

Can a neural network be asked to invent examples? It can suggest situations. The lawyer records the correct answer and checks it against the text of the provision: otherwise the network is testing itself.

How is the set different from the tests a developer writes? A developer checks that the code runs without failures. A legal set finds out whether the answer is correct: the deadline is counted right, the exception is applied, the flag fired.

Who owns the set of examples? A compilation made with creative effort may be protected as a composite work (Art. 1259(2), item 2 of the Civil Code), but copyright does not protect ideas and facts (Art. 1259(5)). The value of the set is better protected also as a trade secret, or know-how (Art. 1465 of the Civil Code). Information about ways of carrying out professional activity falls under this concept if the holder takes reasonable measures to keep it confidential, for example by introducing a trade secret regime (how to do it). In the contract with the developer, write down who holds the rights and whether the developer may use the set for other clients or to train other parties’ products.

Calibration Dataset: How a Lawyer Tests a Legal AI Before Launch