Use cases / Data labeling and evals

Grading model outputs, and comparing models by replay

Grade model outputs. Swap the judge. See where the judges disagree.

Unit: One response to one task, then both responses
Ask: Correct? Quality? Failure mode? Then: which is better?
Get: Per task: both grades, a preference, two judges

Files · 2 reducers · 4 questions · updated Sep 21, 2026

The flow

0 responses source 1,200 tasks, 2 models

0 Grade each response reducer requests, 3 questions each

0 Group by task group no model call

0 Prefer reducer requests, 1 question each

0 Outputs output per task, both grades and the preference

Grade each response alone. Prefer with both in view. Replay with another judge. 3,600 requests per judge. The replay is another 3,600 on the same snapshot.

Follow one record

responses
Grade each response
Group by task
Prefer
Replay with Kev 4B
Outputs

Task TK-0087, two responses

snapshot v1, records TK-0087/A and TK-0087/B

promptFrom the invoice text below, return JSON with keys total and due_date. total is a number; due_date is ISO 8601.
criteriaBoth keys present. total equals 1,284.50. due_date equals 2026-10-15.
response A{"total": 1284.50, "due_date": "2026-10-15"}
response BTotal: $1,284.50. Due date: October 15, 2026. Let me know if you need anything else.

Requests 173 and 174 of 2,400

one request per response, model jev-1.13.0

correctbinary

94.0%

true0.94

both keys

qualityordinal

3.94

Unusable0.00
Poor0.01
Acceptable0.12
Good0.78
Excellent0.09

confidence 0.76, provider reported

failure_modecategorical

none

none0.91
format violation0.06
incomplete0.02
wrong answer0.01

confidence 0.88

{
  "model": "jev-1.13.0",
  "state": {
    "task": {
      "id": "TK-0087",
      "prompt": "From the invoice text below, return JSON with keys total and due_date. total is a number; due_date is ISO 8601.",
      "criteria": ["Both keys present", "total equals 1284.50", "due_date equals 2026-10-15"]
    },
    "rubric": "Unusable: wrong or harmful. Poor: partly right, needs rework. Acceptable: right, rough. Good: right and clean. Excellent: right, clean, and handles the edge cases.",
    "response": { "label": "A", "model": "model-a", "text": "{\"total\": 1284.50, \"due_date\": \"2026-10-15\"}" }
  },
  "questions": {
    "correct":      { "type": "noul", "instructions": "Does `response` satisfy every acceptance criterion listed in `task.criteria`?" },
    "quality":      { "type": "score", "instructions": "Using `rubric` as the scale, how good is `response` as an answer to `task.prompt`?", "criteria": ["Unusable", "Poor", "Acceptable", "Good", "Excellent"] },
    "failure_mode": { "type": "choice", "instructions": "If `response` falls short of `task.criteria`, how? Choose `none` if it does not.",
                      "criteria": { "wrong_answer": "The substantive answer is incorrect", "incomplete": "Part of the task is missing",
                                    "unsupported_claim": "It asserts something the task material does not support",
                                    "format_violation": "The required output format is not followed", "refused": "It declined or deflected",
                                    "none": "It meets the criteria" } }
  }
}

{
  "model": "jev-1.13.0",
  "answers": {
    "correct":      { "probability": 0.94 },
    "quality":      { "expected": 3.94, "probabilities": [0.00, 0.01, 0.12, 0.78, 0.09], "confidence": 0.76 },
    "failure_mode": { "choice": "none", "probabilities": { "none": 0.91, "format_violation": 0.06, "incomplete": 0.02, "wrong_answer": 0.01, "unsupported_claim": 0.00, "refused": 0.00 }, "confidence": 0.88 }
  }
}

Two graded responses become one unit

deterministic, no model call

key: TK-0087
members: TK-0087/A, TK-0087/B
grades carried: A: correct 0.94, quality 3.94. B: correct 0.31, quality 2.31

Request 87 of 1,200

model jev-1.13.0, attempt 1, succeeded

preferencecategorical

A0.86
tie0.08
B0.06

confidence 0.83, provider reported

{
  "model": "jev-1.13.0",
  "answers": {
    "preference": { "choice": "A", "probabilities": { "A": 0.86, "tie": 0.08, "B": 0.06 }, "confidence": 0.83 }
  }
}

Same snapshot, same definition, judge kev-4b

run 6c1e44d0, parent 2f8b9a77

preference: A 0.79, tie 0.12, B 0.09
A correct: 0.90
B correct: 0.38
agreement on TK-0087: both judges prefer A; the margin differs
what changed: the model profile in the run manifest, nothing else

One task row with two judges

two runs, joined on task_id in the export

task: TK-0087
A correct: 0.94 jev, 0.90 kev-4b
B correct: 0.31 jev, 0.38 kev-4b
preference: A 0.86 jev, A 0.79 kev-4b
failure mode B: format violation (0.64)
trace: 3 attempts per judge, 6 raw payloads

1 of 6

What comes out

Eval set v1, judged by jev and by kev-4bruns 2f8b9a77 and 6c1e44d0, 1,200 rows

Task	A correct	B correct	A quality	B quality	Preference (jev)	Preference (kev-4b)
TK-0087	0.94	0.31	3.94	2.31	A 0.86	A 0.79
TK-0088	0.62	0.71	3.10	3.22	B 0.51	tie 0.44
TK-0089	0.08	0.91	1.42	4.05	B 0.93	B 0.90
TK-0090	0.88	0.87	3.80	3.77	tie 0.58	A 0.49
TK-0091	0.45	0.12	2.60	1.31	A 0.81	A 0.85

Synthetic rows in the app's Outputs layout. The two preference columns come from two runs on the same snapshot; the second is a replay of the first with a different model profile.

Details, if you want them

whyWhy this shape

Why this shape

Grade alone, prefer together. Independent grades, then A against B in one view.
Typed, not parsed. A probability, a rubric position, a failure name, a preference split.
Swap the judge. One field in the manifest. Same snapshot. Two columns.
Disagreements are the labelling queue. Both judges’ raw responses are in the trace.

questionsThe 4 questions, verbatim

Grade each response correctbinary, noul
Does `response` satisfy every acceptance criterion listed in `task.criteria`?
Grade each response qualityordinal, score
Using `rubric` as the scale, how good is `response` as an answer to `task.prompt`?
UnusablePoorAcceptableGoodExcellent
Grade each response failure_modecategorical, choice
If `response` falls short of `task.criteria`, how? Choose `none` if it does not.
wrong answerincompleteunsupported claimformat violationrefusednone
Prefer preferencecategorical, choice
Considering `task.prompt`, `task.criteria` and both responses in `responses`, which response is the better answer?
ABtie

buildHow each stage is built, as JSON

1 reducer Grade each response

One request per response. correct is a probability against the task’s own acceptance criteria, which are in the state. quality is an ordinal over your five-level rubric. failure_mode names how it went wrong, or none. Grading A and B in separate requests keeps the grades independent; the model never sees the other response here.

stage definition

{
  "id": "grade",
  "input": "source:responses",
  "kind": "evaluate",
  "model": "jev",
  "name": "Grade each response",
  "questions": {
    "correct": {
      "instructions": "Does `response` satisfy every acceptance criterion listed in `task.criteria`?",
      "type": "noul"
    },
    "failure_mode": {
      "criteria": {
        "format_violation": "The required output format is not followed",
        "incomplete": "Part of the task is missing",
        "none": "It meets the criteria",
        "refused": "It declined or deflected",
        "unsupported_claim": "It asserts something the task material does not support",
        "wrong_answer": "The substantive answer is incorrect"
      },
      "instructions": "If `response` falls short of `task.criteria`, how? Choose `none` if it does not.",
      "type": "choice"
    },
    "quality": {
      "criteria": [
        "Unusable",
        "Poor",
        "Acceptable",
        "Good",
        "Excellent"
      ],
      "instructions": "Using `rubric` as the scale, how good is `response` as an answer to `task.prompt`?",
      "type": "score"
    }
  },
  "state_path": []
}

2 group Group by task

Deterministic. The two graded responses for each task become one unit, with their grades attached. The preference reducer sees both, plus the task.

stage definition

{
  "id": "by_task",
  "input": "grade",
  "key": [
    "task_id"
  ],
  "kind": "group",
  "name": "Group by task"
}

3 reducer Prefer

One request per task with both responses in the state. preference is a choice among A, B and tie, returned as a distribution, so a 0.52 to 0.48 split looks different from 0.90 to 0.05. Replaying this stage with a different judge model gives you a second preference column on the same receipts.

stage definition

{
  "id": "prefer",
  "input": "by_task",
  "kind": "evaluate",
  "model": "jev",
  "name": "Prefer",
  "questions": {
    "preference": {
      "criteria": {
        "A": "The response labelled A",
        "B": "The response labelled B",
        "tie": "They are equally good or equally bad"
      },
      "instructions": "Considering `task.prompt`, `task.criteria` and both responses in `responses`, which response is the better answer?",
      "type": "choice"
    }
  },
  "state_path": []
}

4 definition The whole workflow, as saved

Saving publishes an immutable version. A run freezes this definition, the snapshot IDs and each reducer’s model profile, so the trace can always show exactly what ran. Edits create a new version; old runs keep theirs.

jevreduce.local.v1

{
  "name": "Grading model outputs, and comparing models by replay",
  "schema_version": "jevreduce.local.v1",
  "sources": {
    "responses": "snap_eval_responses_v1"
  },
  "stages": [
    {
      "id": "grade",
      "input": "source:responses",
      "kind": "evaluate",
      "model": "jev",
      "name": "Grade each response",
      "questions": {
        "correct": {
          "instructions": "Does `response` satisfy every acceptance criterion listed in `task.criteria`?",
          "type": "noul"
        },
        "failure_mode": {
          "criteria": {
            "format_violation": "The required output format is not followed",
            "incomplete": "Part of the task is missing",
            "none": "It meets the criteria",
            "refused": "It declined or deflected",
            "unsupported_claim": "It asserts something the task material does not support",
            "wrong_answer": "The substantive answer is incorrect"
          },
          "instructions": "If `response` falls short of `task.criteria`, how? Choose `none` if it does not.",
          "type": "choice"
        },
        "quality": {
          "criteria": [
            "Unusable",
            "Poor",
            "Acceptable",
            "Good",
            "Excellent"
          ],
          "instructions": "Using `rubric` as the scale, how good is `response` as an answer to `task.prompt`?",
          "type": "score"
        }
      },
      "state_path": []
    },
    {
      "id": "by_task",
      "input": "grade",
      "key": [
        "task_id"
      ],
      "kind": "group",
      "name": "Group by task"
    },
    {
      "id": "prefer",
      "input": "by_task",
      "kind": "evaluate",
      "model": "jev",
      "name": "Prefer",
      "questions": {
        "preference": {
          "criteria": {
            "A": "The response labelled A",
            "B": "The response labelled B",
            "tie": "They are equally good or equally bad"
          },
          "instructions": "Considering `task.prompt`, `task.criteria` and both responses in `responses`, which response is the better answer?",
          "type": "choice"
        }
      },
      "state_path": []
    }
  ]
}

limitsWhat it will not do

A judge has an error rate. Compare with human labels.
Adversarial text can sway a judge.
Two judges agreeing does not make them right.
No critique text. A failure mode is a name, not a paragraph.
Deterministic checks belong in code, not in a question.

relatedRelated pages

The flow

Follow one record

Task TK-0087, two responses

Requests 173 and 174 of 2,400

Two graded responses become one unit

Request 87 of 1,200

Same snapshot, same definition, judge kev-4b

One task row with two judges

What comes out

Details, if you want them

Why this shape

Try it on your own records.