jevreduce

Use cases / Hiring and talent

Interview transcripts scored against a rubric

Every interview scored against one rubric, with the quote behind each score.

Unit
One interview transcript plus the role’s rubric
Ask
Three competency scores. The evidence turn. Did they answer?
Get
Scores with distributions, and the one answer to read

Fireflies · Files · 2 reducers · 7 questions · updated Sep 21, 2026

The flow

0 interviews source snapshot v4
0 Attach the rubric join by role_id, no model call
0 Score competencies reducer requests, 5 questions each
0 Keep problem solving ≥ 2.0 filter on the rubric's own scale
0 Attach the resume join no model call
0 Must-haves reducer requests, 2 questions each
0 Outputs output scored, with the quote

Same frozen rubric for every candidate. Must-haves only for those above the bar. 343 requests for 212 interviews. Every score links to a rubric version and a turn.

Follow one record

What comes out

Hiring round, senior support engineerrun 5d20b7c4, 131 rows above the bar, 81 in the excluded collection
InterviewProblem solvingCommunicationOwnershipEvidencePython in prodLed incident
I-03883.413.123.05A40.960.91
I-04122.703.082.24A20.880.34
I-04012.662.312.90A30.520.78
I-03772.412.882.12A20.910.12
I-04202.082.022.55A50.230.61

Synthetic rows in the app's Outputs layout. Scores are expected positions on the four-level rubric; the distributions are one click away.

Details, if you want them

whyWhy this shape

Why this shape

  • One rubric, frozen. Every candidate scored against the same words.
  • Ordinal scores. 2.7 means between two levels, and you can see how sure.
  • The evidence turn. Read one answer, not the transcript.
  • Must-haves only above the bar. The other 81 keep their scores.

What it replaces

  • Six people remembering what was said.
  • A 3 that means a 4 to someone else.
  • Not the interviewer. People decide.
questionsThe 7 questions, verbatim
  • Score competencies problem_solvingordinal, score

    Using `rubric.problem_solving` as the scale, how does the candidate’s reasoning in `transcript` rate?

    Below the barMeets with gapsMeetsExceeds
  • Score competencies communicationordinal, score

    Using `rubric.communication` as the scale, how clearly does the candidate explain their thinking in `transcript`?

    Below the barMeets with gapsMeetsExceeds
  • Score competencies ownershipordinal, score

    Using `rubric.ownership` as the scale, how much personal responsibility for outcomes does the candidate show in `transcript`?

    Below the barMeets with gapsMeetsExceeds
  • Score competencies evidence_turncategorical, choice

    Which of the candidate’s answers in `transcript` best evidences the problem-solving score?

    A1A2A3A4A5A6
  • Score competencies answeredbinary, noul

    Did the candidate answer the questions actually asked in `transcript`, rather than a different question?

  • Must-haves python_productionbinary, noul

    Do `transcript` and `resume` together show the candidate has written Python that ran in production?

  • Must-haves led_incidentbinary, noul

    Do `transcript` and `resume` together show the candidate has led the response to a customer-facing incident?

buildHow each stage is built, as JSON
1 join Attach the rubric
A join on role_id puts the rubric text into each interview’s state. The rubric is a snapshot, so every candidate in the run is scored against the same words; editing it later creates a new version and does not touch this run.

stage definition

{
  "id": "with_rubric",
  "input": "source:interviews",
  "kind": "join",
  "left_key": [
    "role_id"
  ],
  "name": "Attach the rubric",
  "right": "source:rubrics",
  "right_key": [
    "role_id"
  ]
}
2 reducer Score competencies
One request per interview, five questions inside. The three competency scores are ordinals over the rubric’s four levels, so an expected score of 2.7 means “between meets-with-gaps and meets, leaning meets”, and the distribution shows how sure that is. evidence_turn picks the candidate’s answer that best supports the problem-solving score; answered catches the answer that dodged the question.

stage definition

{
  "id": "score",
  "input": "with_rubric",
  "kind": "evaluate",
  "model": "jev",
  "name": "Score competencies",
  "questions": {
    "answered": {
      "instructions": "Did the candidate answer the questions actually asked in `transcript`, rather than a different question?",
      "type": "noul"
    },
    "communication": {
      "criteria": [
        "Below the bar",
        "Meets with gaps",
        "Meets",
        "Exceeds"
      ],
      "instructions": "Using `rubric.communication` as the scale, how clearly does the candidate explain their thinking in `transcript`?",
      "type": "score"
    },
    "evidence_turn": {
      "criteria": {
        "A1": "The candidate's first answer",
        "A2": "The candidate's second answer",
        "A3": "The candidate's third answer",
        "A4": "The candidate's fourth answer",
        "A5": "The candidate's fifth answer",
        "A6": "The candidate's sixth answer"
      },
      "instructions": "Which of the candidate's answers in `transcript` best evidences the problem-solving score?",
      "type": "choice"
    },
    "ownership": {
      "criteria": [
        "Below the bar",
        "Meets with gaps",
        "Meets",
        "Exceeds"
      ],
      "instructions": "Using `rubric.ownership` as the scale, how much personal responsibility for outcomes does the candidate show in `transcript`?",
      "type": "score"
    },
    "problem_solving": {
      "criteria": [
        "Below the bar",
        "Meets with gaps",
        "Meets",
        "Exceeds"
      ],
      "instructions": "Using `rubric.problem_solving` as the scale, how does the candidate's reasoning in `transcript` rate?",
      "type": "score"
    }
  },
  "state_path": []
}
3 filter Keep problem solving ≥ 2.0
A comparison on the expected problem-solving score. It gates the second pass; it does not reject anyone. Candidates below the line keep their scores and their evidence in the excluded collection, where the hiring manager can still read them.

stage definition

{
  "comparison": "gte",
  "field": [
    "answers",
    "problem_solving",
    "expected"
  ],
  "id": "above_bar",
  "input": "score",
  "kind": "filter",
  "name": "Keep problem solving ≥ 2.0",
  "value": 2
}
4 join Attach the resume
The must-haves need the resume as well as the transcript, so the resume snapshot is joined on candidate_id only for the candidates above the bar.

stage definition

{
  "id": "with_resume",
  "input": "above_bar",
  "kind": "join",
  "left_key": [
    "candidate_id"
  ],
  "name": "Attach the resume",
  "right": "source:resumes",
  "right_key": [
    "candidate_id"
  ]
}
5 reducer Must-haves
Two binary questions with the transcript and the resume in the state. Each must-have is its own proposition, so a candidate can be strong on one and weak on the other, and you can see which.

stage definition

{
  "id": "must_haves",
  "input": "with_resume",
  "kind": "evaluate",
  "model": "jev",
  "name": "Must-haves",
  "questions": {
    "led_incident": {
      "instructions": "Do `transcript` and `resume` together show the candidate has led the response to a customer-facing incident?",
      "type": "noul"
    },
    "python_production": {
      "instructions": "Do `transcript` and `resume` together show the candidate has written Python that ran in production?",
      "type": "noul"
    }
  },
  "state_path": []
}
6 definition The whole workflow, as saved

Saving publishes an immutable version. A run freezes this definition, the snapshot IDs and each reducer’s model profile, so the trace can always show exactly what ran. Edits create a new version; old runs keep theirs.

jevreduce.local.v1

{
  "name": "Interview transcripts scored against a rubric",
  "schema_version": "jevreduce.local.v1",
  "sources": {
    "interviews": "snap_fireflies_interviews_v4",
    "resumes": "snap_resumes_v4",
    "rubrics": "snap_rubrics_v2"
  },
  "stages": [
    {
      "id": "with_rubric",
      "input": "source:interviews",
      "kind": "join",
      "left_key": [
        "role_id"
      ],
      "name": "Attach the rubric",
      "right": "source:rubrics",
      "right_key": [
        "role_id"
      ]
    },
    {
      "id": "score",
      "input": "with_rubric",
      "kind": "evaluate",
      "model": "jev",
      "name": "Score competencies",
      "questions": {
        "answered": {
          "instructions": "Did the candidate answer the questions actually asked in `transcript`, rather than a different question?",
          "type": "noul"
        },
        "communication": {
          "criteria": [
            "Below the bar",
            "Meets with gaps",
            "Meets",
            "Exceeds"
          ],
          "instructions": "Using `rubric.communication` as the scale, how clearly does the candidate explain their thinking in `transcript`?",
          "type": "score"
        },
        "evidence_turn": {
          "criteria": {
            "A1": "The candidate's first answer",
            "A2": "The candidate's second answer",
            "A3": "The candidate's third answer",
            "A4": "The candidate's fourth answer",
            "A5": "The candidate's fifth answer",
            "A6": "The candidate's sixth answer"
          },
          "instructions": "Which of the candidate's answers in `transcript` best evidences the problem-solving score?",
          "type": "choice"
        },
        "ownership": {
          "criteria": [
            "Below the bar",
            "Meets with gaps",
            "Meets",
            "Exceeds"
          ],
          "instructions": "Using `rubric.ownership` as the scale, how much personal responsibility for outcomes does the candidate show in `transcript`?",
          "type": "score"
        },
        "problem_solving": {
          "criteria": [
            "Below the bar",
            "Meets with gaps",
            "Meets",
            "Exceeds"
          ],
          "instructions": "Using `rubric.problem_solving` as the scale, how does the candidate's reasoning in `transcript` rate?",
          "type": "score"
        }
      },
      "state_path": []
    },
    {
      "comparison": "gte",
      "field": [
        "answers",
        "problem_solving",
        "expected"
      ],
      "id": "above_bar",
      "input": "score",
      "kind": "filter",
      "name": "Keep problem solving ≥ 2.0",
      "value": 2
    },
    {
      "id": "with_resume",
      "input": "above_bar",
      "kind": "join",
      "left_key": [
        "candidate_id"
      ],
      "name": "Attach the resume",
      "right": "source:resumes",
      "right_key": [
        "candidate_id"
      ]
    },
    {
      "id": "must_haves",
      "input": "with_resume",
      "kind": "evaluate",
      "model": "jev",
      "name": "Must-haves",
      "questions": {
        "led_incident": {
          "instructions": "Do `transcript` and `resume` together show the candidate has led the response to a customer-facing incident?",
          "type": "noul"
        },
        "python_production": {
          "instructions": "Do `transcript` and `resume` together show the candidate has written Python that ran in production?",
          "type": "noul"
        }
      },
      "state_path": []
    }
  ]
}
limitsWhat it will not do
  • A score is a position on your rubric, not a hiring decision.
  • Rubric versions are frozen per run. Replay to rescore.
  • Transcription errors are scored as said.
  • Measure consistency on a labelled sample first.
  • No written feedback. Jev returns scores and a turn.
relatedRelated pages

Try it on your own records.

Open app All use cases