Blog
A question is not a stage
Independent questions over one state share one request. A second stage needs a real dependency: an earlier answer, new candidates, new context or a new model.
In short
- Five questions about one customer = one request, not five.
- A second stage needs a real dependency: an earlier answer, new candidates, new context, or a new model.
- Fewer requests, same number of recorded judgments.
Five questions about a customer do not make five model calls. In jevreduce they make one, unless one of them cannot be asked until another is answered.
The packing rule
The compiler groups questions that can be answered from the same state with compatible model settings into a single request. Their answers do not become hidden context for each other. TypeSafe’s documentation is explicit that sibling answers stay separate, and the trace keeps one recorded judgment per question either way.
If a later question actually needs an earlier answer, that is a dependency. It goes to the next reducer.
What justifies a second reducer
- A computed answer feeds the next question. A rubric that consumes returned signal values has a true dependency.
- The candidate set changes. Only the customers who passed a filter get the expensive follow-up.
- New context is retrieved. The ticket example screens on recent tickets, then joins account history for the customers that were kept. The second reducer reads data the first one never had.
- The model changes. A stage pinned to a different model is a different request.
Capacity is a fifth reason. When a question set does not fit the provider’s request limit the compiler splits it and records why.
Customer signals
- Cancel intent
- Noul
- Concern
- Choice
- Severity
- Score
Investigate reasons
- Reason
- Choice
- Explicit intent
- Noul
The ticket example, counted
- Group by customer. 5 tickets, 4 customers. No model call.
- Customer signals. One request per customer, three questions inside. 4 requests.
- Keep high-risk customers. p(cancel) at or above 0.65. 2 kept, 2 excluded. No model call.
- Attach customer history. 3 history records joined on the customer key. No model call.
- Why might they leave? Two questions over the joined state. 2 requests.
Six requests for five questions answered across four customers. If the second reducer’s questions could be answered from the first state, the compiler would fuse them and send four requests, not six. A speculative question is cheap when the state is already in the request. A gate is worth it when the follow-up needs data the first stage did not have, or when the follow-up is expensive and most units will not need it.
Why it matters at scale
Jev’s published quota at review time is 1,200 requests a minute and 250,000 tokens a second, both subject to change. One request per record over a million records is at least 13.9 hours at that rate, before any other bottleneck. Adding workers cannot pass an upstream account quota. Fewer, fuller requests are the lever that moves that number, and packing is how a reducer pulls it.
What packing never does
- It never merges two questions into one vaguer prompt. Atomic questions stay atomic.
- It never puts unrelated records into one state to save a call. Shared-array packing is a separate optimisation that needs its own quality measurements, and the docs warn that unrelated context lowers accuracy.
- It never drops a trace. Fewer requests, the same number of recorded judgments, each mapped to the request that carried it.
Next: in the app, open the Frozen definition tab on any run. The stage list is the physical plan, and the request count next to each reducer is the packing at work.