JEV RECIPES
Putting one large model in charge of grading another is slow, expensive, and the confidence it reports clusters around 95%, so it sorts nothing. Grading is a judgement, not an essay — which is the shape a System One model is built for.
Every number on this page comes from our own 2,390-question run; method and limits are in the benchmark report (written in Chinese).
Grounded in the source, complete, and not padded — in a single call.
import { experimental_evaluate as evaluate } from 'ai';
const { answers } = await evaluate({
model: 'typesafe-ai/jev',
state, // Paste the question, the source material and the model's answer
questions: {
grounded: { type: 'boolean', instructions: 'Is every factual statement in the answer supported by the source material?' },
complete: { type: 'boolean', instructions: 'Does the answer fully address the question that was asked?' },
concise: { type: 'score', instructions: 'How efficiently is this answer written?',
criteria: ['Mostly padding', 'Wordy', 'About right', 'Tight and clear'] },
},
});Check the answer against what was retrieved, and get back ship / revise / block.
Put a threshold on the probability of `supported` — say 0.8 — to auto-ship, and send the rest to a person. The band Jev marks as uncertain is precisely the band a human should see.
import { experimental_evaluate as evaluate } from 'ai';
const { answers } = await evaluate({
model: 'typesafe-ai/jev',
state, // Paste the retrieved material and the generated answer
questions: {
supported: { type: 'boolean', instructions: 'Is every specific claim in the answer supported by the retrieved material?' },
contradicts: { type: 'boolean', instructions: 'Does the answer state anything that directly contradicts the retrieved material?' },
action: { type: 'choice', instructions: 'Given how well the material supports it, what should happen to this answer?',
criteria: { accept: 'Ship it to the user as written', revise: 'Needs correction or qualification first', reject: 'Departs from the source; regenerate' } },
},
});A/B two versions, with a margin, for offline evals or a live rollout.
import { experimental_evaluate as evaluate } from 'ai';
const { answers } = await evaluate({
model: 'typesafe-ai/jev',
state, // Paste two answers to the same question
questions: {
winner: { type: 'choice', instructions: 'Which version is more useful to the person who asked?',
criteria: { a: 'Answer A is better', b: 'Answer B is better', tie: 'They are about equal' } },
margin: { type: 'score', instructions: 'How large is the quality gap between the two?',
criteria: ['Indistinguishable', 'Slight edge', 'Clear gap', 'One is unacceptable'] },
},
});Jev does not write; it picks from an answer space you define in advance and returns probabilities. On the same 2,390 questions its true/false accuracy matched GPT-5.6 Sol while running 3× faster, costing 88× less, and being confidently wrong about one sixth as often.
Many. Fifty questions in one call took about the same wall-clock time as one, and only 2 of the 50 answers differed from asking them separately. At 100 per call the end-to-end time became erratic — 1.0 s, 4.1 s and 8.2 s across three runs.
On true/false, expected calibration error is 0.048 — level with GPT-5.6 Sol. On multiple choice it is overconfident: an average of 89% stated confidence against 77% actual accuracy. Leave headroom when you set thresholds.
Jev cannot. It returns typed answers and probabilities, nothing else. The usual pattern is to let Jev judge everything and send only the failures and the uncertain cases to a large model for a written rationale — which is a small fraction of the traffic.
Each of these has a line of code that calls Jev. More of them in awesome-jev-verified.
The questions above are fixed. To write your own criteria, options and score levels, open the playground.
Open the playground