Benchmarks & research

jev-as-a-judge

@danielgshea★ 12Pythonupdated 2026-09-19▶ runnable

Using Jev as an evaluator.

danielgshea/jev-as-a-judge ↗

Where it calls Jev

JUDGES = ("jev", "gpt_luna", "gpt_terra", "claude_sonnet")

src/evals/judge_reliability.py:12

The link points at the commit we read, so the line number still holds.

What it asks Jev

These question sets are lifted from this project's source exactly as written.

from src/evals/judges/system_one.py:32

  1. does_passtrue/false

    Does the final answer pass?

Run it on your own text

The code
import { experimental_evaluate as evaluate } from 'ai';

const { answers } = await evaluate({
  model: 'typesafe-ai/jev',
  state,
  questions: {
    does_pass: { type: 'boolean', instructions: 'Does the final answer pass?' },
  },
});

Other projects in this category