Benchmarks & research

jev-labs

@copyleftdev★ 1PythonMITupdated 2026-09-19▶ runnable

Never confidently wrong: a TLA+-verified consensus kernel around TypeSafe's Jev, run through 1,680 chaos-tested pharmacy decisions with zero wrong verdicts. Film, code, and every captured call.

copyleftdev/jev-labs ↗

Where it calls Jev

const DEFAULT_BASE_URL: &str = "https://api.typesafe.ai";

consensus/rust/consensus-kernel/src/jev.rs:25

The link points at the commit we read, so the line number still holds.

What it asks Jev

These question sets are lifted from this project's source exactly as written.

from experiments.py:52

  1. is_regulationtrue/false

    Is the subject of this text a law or regulation?

  2. mentions_deadlinetrue/false

    Does this text state a reporting deadline?

  3. mentions_finestrue/false

    Does this text describe financial penalties?

  4. mentions_scopetrue/false

    Does this text describe a geographic scope?

  5. mentions_datetrue/false

    Does this text state a date it took effect?

  6. mentions_controllerstrue/false

    Does this text place obligations on data controllers?

  7. mentions_consenttrue/false

    Does this text discuss obtaining user consent?

  8. mentions_dpotrue/false

    Does this text mention a Data Protection Officer?

Run it on your own text

The code
import { experimental_evaluate as evaluate } from 'ai';

const { answers } = await evaluate({
  model: 'typesafe-ai/jev',
  state,
  questions: {
    is_regulation: { type: 'boolean', instructions: 'Is the subject of this text a law or regulation?' },
    mentions_deadline: { type: 'boolean', instructions: 'Does this text state a reporting deadline?' },
    mentions_fines: { type: 'boolean', instructions: 'Does this text describe financial penalties?' },
    mentions_scope: { type: 'boolean', instructions: 'Does this text describe a geographic scope?' },
    mentions_date: { type: 'boolean', instructions: 'Does this text state a date it took effect?' },
    mentions_controllers: { type: 'boolean', instructions: 'Does this text place obligations on data controllers?' },
    mentions_consent: { type: 'boolean', instructions: 'Does this text discuss obtaining user consent?' },
    mentions_dpo: { type: 'boolean', instructions: 'Does this text mention a Data Protection Officer?' },
  },
});

from experiments.py:249

  1. is_regulationtrue/false

    Is the subject of this text a law or regulation?

  2. mentions_consenttrue/false

    Does this text discuss obtaining user consent?

  3. severityscore

    How severe are the penalties described?

    • 0 — No penalties
    • 1 — Moderate penalties
    • 2 — Severe penalties
  4. topicchoice

    What is the primary subject of this text?

    • privacy_law — Data protection or privacy regulation
    • tax_law — Taxation rules
    • employment_law — Labour or employment rules

Run it on your own text

The code
import { experimental_evaluate as evaluate } from 'ai';

const { answers } = await evaluate({
  model: 'typesafe-ai/jev',
  state,
  questions: {
    is_regulation: { type: 'boolean', instructions: 'Is the subject of this text a law or regulation?' },
    mentions_consent: { type: 'boolean', instructions: 'Does this text discuss obtaining user consent?' },
    severity: { type: 'score', instructions: 'How severe are the penalties described?',
      criteria: ['No penalties', 'Moderate penalties', 'Severe penalties'] },
    topic: { type: 'choice', instructions: 'What is the primary subject of this text?',
      criteria: { privacy_law: 'Data protection or privacy regulation', tax_law: 'Taxation rules', employment_law: 'Labour or employment rules' } },
  },
});

from harness/noise.py:110

  1. billingtrue/false

    Is this message about a billing or payment matter?

  2. technicaltrue/false

    Does this message report a technical malfunction?

  3. urgenttrue/false

    Does this message convey urgency?

Run it on your own text

The code
import { experimental_evaluate as evaluate } from 'ai';

const { answers } = await evaluate({
  model: 'typesafe-ai/jev',
  state,
  questions: {
    billing: { type: 'boolean', instructions: 'Is this message about a billing or payment matter?' },
    technical: { type: 'boolean', instructions: 'Does this message report a technical malfunction?' },
    urgent: { type: 'boolean', instructions: 'Does this message convey urgency?' },
  },
});

from smoke_test.py:36

  1. departmentchoice

    Which team should handle this ticket?

    • billing — Payment, invoicing, or subscription issues
    • technical — Bugs, outages, or integration problems
    • sales — Pricing, upgrades, or new accounts
  2. frustrationscore

    How frustrated the customer appears

    • 0 — Calm, just stating facts
    • 1 — Frustrated but civil
    • 2 — Very angry, strong language
  3. is_urgenttrue/false

    The message conveys urgency or time-sensitivity

Run it on your own text

The code
import { experimental_evaluate as evaluate } from 'ai';

const { answers } = await evaluate({
  model: 'typesafe-ai/jev',
  state,
  questions: {
    department: { type: 'choice', instructions: 'Which team should handle this ticket?',
      criteria: { billing: 'Payment, invoicing, or subscription issues', technical: 'Bugs, outages, or integration problems', sales: 'Pricing, upgrades, or new accounts' } },
    frustration: { type: 'score', instructions: 'How frustrated the customer appears',
      criteria: ['Calm, just stating facts', 'Frustrated but civil', 'Very angry, strong language'] },
    is_urgent: { type: 'boolean', instructions: 'The message conveys urgency or time-sensitivity' },
  },
});

Other projects in this category