Benchmarks & research
A garage full of tiny experiments for building critical systems with System One & Jev ๐ง๐ง โก
from typesafe_sdk import (The link points at the commit we read, so the line number still holds.
These question sets are lifted from this project's source exactly as written.
from berserk/global-payment-incident/demo.py:54
failure_domainchoiceIdentify the dominant failure domain supported by the complete incident snapshot.
application_release โ The active application release is the primary causal factor.payment_provider โ An external payment provider is the primary causal factor.routing_configuration โ Routing or timeout configuration is the primary causal factor.capacity โ Internal capacity exhaustion is the primary causal factor.multi_factor โ Several coupled causes prevent a single dominant attribution.unknown โ Evidence is insufficient for stable attribution.customer_impactscoreScore current customer and financial impact.
0 โ contained1 โ minor2 โ material3 โ major4 โ systemicsystemic_outagetrue/falseIs this a multi-region, customer-visible payment outage requiring incident command?
import { experimental_evaluate as evaluate } from 'ai';
const { answers } = await evaluate({
model: 'typesafe-ai/jev',
state,
questions: {
failure_domain: { type: 'choice', instructions: 'Identify the dominant failure domain supported by the complete incident snapshot.',
criteria: { application_release: 'The active application release is the primary causal factor.', payment_provider: 'An external payment provider is the primary causal factor.', routing_configuration: 'Routing or timeout configuration is the primary causal factor.', capacity: 'Internal capacity exhaustion is the primary causal factor.', multi_factor: 'Several coupled causes prevent a single dominant attribution.', unknown: 'Evidence is insufficient for stable attribution.' } },
customer_impact: { type: 'score', instructions: 'Score current customer and financial impact.',
criteria: ['contained', 'minor', 'material', 'major', 'systemic'] },
systemic_outage: { type: 'boolean', instructions: 'Is this a multi-region, customer-visible payment outage requiring incident command?' },
},
});from berserk/global-payment-incident/demo.py:86
change_riskscoreScore the blast radius and reversibility risk of the selected intervention.
0 โ minimal1 โ low2 โ moderate3 โ high4 โ unacceptableevidence_sufficienttrue/falseIs the evidence sufficient for operators to consider the selected intervention now?
import { experimental_evaluate as evaluate } from 'ai';
const { answers } = await evaluate({
model: 'typesafe-ai/jev',
state,
questions: {
change_risk: { type: 'score', instructions: 'Score the blast radius and reversibility risk of the selected intervention.',
criteria: ['minimal', 'low', 'moderate', 'high', 'unacceptable'] },
evidence_sufficient: { type: 'boolean', instructions: 'Is the evidence sufficient for operators to consider the selected intervention now?' },
},
});from berserk/infinity-of-jevs/demo.py:58
positionchoiceChoose the seismic hazard interpretation best supported by the source and prior consensus.
background โ Activity is consistent with ordinary global background seismicity.elevated_cluster โ A meaningful cluster or sequence warrants focused monitoring.major_event โ A major event creates credible disruption potential.unclear โ The evidence cannot support a stable hazard interpretation.strengthscoreScore the operational strength of the seismic hazard signal.
0 โ negligible1 โ weak2 โ moderate3 โ strong4 โ extremesupportedtrue/falseIs this hazard interpretation directly supported by the supplied events?
import { experimental_evaluate as evaluate } from 'ai';
const { answers } = await evaluate({
model: 'typesafe-ai/jev',
state,
questions: {
position: { type: 'choice', instructions: 'Choose the seismic hazard interpretation best supported by the source and prior consensus.',
criteria: { background: 'Activity is consistent with ordinary global background seismicity.', elevated_cluster: 'A meaningful cluster or sequence warrants focused monitoring.', major_event: 'A major event creates credible disruption potential.', unclear: 'The evidence cannot support a stable hazard interpretation.' } },
strength: { type: 'score', instructions: 'Score the operational strength of the seismic hazard signal.',
criteria: ['negligible', 'weak', 'moderate', 'strong', 'extreme'] },
supported: { type: 'boolean', instructions: 'Is this hazard interpretation directly supported by the supplied events?' },
},
});from berserk/infinity-of-jevs/demo.py:77
positionchoiceChoose the plausible exposure pattern after considering hazard and prior-round evidence.
remote โ Events are unlikely to disrupt significant infrastructure or population centers.regional โ One region could face meaningful disruption.multi_region โ Several regions or global networks could be affected.unclear โ Locations and impact context do not support a stable exposure judgment.strengthscoreScore plausible infrastructure and population exposure.
0 โ negligible1 โ low2 โ moderate3 โ high4 โ extremesupportedtrue/falseIs this exposure judgment supported without inventing missing asset data?
import { experimental_evaluate as evaluate } from 'ai';
const { answers } = await evaluate({
model: 'typesafe-ai/jev',
state,
questions: {
position: { type: 'choice', instructions: 'Choose the plausible exposure pattern after considering hazard and prior-round evidence.',
criteria: { remote: 'Events are unlikely to disrupt significant infrastructure or population centers.', regional: 'One region could face meaningful disruption.', multi_region: 'Several regions or global networks could be affected.', unclear: 'Locations and impact context do not support a stable exposure judgment.' } },
strength: { type: 'score', instructions: 'Score plausible infrastructure and population exposure.',
criteria: ['negligible', 'low', 'moderate', 'high', 'extreme'] },
supported: { type: 'boolean', instructions: 'Is this exposure judgment supported without inventing missing asset data?' },
},
});