Benchmarks & research
TypeSafe AI (jev) 시범 사용 프로젝트
BASE_URL="${TYPESAFE_BASE_URL:-https://api.typesafe.ai}"The link points at the commit we read, so the line number still holds.
These question sets are lifted from this project's source exactly as written.
from samples/comment_triage/triage.py:68
impactscoreRate the likely cost and schedule impact on the Builder of implementing `comment.text`.
0 — None: a typo fix, a confirmation, or a change absorbed in normal drawing revision.1 — Minor: drawing reissue and small material change, no effect on any milestone.2 — Moderate: re-work of one to four weeks, new material order, or a vendor re-engineering loop.3 — Major: re-work over four weeks, a milestone at risk, or a change to class-approved design.import { experimental_evaluate as evaluate } from 'ai';
const { answers } = await evaluate({
model: 'typesafe-ai/jev',
state,
questions: {
impact: { type: 'score', instructions: 'Rate the likely cost and schedule impact on the Builder of implementing `comment.text`.',
criteria: ['None: a typo fix, a confirmation, or a change absorbed in normal drawing revision.', 'Minor: drawing reissue and small material change, no effect on any milestone.', 'Moderate: re-work of one to four weeks, new material order, or a vendor re-engineering loop.', 'Major: re-work over four weeks, a milestone at risk, or a change to class-approved design.'] },
},
});from tests/test_mock_systemone.py:70
schedule_risktrue/falseWill this delay erection?
import { experimental_evaluate as evaluate } from 'ai';
const { answers } = await evaluate({
model: 'typesafe-ai/jev',
state,
questions: {
schedule_risk: { type: 'boolean', instructions: 'Will this delay erection?' },
},
});