评测与研究

typesafe-ai-firewall

@AnshChoudhary1Python更新于 2026-09-17可当场跑

TypeSafe/Jev 的 AI agent 工具调用预执行防火墙的影子模式验证工具,实际运行结果见 report.md。

英文原文

Shadow-mode validation harness for a pre-execution firewall on AI agent tool calls (TypeSafe/Jev). Real run, findings in report.md.

AnshChoudhary/typesafe-ai-firewall

它在哪儿调用了 Jev

"""Same battery as a raw HTTP payload, for POST /v1/systemone without the SDK."""

firewall/battery.py:85

链接指向我们抓取当天的那个 commit,行号是准的。

它问 Jev 的问题

下面是从这个项目源码里原样取出来的 question 组合。

取自 firewall/judge.py:158

  1. is_urgent是/否

    The message conveys urgency

  2. severity打分

    How severe the problem is for the customer

    • 0Minor annoyance
    • 1Real problem
    • 2Losing money now

用你自己的内容跑一遍

代码
import { experimental_evaluate as evaluate } from 'ai';

const { answers } = await evaluate({
  model: 'typesafe-ai/jev',
  state,
  questions: {
    is_urgent: { type: 'boolean', instructions: 'The message conveys urgency' },
    severity: { type: 'score', instructions: 'How severe the problem is for the customer',
      criteria: ['Minor annoyance', 'Real problem', 'Losing money now'] },
  },
});

同类的其他项目