Add safeagents judge bench results

#2
by ybelkada HF Staff - opened

Hi! This adds knowledgator/gliclass-v4-edge's results on the safeagents judge bench, a benchmark of real agent tool calls that safeagents' scanner flags as suspicious, each labelled block or allow and with a kind of misuse (reward hacking, harness introspection, credential misuse, ...).

On the 562 test calls: balanced accuracy 0.644, AUROC 0.684, false-block rate 0.41, category macro F1 0.253. Run zero-shot, one System One request per call, on NVIDIA L4 (HF Jobs). Per-call verdicts are in the dataset's results/ folder.

The file is named after the benchmark, so it doesn't touch any existing .eval_results file.

Updated to the v2 release.

Hi! This adds knowledgator/gliclass-v4-edge's results on the safeagents judge bench v2, a benchmark of real agent tool calls that safeagents' scanner flags as suspicious, each labelled block or allow and with a kind of misuse (reward hacking, harness introspection, credential misuse, ...).

On the 661 test calls: balanced accuracy 0.676, AUROC 0.753, false-block rate 0.344, category macro F1 0.373. Run zero-shot, one System One request per call, on NVIDIA L4 (HF Jobs). Per-call verdicts are in the dataset's results/ folder.

The file is named after the benchmark, so it doesn't touch any existing .eval_results file. v2 keeps every earlier call and adds new ones (see the dataset card); these numbers replace the earlier ones in this file.

Ready to merge
This branch is ready to get merged automatically.

Sign up or log in to comment