patronus-protect commited on
Commit
8be7b54
·
verified ·
1 Parent(s): eb7545f

Model card: link the public benchmark dataset

Browse files
Files changed (1) hide show
  1. README.md +6 -2
README.md CHANGED
@@ -19,6 +19,8 @@ language:
19
  base_model:
20
  - jhu-clsp/mmBERT-small
21
  base_model_relation: finetune
 
 
22
  library_name: transformers
23
  pipeline_tag: text-classification
24
  metrics:
@@ -292,6 +294,8 @@ The detector is evaluated separately from the full gate. Its score measures whet
292
  | **IFlippedJev**: 1,000 claims that flipped at least one of three local decision models on a real case | 46.8 % found | **96.6 %** found |
293
  | **NotClaim**: 1,343 claims and 1,343 benign texts that use the same claim vocabulary | 77.7 % found, 11.0 % of benign flagged, AUROC 0.928 | **98.1 %** found, **2.2 %** of benign flagged, AUROC **0.998** |
294
 
 
 
295
  Threshold `0.3`, PyTorch FP32 checkpoints, inputs up to 128 tokens. The INT8 + INT4 graph gives 96.4 %, 96.3 %, 98.4 % and 2.4 % on the same sets.
296
 
297
  <details>
@@ -535,10 +539,10 @@ Both detectors split these texts into the same number of tokens (34.7 and 35.1 o
535
  | :--- | :--- | :--- |
536
  | Claim-detector training mix | Training | 215,315 text units in 15 languages: 158k units of the v1 corpus (after near-duplicate removal), 41k red-team units (claims tested against local decision models, translated into 14 languages, plus hard negatives), 11k units from two further claim collections and 5.6k contrast negatives |
537
  | Detector validation split | Layer selection, checkpoint comparison, ONNX validation | 13,155 units: 5,702 claims and 7,453 content units; same distribution as training |
538
- | Red-team holdout, IFlippedJev, NotClaim | Held-out evaluation | see [claim detector](#claim-detector) |
539
  | Decision-steering benchmark | Held-out evaluation of the full gate | Five tasks, 100 cases per task, three attacks and three neutral additions per case |
540
 
541
- This repository publishes [aggregated evaluation results](https://huggingface.co/patronus-studio/decision-steering-gate/blob/main/eval_results.json), not the raw training or benchmark datasets.
542
 
543
  ### Training recipe
544
 
 
19
  base_model:
20
  - jhu-clsp/mmBERT-small
21
  base_model_relation: finetune
22
+ datasets:
23
+ - patronus-studio/decision-steering-benchmarks
24
  library_name: transformers
25
  pipeline_tag: text-classification
26
  metrics:
 
294
  | **IFlippedJev**: 1,000 claims that flipped at least one of three local decision models on a real case | 46.8 % found | **96.6 %** found |
295
  | **NotClaim**: 1,343 claims and 1,343 benign texts that use the same claim vocabulary | 77.7 % found, 11.0 % of benign flagged, AUROC 0.928 | **98.1 %** found, **2.2 %** of benign flagged, AUROC **0.998** |
296
 
297
+ The three sets are public: [patronus-studio/decision-steering-benchmarks](https://huggingface.co/datasets/patronus-studio/decision-steering-benchmarks), with the cases of IFlippedJev to replay against your own decision model.
298
+
299
  Threshold `0.3`, PyTorch FP32 checkpoints, inputs up to 128 tokens. The INT8 + INT4 graph gives 96.4 %, 96.3 %, 98.4 % and 2.4 % on the same sets.
300
 
301
  <details>
 
539
  | :--- | :--- | :--- |
540
  | Claim-detector training mix | Training | 215,315 text units in 15 languages: 158k units of the v1 corpus (after near-duplicate removal), 41k red-team units (claims tested against local decision models, translated into 14 languages, plus hard negatives), 11k units from two further claim collections and 5.6k contrast negatives |
541
  | Detector validation split | Layer selection, checkpoint comparison, ONNX validation | 13,155 units: 5,702 claims and 7,453 content units; same distribution as training |
542
+ | Red-team holdout, IFlippedJev, NotClaim | Held-out evaluation | see [claim detector](#claim-detector); [public dataset](https://huggingface.co/datasets/patronus-studio/decision-steering-benchmarks) |
543
  | Decision-steering benchmark | Held-out evaluation of the full gate | Five tasks, 100 cases per task, three attacks and three neutral additions per case |
544
 
545
+ This repository publishes [aggregated evaluation results](https://huggingface.co/patronus-studio/decision-steering-gate/blob/main/eval_results.json). The held-out benchmarks are a public dataset: [patronus-studio/decision-steering-benchmarks](https://huggingface.co/datasets/patronus-studio/decision-steering-benchmarks). The training data is not published.
546
 
547
  ### Training recipe
548