Provael — prove it, prevail

Provael red-teams open Vision-Language-Action (VLA) robot policies in simulation and reports an Attack Success Rate with the evidence a regulator or certifier can actually read: SARIF, OSCAL, and a CycloneDX ML-BOM. Open-core, CPU-first, Apache-2.0.

Everything below is simulation only. Zero runs on physical hardware, and the tooling is defensive — attack families are published so policy builders can measure their own exposure.

The measured result

One reframed instruction (roleplay) drove a real SmolVLA policy off its benign task on 44 of 50 matched pairs across all ten libero_object tasks, against 0 benign twins at the same (task, seed), in LIBERO simulation. McNemar exact p = 4.6e-13, Holm-adjusted 2.7e-12; task-clustered 95% CI [72%, 100%].

Read alongside it, because a rate without these is not a result:


Benign false-positive control	2/50 (4%) — the same predicate on unattacked runs
Clean task success	42/50 (84%) — the competence baseline the ASR is read against
Task success under the attack	0/50 — it destroys completion, not just safety
Instruction family overall	62/150 (41.3%)

Caveats that travel with the number. The keep-out predicate is uncalibrated — it counts trials driven outside the configured benign envelope, not a certified hazard threshold. Zero observed benign events does not establish a zero benign rate; that Wilson interval runs 0–27.8%. One policy, one suite. This does not transfer to your checkpoint until you run it there.

The nulls, published as measured

Two attack families were run against the same real policy and did not transfer:

decoy_object 0/50 · patch 0/50 — the visual family, 0/100
scene_text 0/50 — injection

These stay published exactly as measured. Three separate papers now report image-space patch attacks that far exceed our 0/50 on the same channel; the honest reading is that Provael has no image-space patch attack, and these numbers measure its absence rather than the policy's safety.

Registered is not measured

15 adversarial attack families are registered — implemented and unit-tested in the CPU harness. 3 have been exercised against a real policy in a real simulator. 12 have never met a real model. Registration implies no real-policy transfer, no hardware corroboration, and no external reproduction.

Coverage against the Embodied AI Security Top 10 is 8 of 10. The two uncovered risks are named rather than dropped: EAI07 (CPS/firmware/comms) and EAI10 (evaluation/observability) are out of scope for a VLA-policy red-teamer by design.

The leaderboard

Provael ASR leaderboard — attack-success rates with their benign controls. The Space is a rendering; the signed JSON in the repository is the artifact to verify.
Submissions dataset — open to anyone, via provael submit.

Elsewhere

Independent project, solo maintainer. Naming a benchmark or standards body anywhere here is not an affiliation or an endorsement, and none is implied in either direction.