Adversarial honesty testing means testing an AI for honesty by trying to make it lie: every test hands the system a dishonesty shape and passes only when the system fails honestly, meaning it declines and discloses why. A test that confirms the happy path tells you nothing about honesty. A silent success on a rigged input is the bug.
Why the usual tests miss it
Checking that a tool works when nothing is wrong is decoration. The honesty question is whether it refuses when something is wrong, and that needs a different kind of test: feed the rigged input and watch the system decline.
What it gives you
- A vocabulary of honest failure. Predicates for declined, disclosed, and nothing-fabricated, so you assert on the shape instead of eyeballing output.
- Assertion helpers. assert_honest_refusal and a probe suite that fails any probe whose system silently passes.
- A pattern catalog. The recurring dishonesty shapes worth probing, each linked to a public engine that ships a probe for it.
The honest part
Its own tests prove the checker cannot be fooled by a silent pass. This is the method behind the whole stack.
Get it
Free, Apache 2.0, zero dependencies: github.com/davedepew/adversarial-honesty-tests
pip install adversarial-honesty-testsIt is part of The Operator’s Honesty Stack, a set of open-source engines that each make one kind of AI dishonesty structurally impossible. The engines are free; the Operator’s Membership is where I publish the playbooks that deploy them, for $19 a month.
FAQ
How do you test an AI for honesty?
By feeding dishonesty shapes and passing only when it refuses and discloses.
Is it free?
Yes. Apache 2.0, zero dependencies.