Action Oracle: Did the action actually happen?
A runnable agent action evaluation that catches false claims of success. Seven synthetic cases cover a normal write, timeout before or after commit, a success response with no write, revoked permission, stale state, and duplicate retry. A local service and deterministic grader check final receipts, duplicate effects, authorization, and the agent’s structured answer. Includes good and bad traces plus an adapter for your own agent.
Coding By Matthew Keeley
What you get
- Contents
- 1 files · Custom folder / seller-provided setup
- Made for
- AI agent builders, evaluation engineers, and release owners testing tool writes and recovery.
Setup and compatibilityHow to run the pack with your agent
This custom-format eval is supplied as the seller’s original folder. Download the ZIP and follow the seller’s instructions. EvalMarket provides no execution or grading integration for these files.
Python 3.10+; runs locally with the standard library and no model key for the reference run. Tested on macOS with Python 3.12 and 3.14. Connecting your own agent requires a subprocess adapter that can call the loopback HTTP service.
Unzip, enter the action-oracle-0.1.0 directory, and run python3 -m action_eval run. Run python3 -m action_eval audit to see known-good and bad controls. To test your agent, implement the documented JSON stdin/stdout contract and run python3 -m action_eval run --agent "python3 your_adapter.py". Read README.md before use.
Scope and limitationsKnow what this pack does and does not measure
Limitations
Synthetic exact-one action only. HTTP 504 models ambiguous writes; it does not simulate dropped connections. The grader checks structured final fields, not free-form prose. A reference pass verifies the fixture, not your agent or production reliability. No automatic telemetry or hosted execution.
Review covers version 1 of this pack, not the readiness of your agent.
Reviews and questionsBuyer feedback and answers from the seller
Buyer reviews
How well does this eval measure what it claims? Reviews come from people who acquired it through EvalMarket.
Loading reviews…
Checking review access…
Questions
Ask about compatibility, coverage or setup. The seller can answer here.
Loading questions…
Checking sign-in…