Formally verifiable abstract-reasoning task generated by an LLM pipeline with code-execution validation (A2RBench, 1,054 tasks: seeds plus 631 expansions)
$0.016 per problem averaged over 1,054 tasks per task, actual API cost
Dated 17 May 2026Source checked 19 Oct 2026Its page was there at the last check, 28 Oct 2026; it is, or links to, a document whose words the automatic check could not matchBasis: stated
In the source's own words
“Seed generation, which involves multi-stage LLM verification (rule design, code implementation, and cycle consistency checks), costs $0.19 per task. ... Averaged across the entire dataset of 1,054 tasks, the cost is $0.016 per problem.”
What it is about
Figures it counts in
- $37 per taskDatasets, tasks and environments, cost to produce, per task
Cited in this guide