What is APIEval-20?
APIEval-20 is an AI benchmark from KushoAI that measures how well API-testing agents can find bugs when given only a JSON schema and a single sample payload. It runs an agent's generated test suite against live reference APIs that contain planted bugs, then scores bug-detection accuracy, coverage, and efficiency objectively. The benchmark spans 20 scenarios across 7 domains, so results reflect performance on varied real-world API shapes rather than a single case. For developers and QA teams building or evaluating AI test-generation agents, APIEval-20 provides a repeatable yardstick to compare approaches and see where an agent misses edge cases. It's positioned as a research-grade benchmark for the API-testing-agent space rather than a tool you point at your own production APIs.
What is APIEval-20 used for?
APIEval-20 is used by developers and QA teams to benchmark AI API-testing agents on bug detection, coverage, and efficiency using only a schema and a sample payload.
How much does APIEval-20 cost?
Pros and cons
Pros
• Scores agents from just a JSON schema and one sample payload • Tests against live reference APIs with planted bugs • 20 scenarios across 7 domains for varied coverage • Objective metrics: bug detection, coverage, efficiency
Cons
• A benchmark, not a tool to test your own production APIs • Aimed at developers building API-testing agents • Narrow, research-focused scope
Questions about APIEval-20
What is APIEval-20 used for?
Benchmarking how well AI API-testing agents detect bugs and cover APIs using only a JSON schema and a sample payload.
How does APIEval-20 work?
It runs an agent's generated test suite against live reference APIs with planted bugs and scores detection accuracy, coverage, and efficiency.
Who is APIEval-20 for?
Developers and QA teams building or evaluating AI agents for automated API testing.
Is APIEval-20 free?
It's a benchmark resource from KushoAI; see the KushoAI page for current access details.
Reviews
Sign in to rate and review APIEval-20.
Sign in to reviewNo reviews yet. Be the first to review APIEval-20!