by SeraphimSerapis · Agent Tool · ★ 172
tool-eval-bench A tool-calling quality benchmark for evaluating LLM tool-use in agentic workflows across open-weight model serving stacks (vLLM, LiteLLM, llama.cpp). Also includes pluggable accuracy benchmarks (GSM8K, MMLU, IFEval) via the same OpenAI-compatible endpoints. Inspired by ToolCall-15, this tool runs 69 deterministic scenarios (+ 15 opt-in Hard Mode) through OpenAI-compatible endpoints, scores each result as pass, partial, or fail, and produces detailed trace reports.
| Stars | 172 |
| Forks | 17 |
| Language | Python |
| Category | Agent Tool |
| License | MIT |
| Quality Score | 72.5566441930571/100 |
| Last Updated | 2026-07-06 |
| Created | 2026-04-17 |
| Platforms | python |
| Est. Tokens | ~23k |
These tools work well together with tool-eval-bench for enhanced workflows:
Explore other popular agent tool tools:
tool-eval-bench is Tool-calling quality benchmark for LLM serving stacks. 80+ deterministic scenarios testing multi-turn orchestration, safety boundaries, and structured output. Supports vLLM, SGLang, and llama.cpp.. It is categorized as a Agent Tool with 172 GitHub stars.
tool-eval-bench is primarily written in Python.
You can find installation instructions and usage details in the tool-eval-bench GitHub repository at github.com/SeraphimSerapis/tool-eval-bench. The project has 172 stars and 17 forks, indicating an active community.
tool-eval-bench is released under the MIT license, making it free to use and modify according to the license terms.