ZIm/crates/eval
Ben Brandt 709523bf36
Store profile per thread (#31907)
This allows storing the profile per thread, as well as moving the logic
of which tools are enabled or not to the profile itself.

This makes it much easier to switch between profiles, means there is
less global state being changed on every profile change.

Release Notes:

- agent panel: allow saving the profile per thread

---------

Co-authored-by: Ben Kunkle <ben.kunkle@gmail.com>
2025-06-06 12:05:27 +00:00
..
docs eval: Add HTML overview for evaluation runs (#29413) 2025-04-25 17:49:05 +03:00
src Store profile per thread (#31907) 2025-06-06 12:05:27 +00:00
.gitignore Add judge to new eval + provide LSP diagnostics (#28713) 2025-04-14 20:18:47 +00:00
Cargo.toml Pass up intent with completion requests (#31710) 2025-05-29 20:43:12 +00:00
LICENSE-GPL Lay the groundwork for a Rust-based eval (#28488) 2025-04-10 04:45:27 +00:00
README.md eval: Add support for reading from a .env file (#29426) 2025-04-25 15:53:02 +00:00
runner_settings.json Introduce a new StreamingEditFileTool (#29733) 2025-05-01 17:37:43 +02:00

Eval

This eval assumes the working directory is the root of the repository. Run it with:

cargo run -p eval

The eval will optionally read a .env file in crates/eval if you need it to set environment variables, such as API keys.

Explorer Tool

The explorer tool generates a self-contained HTML view from one or more thread JSON file. It provides a visual interface to explore the agent thread, including tool calls and results. See ./docs/explorer.md for more details.

Usage

cargo run -p eval --bin explorer -- --input <path-to-json-files> --output <output-html-path>

Example:

cargo run -p eval --bin explorer -- --input ./runs/2025-04-23_15-53-30/fastmcp_bugifx/*/last.messages.json --output /tmp/explorer.html