ZIm/crates/eval
Oleksiy Syvokon d87603dd60
agent: Send stale file notifications using the project_notifications tool (#34005)
This commit introduces the `project_notifications` tool, which
proactively pushes notifications to the agent.

Unlike other tools, `Thread` automatically invokes this tool on every
turn, even when the LLM doesn't ask for it. When notifications are
available, the tool use and results are inserted into the thread,
simulating an LLM tool call.

As with other tools, users can disable `project_notifications` in
Profiles if they do not want them.

Currently, the tool only notifies users about stale files: that is,
files that have been edited by the user while the agent is also working
on them. In the future, notifications may be expanded to include
compiler diagnostics, long-running processes, and more.

Release Notes:

- Added `project_notifications` tool
2025-07-07 19:48:18 +03:00
..
docs eval: Add HTML overview for evaluation runs (#29413) 2025-04-25 17:49:05 +03:00
src agent: Send stale file notifications using the project_notifications tool (#34005) 2025-07-07 19:48:18 +03:00
.gitignore Add judge to new eval + provide LSP diagnostics (#28713) 2025-04-14 20:18:47 +00:00
Cargo.toml debugger: Handle the envFile setting for Go (#33666) 2025-07-01 09:14:59 -07:00
LICENSE-GPL Lay the groundwork for a Rust-based eval (#28488) 2025-04-10 04:45:27 +00:00
README.md eval: Add support for reading from a .env file (#29426) 2025-04-25 15:53:02 +00:00
runner_settings.json Introduce a new StreamingEditFileTool (#29733) 2025-05-01 17:37:43 +02:00

Eval

This eval assumes the working directory is the root of the repository. Run it with:

cargo run -p eval

The eval will optionally read a .env file in crates/eval if you need it to set environment variables, such as API keys.

Explorer Tool

The explorer tool generates a self-contained HTML view from one or more thread JSON file. It provides a visual interface to explore the agent thread, including tool calls and results. See ./docs/explorer.md for more details.

Usage

cargo run -p eval --bin explorer -- --input <path-to-json-files> --output <output-html-path>

Example:

cargo run -p eval --bin explorer -- --input ./runs/2025-04-23_15-53-30/fastmcp_bugifx/*/last.messages.json --output /tmp/explorer.html