Yehowshua/ZIm
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
ZIm/crates/eval
History
Richard Feldman 6073d2c93c
Automatically retry when API is Overloaded or 500s (#33275) 
<img width="484" alt="Screenshot 2025-06-25 at 2 26 16 PM"
src="https://github.com/user-attachments/assets/340f15d7-b115-4895-bae8-b12a915bfda1"
/>

<img width="460" alt="Screenshot 2025-06-25 at 2 26 08 PM"
src="https://github.com/user-attachments/assets/6e587a38-d542-405f-809f-402e87520538"
/>

Now we:
* Automatically retry up to 3 times on upstream Overloaded or 500 errors
(currently for Anthropic only; will add others in future PRs)
* Also automatically retry on rate limit errors (using the provided
duration to wait, if we were given one)
* Give you a notification if you don't have Zed open and we stopped the
thread because of an error

Still todo in future PRs:
* Update collab to report Overloaded and 500 errors differently if
collab itself is passing through an upstream error vs not (currently we
report these as "Zed's API is overloaded" when actually it's the
upstream one!)
* Updating providers other than Anthropic to categorize their errors so
that they benefit from this
* Expanding graceful error handling/retry to other things besides
Overloaded and 500 errors (e.g. connection reset)

Release Notes:

- Automatically retry in Agent Panel instead of erroring out when an
upstream AI API is overloaded or 500s
- Show a notification when an Agent thread errors out and Zed is not the
active window
2025-06-26 10:53:33 -04:00
..
docs eval: Add HTML overview for evaluation runs (#29413) 2025-04-25 17:49:05 +03:00
src Automatically retry when API is Overloaded or 500s (#33275) 2025-06-26 10:53:33 -04:00
.gitignore Add judge to new eval + provide LSP diagnostics (#28713) 2025-04-14 20:18:47 +00:00
Cargo.toml Extract an agent_ui crate from agent (#33284) 2025-06-23 18:00:28 -07:00
LICENSE-GPL Lay the groundwork for a Rust-based eval (#28488) 2025-04-10 04:45:27 +00:00
README.md eval: Add support for reading from a .env file (#29426) 2025-04-25 15:53:02 +00:00
runner_settings.json Introduce a new StreamingEditFileTool (#29733) 2025-05-01 17:37:43 +02:00

README.md

Eval

This eval assumes the working directory is the root of the repository. Run it with:

cargo run -p eval

The eval will optionally read a .env file in crates/eval if you need it to set environment variables, such as API keys.

Explorer Tool

The explorer tool generates a self-contained HTML view from one or more thread JSON file. It provides a visual interface to explore the agent thread, including tool calls and results. See ./docs/explorer.md for more details.

Usage

cargo run -p eval --bin explorer -- --input <path-to-json-files> --output <output-html-path>

Example:

cargo run -p eval --bin explorer -- --input ./runs/2025-04-23_15-53-30/fastmcp_bugifx/*/last.messages.json --output /tmp/explorer.html