Yehowshua/ZIm - Forgejo: Beyond coding. We Forge.

Author SHA1 Message Date

Author	SHA1	Message	Date
Piotr Osiewicz	05fc0c432c	Fix a bunch of other low-hanging style lints (#36498 ) - Fix a bunch of low hanging style lints like unnecessary-return - Fix single worktree violation - And the rest Release Notes: - N/A	2025-08-19 21:26:17 +02:00
Oleksiy Syvokon	6420df3975	eval: Count execution errors as failures (#30712 ) - Evals returning an error (e.g., LLM API format mismatch) were silently skipped in the aggregated results. Now we count them as a failure (0% success score). - Setting the `VERBOSE` environment variable to something non-empty disables string truncation Release Notes: - N/A	2025-05-14 20:44:19 +03:00
Agus Zubiaga	ce1a674eba	eval: Fine-grained assertions (#29246 ) - Support programmatic examples ([example](`17feb260a0/crates/eval/src/examples/file_search.rs`)) - Combine data-driven example declarations into a single `.toml` file ([example](`17feb260a0/crates/eval/src/examples/find_and_replace_diff_card.toml`)) - Run judge on individual assertions (previously called "criteria") - Report judge and programmatic assertions in one combined table Note: We still need to work on concept naming <img width=400 src="https://github.com/user-attachments/assets/fc719c93-467f-412b-8d47-68821bd8a5f5"> Release Notes: - N/A --------- Co-authored-by: Richard Feldman <oss@rtfeldman.com> Co-authored-by: Max Brunsfeld <maxbrunsfeld@gmail.com> Co-authored-by: Thomas Mickley-Doyle <tmickleydoyle@gmail.com>	2025-04-22 23:58:58 -03:00

Piotr Osiewicz

05fc0c432c

Fix a bunch of other low-hanging style lints (#36498 )

- **Fix a bunch of low hanging style lints like unnecessary-return**
- **Fix single worktree violation**
- **And the rest**

Release Notes:

- N/A

2025-08-19 21:26:17 +02:00

Oleksiy Syvokon

6420df3975

eval: Count execution errors as failures (#30712 )

- Evals returning an error (e.g., LLM API format mismatch) were silently
skipped in the aggregated results. Now we count them as a failure (0%
success score).

- Setting the `VERBOSE` environment variable to something non-empty
disables string truncation

Release Notes:

- N/A

2025-05-14 20:44:19 +03:00

Agus Zubiaga

ce1a674eba

eval: Fine-grained assertions (#29246 )

- Support programmatic examples
([example](17feb260a0/crates/eval/src/examples/file_search.rs))
- Combine data-driven example declarations into a single `.toml` file
([example](17feb260a0/crates/eval/src/examples/find_and_replace_diff_card.toml))
- Run judge on individual assertions (previously called "criteria")
- Report judge and programmatic assertions in one combined table

Note: We still need to work on concept naming 

<img width=400
src="https://github.com/user-attachments/assets/fc719c93-467f-412b-8d47-68821bd8a5f5">

Release Notes:

- N/A

---------

Co-authored-by: Richard Feldman <oss@rtfeldman.com>
Co-authored-by: Max Brunsfeld <maxbrunsfeld@gmail.com>
Co-authored-by: Thomas Mickley-Doyle <tmickleydoyle@gmail.com>

2025-04-22 23:58:58 -03:00

3 commits