In response to an ongoing BuildJet outage, consider migrating CI to
GitHub hosted runners.
Also includes revert of (causing flaky tests):
- https://github.com/zed-industries/zed/pull/35741
Downsides:
- Cost (2x)
- Force migration to Ubuntu 22.04 from 20.04 will bump our glibc minimum
from 2.31 to 2.35. Which would break RHEL 9.x (glibc 2.34), Ubuntu 20.04
(EOL) and derivatives.
Release Notes:
- N/A
Added [rhysd/actionlint](https://github.com/rhysd/actionlint/) a static
checker for GitHub Actions workflow files.
Install locally with `brew install actionlint` the run with
`actionlint`.
Inspired by: https://github.com/zed-industries/zed/pull/34704 which
yielded this observation:
> In github actions:
> 1. strings are truthy
> 2. `${{ }}` will become a string if it doesn't wrap the whole value.
>
> So `if: false && true` becomes `false`
> and `if: ${{ false && true }}` becomes `false`
> but `if: false && ${{ true }}` becomes `"false && true"` which
evaluates true
> The reason you sometimes need `${{ }}` is because YAML doesn't like
`!`
> so `if: !false` is invalid yaml
> and `if: ${{ !false }}` works just fine.
Changes:
- Add `actionlint` job
- Refactor `job_spec` job to be more readable
- Fix all `actionlint` and `shellcheck` errors in Actions workflows (62
in all)
- Add `self-mini-macos` and `self-32vcpu-windows-2022` labels to
self-hosted runners. Not strictly related, but useful if you need to
take a runner out of the rotation (since `macOS`, `self-hosted`, and
`ARM64` are auto-set and cannot be added/removed).
- Change ci.yml macos_relase to target `self-mini-macos` instead of
`bundle` which was previously deprecated.
This would've caught the error fixed in
https://github.com/zed-industries/zed/pull/34704. Here's what that [job
failure](https://github.com/zed-industries/zed/actions/runs/16376993944/job/46279281842?pr=34729)
would've looked like.
Release Notes:
- N/A
This is based on having observed that there is a lot of variation
between runs on `n=1` and `n=3`.
* With `n=8` two runs on the same branch give answers that seem close
enough to be reasonably consistent.
* With higher concurrency, trying to run this many repetitions seems to
lead language servers to time out a lot, causing evals to fail.
Release Notes:
- N/A
This PR adds a no-op job for the "Run Agent Eval" workflow.
This aims to avoid marking the check as failed on a PR that does not
include the `run-eval` label.
Release Notes:
- N/A
### Problem
We want to start continuously tracking our progress on agent evals over
time. As part of this, we'd like the *score* to have a clear,
interpretable meaning. Right now, it's a number from 0 to 5, but it's
not clear what any particular number works. In addition, scores vary
widely from run to run, because the agent's output is deterministic. We
try to stabilize the score using a panel of judges, but the behavior of
the agent itself varies much more widely than the judges' scores for a
given run.
### Solution
* **explicit meanings of scores** - In this PR, we're prescribing the
diff and thread criteria files so that they *must* be unordered lists of
assertions. For both the thread and the diff, rather than providing an
abstract score, the judge's task is simply to count how many of these
assertions are satisfied. A percentage score can be derived from this
number, divided by the total number of assertions.
* **repetitions** - Rather than running each example once, and judging
it N times, we'll **run** the example N times. Right now, I'm just
judging the output once per run, because I believe that with these more
clear scoring criteria, the main source of non-determinism will be the
*agent's* behavior, not the judge's
### Questions
* **accounting for diagnostic errors** - Previously, the judge was asked
to incorporate diagnostics into their abstract scores. Now that the
"score" is determined directly from the criteria, the diagnostic will
not be captured in the score. How should the diagnostics be accounted
for in the eval? One thought is - let's simply count and report the
number of errors remaining after the agent finishes, as a separate field
of the run (along with diff score and thread score). We could consider
normalizing it using the total lines of added code (like errors per 100
lines of code added) in order to give it some semblance of stability
between examples.
* **repetitions** - How many repetitions should we run on CI? Each
repetition takes significant time, but I think running more than one
repetition will make the scores significantly less volatile.
### Todo
* [x] Fix `--concurrency` implementation so that only N tasks are
spawned
* [x] Support `--repetitions` efficiently (re-using the same worktree)
* [x] Restructure judge prompts to count passing criteria, not compute
abstract score
* [x] Report total number of diagnostics in some way
* [x] Format output nicely
Release Notes:
- N/A *or* Added/Fixed/Improved ...
---------
Co-authored-by: Antonio Scandurra <me@as-cii.com>
The old one wasn't linking, and
https://github.com/zed-industries/zed/pull/29081 has a bunch of merge
conflicts. Wanted to start simple/small.
## Todo
* [x] Remove low-signal examples
* [x] Make the eval run on a cron, on main, and on any PR with the
`run-eval` label
* [x] Noise in logs about failure to write settings
```
[2025-04-21T20:45:04Z ERROR settings] Failed to write settings to file
"/home/runner/.config/zed/settings.json"
Caused by:
No such file or directory (os error 2) at path
"/home/runner/.config/zed/.tmpLewFEs"
```
* [x] `Agentic loop stalled`
(https://github.com/zed-industries/zed/actions/runs/14581044243/job/40897622894)
* [x] Make sure that events are recorded in snowflake
* [ ] Change judge criteria to be more explicit about meanings of scores
Release Notes:
- N/A
---------
Co-authored-by: Antonio Scandurra <me@as-cii.com>
Co-authored-by: Agus Zubiaga <hi@aguz.me>
Co-authored-by: Max Brunsfeld <maxbrunsfeld@gmail.com>
Co-authored-by: Thomas Mickley-Doyle <tmickleydoyle@gmail.com>
