Commit graph

70 commits

Author SHA1 Message Date
Nathan Sobo
458ffaa134
Add new action to run agent eval (#29158)
The old one wasn't linking, and
https://github.com/zed-industries/zed/pull/29081 has a bunch of merge
conflicts. Wanted to start simple/small.

## Todo

* [x] Remove low-signal examples
* [x] Make the eval run on a cron, on main, and on any PR with the
`run-eval` label
* [x] Noise in logs about failure to write settings
    ```
[2025-04-21T20:45:04Z ERROR settings] Failed to write settings to file
"/home/runner/.config/zed/settings.json"
    
       Caused by:
No such file or directory (os error 2) at path
"/home/runner/.config/zed/.tmpLewFEs"
    ```
* [x] `Agentic loop stalled`
(https://github.com/zed-industries/zed/actions/runs/14581044243/job/40897622894)
* [x] Make sure that events are recorded in snowflake
* [ ] Change judge criteria to be more explicit about meanings of scores

Release Notes:

- N/A

---------

Co-authored-by: Antonio Scandurra <me@as-cii.com>
Co-authored-by: Agus Zubiaga <hi@aguz.me>
Co-authored-by: Max Brunsfeld <maxbrunsfeld@gmail.com>
Co-authored-by: Thomas Mickley-Doyle <tmickleydoyle@gmail.com>
2025-04-21 21:30:21 -07:00
Max Brunsfeld
0dc3dffe38
Use insert_id as partition key for crash events (#28293)
Release Notes:

- N/A
2025-04-07 17:24:31 -07:00
Max Brunsfeld
d385a60ed1 Fix phrasing of crash/panic event names 2025-04-07 16:31:29 -07:00
Max Brunsfeld
6220b86f94
Write panics and crashes to snowflake (#28284)
This will let us create a better crashes dashboard, using Hex.

Release Notes:

- N/A
2025-04-07 15:50:16 -07:00
Piotr Osiewicz
dc64ec9cc8
chore: Bump Rust edition to 2024 (#27800)
Follow-up to https://github.com/zed-industries/zed/pull/27791

Release Notes:

- N/A
2025-03-31 20:55:27 +02:00
Joseph T. Lyons
ed510b5e93
Remove unused AssistantThreadFeedback event (#27021)
It looks like:

- https://github.com/zed-industries/zed/pull/26780

accidentally added a new event type, `AssistantThreadFeedback`, using
the old event system, that it didn't end up actually using, as the code
actually relies on using the newer (preferred) `telemetry::event!()`.

Release Notes:

- N/A
2025-03-18 20:39:54 +00:00
Antonio Scandurra
f68a475eca
Introduce rating for assistant threads (#26780)
Release Notes:

- N/A

---------

Co-authored-by: Richard Feldman <oss@rtfeldman.com>
Co-authored-by: Agus Zubiaga <hi@aguz.me>
2025-03-14 14:41:50 +00:00
Richard Feldman
667396c44b
Use the term "edit prediction" over "inline completion" (#24211)
Note that this does *not* involve any breaking code changes.

cc @0xtimsb - I didn't change any settings or anything here. That can
happen separately!

Release Notes:

- N/A
2025-02-04 10:33:01 -08:00
Joseph T. Lyons
a8741dc310
Migrate more events to telemetry::event! macro (#24102)
I believe this takes care of the remaining events running through the
old flow that requires transformation at the collab server level. I
think all events are now going through `telemetry::event!()`.

For anyone curious where the new telemetry names are coming from, you
can check the `for_snowflake` function within
`crates/collab/src/api/events.rs`, to see how collab is currently
transforming the events going through the old flow.

Release Notes:

- N/A
2025-02-03 16:38:45 +00:00
Cole Miller
a2022d7da3
Improve Linux panic reporting (#22202)
- [x] Upload separate debug symbols for Linux binaries to DigitalOcean
- [x] Send raw offsets with panic report JSON on Linux
- [x] Update `symbolicate` script to handle Linux crashes
- [x] Demangle backtraces 🎉 
- [x] Check that it works
- [x] Improve deduplication (?)
 
Release Notes:

- N/A

---------

Co-authored-by: Conrad <conrad@zed.dev>
2024-12-22 08:20:17 +00:00
Conrad Irwin
7425d242bc
Add telemetry::event! (#22146)
CC @JosephTLyons

Release Notes:

- N/A
2024-12-17 11:39:18 -07:00
Joseph T. Lyons
f80eb73213
Update event type to conform to standard (#21827)
Release Notes:

- N/A
2024-12-10 16:14:31 -05:00
Conrad Irwin
03efd0d1d9
Stop sending data to Clickhouse (#21763)
Release Notes:

- N/A
2024-12-10 08:47:29 -07:00
Marshall Bowers
8a85d6ef96
collab: Make metrics_id required in LlmTokenClaims (#21771)
This PR makes the `metrics_id` field on the `LlmTokenClaims` required,
as we always have one in practice.

Release Notes:

- N/A
2024-12-09 17:58:14 -05:00
Antonio Scandurra
77b8296fbb
Introduce staff-only inline completion provider (#21739)
Release Notes:

- N/A

---------

Co-authored-by: Thorsten Ball <mrnugget@gmail.com>
Co-authored-by: Bennet <bennet@zed.dev>
Co-authored-by: Thorsten <thorsten@zed.dev>
2024-12-09 14:26:36 +01:00
Joseph T. Lyons
3dcb94c204
Correct more app event inconsistencies (#21129)
- Unify welcome page event type string structure
- Differentiate between metric telemetry event and diagnostic telemetry
event

Release Notes:

- N/A
2024-11-24 00:34:02 -05:00
Joseph T. Lyons
0395d1b037
Clean up app event transformations (#21115)
This needs scrutinized. Detailed breakdown of what events I kept and
threw out here:

https://zed.dev/channel/app-events-17178/notes

I also removed a few fake events and tossed out json properties that
were being inserted for things we don't have logic to track. See PR
review comments below.

I think the only bad data we have are that we were identifying all node,
pnpm, and yarn projects as 'node' in the `project_type` property, so a
few days of lost data there.

Release Notes:

- N/A
2024-11-23 23:11:45 -05:00
Joseph T. Lyons
43f0ea759b
Remove non-existent call event types (#21093)
These are not real (from Clickhouse):

```
    ┌─operation────────────┬──────c─┐
 1. │ join channel         │ 136221 │
 2. │ open channel notes   │  95529 │
 3. │ hang up              │  66264 │
 4. │ disable microphone   │  34116 │
 5. │ enable microphone    │  25090 │
 6. │ enable screen share  │  20751 │
 7. │ invite               │  15827 │
 8. │ share project        │  14580 │
 9. │ accept incoming      │  13708 │
10. │ disable screen share │  10440 │
11. │ unshare project      │   9556 │
12. │ decline incoming     │    455 │
13. │ enable camera        │      6 │
14. │ disable camera       │      4 │
    └──────────────────────┴────────┘
```

Release Notes:

- N/A
2024-11-22 23:49:53 -05:00
Conrad Irwin
984bb192ba
Send llm events to snowflake too (#21091)
Closes #ISSUE

Release Notes:

- N/A
2024-11-22 20:40:39 -07:00
Joseph T. Lyons
933c11a9b2
Remove dead snowflake code (#21041)
Co-authored-by: Nathan Sobo <1789+nathansobo@users.noreply.github.com>

Release Notes:

- N/A
2024-11-21 23:06:03 -05:00
Conrad Irwin
ebca6a8f3d
Send os_version and country to amplitude (#20936)
Release Notes:

- N/A
2024-11-20 15:34:24 -07:00
Conrad Irwin
705a06c3dd
Send Country/OS/Version amplitude style (#20884)
Release Notes:

- N/A
2024-11-19 16:38:14 -07:00
Conrad Irwin
ea5131ce0a
Country Code To Snowflake (#20875)
Release Notes:

- N/A

---------

Co-authored-by: Nathan Sobo <nathan@zed.dev>
2024-11-19 12:52:00 -07:00
Conrad Irwin
496dae968b
Remove old CPU/Memory events (#20865)
Release Notes:

- Telemetry: stop reporting CPU/RAM on a timer
2024-11-19 12:25:16 -07:00
Nathan Sobo
ee207ab77e
Map "feature upsell" events to the new "Noun Verbed" format (#20787)
Release Notes:

- N/A
2024-11-17 07:38:30 -07:00
Nathan Sobo
f9990b42fa
Send events to Snowflake in the format they're expected by Amplitude (#20765)
This will allow us to use the events table directly in Amplitude, which
lets us use the newer event ingestion flow that detects changes to the
table. Otherwise we'll need a transformation.

I think Amplitude's API is probably a pretty good example to follow for
the raw event schema, even if we don't end up using their product. They
also recommend a "Noun Verbed" format for naming events, so I think we
should go with this. This will help us be consistent and encourage the
author of events to think more clearly about what event they're
reporting.

cc @ConradIrwin 

Release Notes:

- N/A
2024-11-16 09:58:19 -07:00
Conrad Irwin
ac5ecf5487
Don't log every value (#20744)
Release Notes:

- N/A
2024-11-15 14:36:58 -07:00
Conrad Irwin
6ff69faf37
Start to send data to Snowflake too (#20698)
This PR adds support for sending telemetry events to AWS Kinesis.

In our AWS account we now have three new things:
* The [Kinesis data
stream](https://us-east-1.console.aws.amazon.com/kinesis/home?region=us-east-1#/streams/details/zed-telemetry/monitoring)
that we will actually write to.
* A [Firehose for
Axiom](https://us-east-1.console.aws.amazon.com/firehose/home?region=us-east-1#/details/telemetry-to-axiom/monitoring)
that sends events from that stream to Axiom for ad-hoc queries over
recent data.
* A [Firehose for
Snowflake](https://us-east-1.console.aws.amazon.com/firehose/home?region=us-east-1#/details/telemetry-to-snowflake/monitoring)
that sends events from that stream to Snowflake for long-term retention.
This Firehose also backs up data into an S3 bucket in case we want to
change how the system works in the future.

In a follow-up PR, we'll add support for ad-hoc telemetry events; and
slowly move away from the current Clickhouse defined schemas; though we
won't move off click house until we have what we need in Snowflake.

Co-Authored-By: Nathan <nathan@zed.dev>

Release Notes:

- N/A
2024-11-15 12:58:00 -07:00
Joseph T. Lyons
1cbaca667f
Remove historical_event column in editor events (#18932)
We have a lot of data in Clickhouse. This column was used when migrating
the events dataset between analytics databases and has no purpose today.

Naive maths: 257,170,993 editor event rows * 1 byte per boolean =
257,170,993 bytes, or ~0.24 GB

I'll drop the column after deploying a new collab.

Going forward, I'd like to remove more data that we never touch, to try
to keep things more focused. We should discuss some TTL at some point.

Release Notes:

- N/A
2024-10-09 13:19:13 -04:00
Joseph T. Lyons
77bf2ad0f1
Add is_via_ssh field to edit events (#18867)
Release Notes:

- N/A
2024-10-08 13:13:40 -04:00
Joseph T. Lyons
d295c46433
Remove deprecated copilot event (#18862)
`CopilotEvent` was succeeded by `InlineCompletionEvent` 5 months ago.

Release Notes:

- N/A
2024-10-08 11:10:20 -04:00
Joseph T. Lyons
4c7a6f5e7f
Add is_via_ssh field to editor events (#18837)
Release Notes:

- N/A
2024-10-08 10:30:04 -04:00
Marshall Bowers
9565a90528
collab: Revert changes to Clickhouse event rows (#18654)
This PR reverts the changes to the Clickhouse event rows that were
included in https://github.com/zed-industries/zed/pull/18414.

The changes don't seem to be correct, as they make the row structs
differ from the underlying table schema.

Release Notes:

- N/A
2024-10-02 16:10:25 -04:00
Conrad Irwin
3a5deb5c6f
Replace isahc with async ureq (#18414)
REplace isahc with ureq everywhere gpui is used.

This should allow us to make http requests without libssl; and avoid a
long-tail of panics caused by ishac.

Release Notes:

- (potentially breaking change) updated our http client

---------

Co-authored-by: Mikayla <mikayla@zed.dev>
2024-10-02 12:30:48 -07:00
Marshall Bowers
167af4bc1d
Use const over static for string literals (#18635)
I noticed a few places where we were storing `&'static str`s in
`static`s instead of `const`s.

This PR updates them to use `const`.

Release Notes:

- N/A
2024-10-02 12:33:13 -04:00
Conrad Irwin
bbf5ed2ba1
Fix collab filtering panics better (#18344)
Release Notes:

- N/A
2024-09-25 09:42:07 -06:00
Conrad Irwin
b69c6ee7df
Exclude initialization failed errors from slack (#18232)
Release Notes:

- N/A
2024-09-24 10:17:43 -06:00
Joseph T. Lyons
601090511b
Remove system_id from all events but editor_events (#18154)
Release Notes:

- N/A
2024-09-20 13:25:06 -04:00
Conrad Irwin
edf2c19250
Hide GPU problems from Slack (#18087)
Release Notes:

- N/A

---------

Co-authored-by: Marshall <marshall@zed.dev>
Co-authored-by: Marshall Bowers <elliott.codes@gmail.com>
2024-09-19 17:28:30 -04:00
Joseph T. Lyons
ca4980df02
Add system_id (#18040)
This PR adds `system_id` to telemetry, which is contained within a new
`global` database (accessible by any release channel of Zed on a single
system). This will help us get a more accurate understanding of user
count, instead of relying on `installationd_id`, which is different per
release channel. This doesn't solve the problem of a user with multiple
machines, but it gets us closer.

Release Notes:

- N/A
2024-09-19 07:20:27 -04:00
Conrad Irwin
b43b800a54
More assistant events (#18032)
Release Notes:

- N/A
2024-09-18 18:07:39 -06:00
Piotr Osiewicz
095a08d9c8
chore: Another round of style lints fixes (#17519)
Closes #ISSUE

Release Notes:

- N/A
2024-09-07 02:36:55 +02:00
Piotr Osiewicz
e6c1c51b37
chore: Fix several style lints (#17488)
It's not comprehensive enough to start linting on `style` group, but
hey, it's a start.

Release Notes:

- N/A
2024-09-06 11:58:39 +02:00
Marshall Bowers
a9441879c3
collab: Fix writing LLM rate limit events to Clickhouse (#16367)
This PR fixes the writing of LLM rate limit events to Clickhouse.

We had a table in the table name: `llm_rate_limits` instead of
`llm_rate_limit_events`.

I also extracted a helper function to write to Clickhouse so we can use
it anywhere we need to.

Release Notes:

- N/A
2024-08-16 14:03:34 -04:00
Max Brunsfeld
8e9c2b1125
Introduce a separate backend service for LLM calls (#15831)
This PR introduces a separate backend service for making LLM calls.

It exposes an HTTP interface that can be called by Zed clients. To call
these endpoints, the client must provide a `Bearer` token. These tokens
are issued/refreshed by the collab service over RPC.

We're adding this in a backwards-compatible way. Right now the access
tokens can only be minted for Zed staff, and calling this separate LLM
service is behind the `llm-service` feature flag (which is not
automatically enabled for Zed staff).

Release Notes:

- N/A

---------

Co-authored-by: Marshall <marshall@zed.dev>
Co-authored-by: Marshall Bowers <elliott.codes@gmail.com>
2024-08-05 20:26:21 -04:00
Marshall Bowers
f11f3f2599
collab: Attach GeoIP country code to RPC sessions (#15814)
This PR updates collab to attach the user's GeoIP country code to their
RPC session.

We source the country code from the
[`CF-IPCountry`](https://developers.cloudflare.com/fundamentals/reference/http-request-headers/#cf-ipcountry)
header.

Release Notes:

- N/A
2024-08-05 11:11:49 -04:00
Joseph T. Lyons
856d9632e4
Add repl events (#15259)
Release Notes:

- N/A

---------

Co-authored-by: Kyle Kelley <rgbkrk@gmail.com>
2024-07-26 03:31:41 -04:00
Joseph T. Lyons
5b7e31c075
Add metrics_id to editor_events (#13525)
Release Notes:

- N/A
2024-06-25 16:47:55 -04:00
Conrad Irwin
4f364d6d09
Hide old linux panics (#13221)
Remove noise from the #panics channel by excluding any linux build
before
0.139.x. We filter on the os_version and os_name because evern older
versions
of linux set app_version = 1.0.0.

Release Notes:

- N/A
2024-06-18 13:48:26 -06:00
Joseph T. Lyons
f489c8b79f
Allow for non-official builds to report telemetry (#13175)
Release Notes:

- N/A
2024-06-17 20:24:18 -04:00