Start to send data to Snowflake too (#20698)

This PR adds support for sending telemetry events to AWS Kinesis.

In our AWS account we now have three new things:
* The [Kinesis data
stream](https://us-east-1.console.aws.amazon.com/kinesis/home?region=us-east-1#/streams/details/zed-telemetry/monitoring)
that we will actually write to.
* A [Firehose for
Axiom](https://us-east-1.console.aws.amazon.com/firehose/home?region=us-east-1#/details/telemetry-to-axiom/monitoring)
that sends events from that stream to Axiom for ad-hoc queries over
recent data.
* A [Firehose for
Snowflake](https://us-east-1.console.aws.amazon.com/firehose/home?region=us-east-1#/details/telemetry-to-snowflake/monitoring)
that sends events from that stream to Snowflake for long-term retention.
This Firehose also backs up data into an S3 bucket in case we want to
change how the system works in the future.

In a follow-up PR, we'll add support for ad-hoc telemetry events; and
slowly move away from the current Clickhouse defined schemas; though we
won't move off click house until we have what we need in Snowflake.

Co-Authored-By: Nathan <nathan@zed.dev>

Release Notes:

- N/A
This commit is contained in:
Conrad Irwin 2024-11-15 12:58:00 -07:00 committed by GitHub
parent f449e8d3d3
commit 6ff69faf37
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
7 changed files with 305 additions and 28 deletions

View file

@ -4,7 +4,7 @@ use semantic_version::SemanticVersion;
use serde::{Deserialize, Serialize};
use std::{fmt::Display, sync::Arc, time::Duration};
#[derive(Serialize, Deserialize, Debug)]
#[derive(Serialize, Deserialize, Debug, Clone)]
pub struct EventRequestBody {
/// Identifier unique to each system Zed is installed on
pub system_id: Option<String>,
@ -32,7 +32,7 @@ impl EventRequestBody {
}
}
#[derive(Serialize, Deserialize, Debug)]
#[derive(Serialize, Deserialize, Debug, Clone)]
pub struct EventWrapper {
pub signed_in: bool,
/// Duration between this event's timestamp and the timestamp of the first event in the current batch