Start to send data to Snowflake too (#20698)

This PR adds support for sending telemetry events to AWS Kinesis.

In our AWS account we now have three new things:
* The [Kinesis data
stream](https://us-east-1.console.aws.amazon.com/kinesis/home?region=us-east-1#/streams/details/zed-telemetry/monitoring)
that we will actually write to.
* A [Firehose for
Axiom](https://us-east-1.console.aws.amazon.com/firehose/home?region=us-east-1#/details/telemetry-to-axiom/monitoring)
that sends events from that stream to Axiom for ad-hoc queries over
recent data.
* A [Firehose for
Snowflake](https://us-east-1.console.aws.amazon.com/firehose/home?region=us-east-1#/details/telemetry-to-snowflake/monitoring)
that sends events from that stream to Snowflake for long-term retention.
This Firehose also backs up data into an S3 bucket in case we want to
change how the system works in the future.

In a follow-up PR, we'll add support for ad-hoc telemetry events; and
slowly move away from the current Clickhouse defined schemas; though we
won't move off click house until we have what we need in Snowflake.

Co-Authored-By: Nathan <nathan@zed.dev>

Release Notes:

- N/A
This commit is contained in:
Conrad Irwin 2024-11-15 12:58:00 -07:00 committed by GitHub
parent f449e8d3d3
commit 6ff69faf37
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
7 changed files with 305 additions and 28 deletions

View file

@ -174,6 +174,31 @@ spec:
secretKeyRef:
name: blob-store
key: bucket
- name: KINESIS_ACCESS_KEY
valueFrom:
secretKeyRef:
name: kinesis
key: access_key
- name: KINESIS_SECRET_KEY
valueFrom:
secretKeyRef:
name: kinesis
key: secret_key
- name: KINESIS_STREAM
valueFrom:
secretKeyRef:
name: kinesis
key: stream
- name: KINESIS_REGION
valueFrom:
secretKeyRef:
name: kinesis
key: region
- name: BLOB_STORE_BUCKET
valueFrom:
secretKeyRef:
name: blob-store
key: bucket
- name: CLICKHOUSE_URL
valueFrom:
secretKeyRef: