Start to send data to Snowflake too (#20698)
This PR adds support for sending telemetry events to AWS Kinesis. In our AWS account we now have three new things: * The [Kinesis data stream](https://us-east-1.console.aws.amazon.com/kinesis/home?region=us-east-1#/streams/details/zed-telemetry/monitoring) that we will actually write to. * A [Firehose for Axiom](https://us-east-1.console.aws.amazon.com/firehose/home?region=us-east-1#/details/telemetry-to-axiom/monitoring) that sends events from that stream to Axiom for ad-hoc queries over recent data. * A [Firehose for Snowflake](https://us-east-1.console.aws.amazon.com/firehose/home?region=us-east-1#/details/telemetry-to-snowflake/monitoring) that sends events from that stream to Snowflake for long-term retention. This Firehose also backs up data into an S3 bucket in case we want to change how the system works in the future. In a follow-up PR, we'll add support for ad-hoc telemetry events; and slowly move away from the current Clickhouse defined schemas; though we won't move off click house until we have what we need in Snowflake. Co-Authored-By: Nathan <nathan@zed.dev> Release Notes: - N/A
This commit is contained in:
parent
f449e8d3d3
commit
6ff69faf37
7 changed files with 305 additions and 28 deletions
|
@ -174,6 +174,31 @@ spec:
|
|||
secretKeyRef:
|
||||
name: blob-store
|
||||
key: bucket
|
||||
- name: KINESIS_ACCESS_KEY
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: kinesis
|
||||
key: access_key
|
||||
- name: KINESIS_SECRET_KEY
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: kinesis
|
||||
key: secret_key
|
||||
- name: KINESIS_STREAM
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: kinesis
|
||||
key: stream
|
||||
- name: KINESIS_REGION
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: kinesis
|
||||
key: region
|
||||
- name: BLOB_STORE_BUCKET
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: blob-store
|
||||
key: bucket
|
||||
- name: CLICKHOUSE_URL
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
|
|
Loading…
Add table
Add a link
Reference in a new issue