agent: Migrate thread storage to SQLite with zstd compression (#31741)
Previously, LMDB was used for storing threads, but it consumed excessive disk space and was capped at 1GB. This change migrates thread storage to an SQLite database. Thread JSON objects are now compressed using zstd. I considered training a custom zstd dictionary and storing it in a separate table. However, the additional complexity outweighed the modest space savings (up to 20%). I ended up using the default dictionary stored with data. Threads can be exported relatively easily from outside the application: ``` $ sqlite3 threads.db "SELECT hex(data) FROM threads LIMIT 5;" | xxd -r -p | zstd -d | fx ``` Benchmarks: - Original heed database: 200MB - Sqlite uncompressed: 51MB - sqlite compressed (this PR): 4.0MB - sqlite compressed with a trained dictionary: 3.8MB Release Notes: - Migrated thread storage to SQLite with compression
This commit is contained in:
parent
9a9e96ed5a
commit
c874f1fa9d
3 changed files with 215 additions and 56 deletions
2
Cargo.lock
generated
2
Cargo.lock
generated
|
@ -114,6 +114,7 @@ dependencies = [
|
|||
"serde_json_lenient",
|
||||
"settings",
|
||||
"smol",
|
||||
"sqlez",
|
||||
"streaming_diff",
|
||||
"telemetry",
|
||||
"telemetry_events",
|
||||
|
@ -133,6 +134,7 @@ dependencies = [
|
|||
"workspace-hack",
|
||||
"zed_actions",
|
||||
"zed_llm_client",
|
||||
"zstd",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
|
|
Loading…
Add table
Add a link
Reference in a new issue