Yehowshua/ZIm
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
11839 commits 1103 branches 1111 tags 398 MiB
Commit graph

77 commits

Author SHA1 Message Date
Kyle Caverly
fd373517c7 update semantic search to show no results if search query is blank (#2942) 
Update semantic search to show no search results if search query is
blank
 2023-09-07 14:58:49 -04:00
Antonio Scandurra
eb8cf7c36e Start indexing right away when project was already indexed before (#2941) 
Release notes:
- Improved semantic search indexing to start in the background if the
project was already indexed before.
 2023-09-07 19:47:40 +02:00
Antonio Scandurra
fedec2d21c Rework how we track projects and worktrees in semantic index (#2938) 
This pull request introduces several improvements to the semantic search
experience. We're still missing collaboration and searching modified
buffers, which we'll tackle after we take a detour into reducing the
number of tokens used to generate embeddings.

Release Notes:

- Fixed a bug that could prevent semantic search from working when
deploying right after opening a project.
- Fixed a panic that could sometimes occur when using semantic search
while simultaneously changing a file.
- Fixed a bug that prevented semantic search from including new
worktrees when adding them to a project.
 2023-09-07 19:47:12 +02:00
Kyle Caverly
f685265cad Token count fix (#2935) 
Fix token count for OpenAIEmbeddings

Release Notes (Preview Only)

- update token count calculation for truncated OpenAIEmbeddings
- increased request timeout for OpenAI
 2023-09-07 19:46:29 +02:00
KCaverly
8dbc0fe033 update pragma settings for improved database performance 2023-09-01 17:07:20 -04:00
KCaverly
54235f4fb1 updated embeddings background delay to 5 minutes 
Co-authored-by: Max <max@zed.dev>
 2023-09-01 13:04:09 -04:00
KCaverly
e86964eb5d optimize insert file in vector database 
Co-authored-by: Max <max@zed.dev>
 2023-09-01 13:01:37 -04:00
KCaverly
524533cfb2 flush embeddings queue when no files are parsed for 250 milliseconds 
Co-authored-by: Antonio <antonio@zed.dev>
 2023-09-01 11:24:08 -04:00
KCaverly
c4db914f0a move embeddings queue to use single hashmap for all changed paths 
Co-authored-by: Antonio <me@as-cii.com>
 2023-09-01 08:59:25 -04:00
KCaverly
afa59abbcd WIP: work towards wiring up a embeddings_for_digest hashmap that is stored for all indexed files 2023-08-31 16:42:39 -04:00
KCaverly
50cfb067e7 fill embeddings with database values and skip during embeddings queue 2023-08-31 13:19:17 -04:00
Antonio Scandurra
220533ff1a WIP 2023-08-31 18:00:57 +02:00
Antonio Scandurra
2503d54d19 Rename Sha1 to DocumentDigest 
Co-Authored-By: Kyle Caverly <kyle@zed.dev>
 2023-08-31 18:00:36 +02:00
Antonio Scandurra
3001a46f69 Reify Embedding/Sha1 structs that can be (de)serialized from SQL 
Co-Authored-By: Kyle Caverly <kyle@zed.dev>
 2023-08-31 17:55:43 +02:00
Antonio Scandurra
c763e728d1 Write to and read from the database in a transactional way 
Co-Authored-By: Kyle Caverly <kyle@zed.dev>
 2023-08-31 16:59:54 +02:00
Antonio Scandurra
35440be98e Abstract away how database transactions are executed 
Co-Authored-By: Kyle Caverly <kyle@zed.dev>
 2023-08-31 16:54:11 +02:00
KCaverly
7d4d6c871b fix bug for truncation ensuring no valid inputs are sent to openai 2023-08-30 17:42:16 -04:00
KCaverly
5abad58b0d moved semantic index to use embeddings queue to batch and managed for atomic database writes 
Co-authored-by: Max <max@zed.dev>
 2023-08-30 16:58:45 -04:00
KCaverly
76ce52df4e move queuing to embedding_queue functionality and update embedding provider to include trait items for max tokens per batch" 
Co-authored-by: Max <max@zed.dev>
 2023-08-30 16:01:28 -04:00
KCaverly
9781047156 move truncation to parsing step leveraging the EmbeddingProvider trait 2023-08-30 12:13:26 -04:00
KCaverly
76caea80f7 add should_truncate to embedding providers 2023-08-30 11:58:45 -04:00
KCaverly
e377ada1a9 added token count to documents during parsing 2023-08-30 11:05:46 -04:00
KCaverly
a7e6a65deb reindex files in the background after they have not been edited for 10 minutes 
Co-authored-by: Max <max@zed.dev>
 2023-08-29 17:14:44 -04:00
KCaverly
4f8b95cf0d add proper handling for open ai rate limit delays 2023-08-29 15:44:51 -04:00
Max Brunsfeld
404f76739c Format let-else statements 2023-08-25 10:11:32 -07:00
KCaverly
ee97bc54cf cleaned up warnings 2023-08-25 10:38:01 +02:00
KCaverly
131950f670 add handling for Added file events to semantic index 2023-08-24 18:40:08 +02:00
KCaverly
a892a51ec3 update initialize project call to accomodate for test scenarios 
Co-authored-by: Piotr <piotr@zed.dev>
 2023-08-24 13:46:43 +02:00
KCaverly
a1519e4c38 move semantic search project intialization to a subscribe event for workspace created 
Co-authored-by: Piotr <piotr@zed.dev>
 2023-08-24 13:14:19 +02:00
KCaverly
e8e7b294d8 add delete files operation for remaining files in database not included in current worktree 
Co-authored-by: Piotr <piotr@zed.dev>
 2023-08-24 12:49:20 +02:00
KCaverly
afe0e74868 remove worktree_file_mtimes in state as it is no longer used 
Co-authored-by: Piotr <piotr@zed.dev>
 2023-08-24 12:42:41 +02:00
KCaverly
aa07872a24 accomodate for duplicate entries in indexing queue 
Co-authored-by: Piotr <piotr@zed.dev>
 2023-08-24 12:36:43 +02:00
KCaverly
3f9f742530 update rate limiting embeddings strategy to delay less 2023-08-24 11:45:52 +02:00
KCaverly
e42b9e910e fix async calls on project updated entries to ensure that all files are updating appropriately 2023-08-23 22:28:30 +02:00
KCaverly
09fd99b1e3 moved semantic_index project intialization to queue and channel method 2023-08-23 15:09:15 +02:00
KCaverly
328b7e523c reorganized to stop the race 2023-08-22 15:01:21 +02:00
KCaverly
aabdfa210f working on initialization + index breakup 2023-08-22 14:45:27 +02:00
KCaverly
ced2b2aec3 reworked ProjectState to include additional context 2023-08-22 11:58:48 +02:00
KCaverly
3d89cd10a4 added sha1 encoding for each document 2023-08-21 16:35:57 +02:00
Piotr Osiewicz
c68b518aec chore: fix compiler warning 
Co-authored-by: Kyle <kyle@zed.dev>
 2023-08-21 13:31:45 +02:00
Piotr Osiewicz
61041b0cd1 Do not attempt to reindex a file if previous attempts have failed. 
Add doc comment to JobHandle

Co-authored-by: Kyle <kyle@zed.dev>
 2023-08-21 13:23:11 +02:00
Piotr Osiewicz
1a88444f2f Increment job counter on JobClient::new 
Co-authored-by: Kyle <kyle@zed.dev>
 2023-08-21 13:00:56 +02:00
KCaverly
def215af9f update job handle to ensure file count is consistent 
Co-authored-by: Piotr <piotr@zed.dev>
 2023-08-21 12:47:43 +02:00
KCaverly
1cae4758cc manage for edge case in which file documents are larger than the allowable limit 2023-08-21 11:29:45 +02:00
Piotr Osiewicz
e2b8e6ce63 chore: fix compiler warnings 2023-08-09 01:16:24 +02:00
Piotr Osiewicz
31fb503418 Rewrite permission queries (it no longer blocks) 
Co-authored-by: Kyle <kyle@zed.dev>
Co-authored-by: Max <max@zed.dev>
 2023-08-03 20:26:56 +02:00
KCaverly
8dd330a160 add semantic search prompt for the indexing permission on first search 
Co-authored-by: Piotr <piotr@zed.dev>
 2023-08-03 12:52:20 -04:00
KCaverly
300c693d55 catchup with main 2023-08-01 10:40:38 -04:00
KCaverly
e221f23018 add support for markdown files to semantic search 2023-08-01 10:30:34 -04:00
KCaverly
9a50b43eaa add templating languages html, erb, heex, svelte as entire parseable file types 2023-07-31 21:03:02 -04:00