Eager Semantic Indexing Queue (#2886)

Optimization to the Semantic Indexing Engine.

We've transitioned from a framework in which the entire project tree is
walked at each index command, to an eager queuing method, in which an
initial queue of outstanding indexing work is initialized upon workspace
creation, and then subscriptions are leveraged for file change events to
continually keep an updated view on outstanding work.

This optimization contributes towards quicker user feedback, when
initializing or using Semantic Search functionality. It also opens the
doors towards better transparency across the system on outstanding
indexing work.

Release Notes:

- Refactored index operation queue to an eager queuing framework.
- Moved semantic search initialization to workspace creation.
- Adjusted rate limiting strategy on api delays to reduce time spent
waiting for rate limits.

crates/search/src/project_search.rs

@@ -640,6 +640,7 @@ impl ProjectSearchView {
             self.search_options = SearchOptions::none();
 
             let project = self.model.read(cx).project.clone();
+
             let index_task = semantic_index.update(cx, |semantic_index, cx| {
                 semantic_index.index_project(project, cx)
             });
@@ -1635,6 +1636,12 @@ impl ToolbarItemView for ProjectSearchBar {
         self.subscription = None;
         self.active_project_search = None;
         if let Some(search) = active_pane_item.and_then(|i| i.downcast::<ProjectSearchView>()) {
+            search.update(cx, |search, cx| {
+                if search.current_mode == SearchMode::Semantic {
+                    search.index_project(cx);
+                }
+            });
+
             self.subscription = Some(cx.observe(&search, |_, _, cx| cx.notify()));
             self.active_project_search = Some(search);
             ToolbarItemLocation::PrimaryLeft {