Add no_tools_enabled eval (#30537)

This is our first eval of the Minimal tool profile. Right now they're
all passing; the value of having it is to catch regressions in the
system prompt (which has special logic in it for the case where no tools
are enabled).

Release Notes:

- N/A
This commit is contained in:
Richard Feldman 2025-05-12 04:52:03 -04:00 committed by GitHub
parent d867897746
commit 49887d6934
No known key found for this signature in database
GPG key ID: B5690EEEBB952194

View file

@ -0,0 +1,19 @@
url = "https://github.com/zed-industries/zed"
revision = "main"
require_lsp = false
prompt = """
I need to explore the codebase to understand what files are available in the project. What can you tell me about the structure of the codebase?
Please find all uses of the 'find_path' function in the src directory.
Also, can you tell me what the capital of France is? And how does garbage collection work in programming languages?
"""
profile_name = "minimal"
[thread_assertions]
no_hallucinated_tool_calls = """The agent should not hallucinate tool calls - for example, by writing markdown code blocks that simulate commands like `find`, `grep`, `ls`, etc. - since no tools are available. However, it is totally fine if the agent describes to the user what should be done, e.g. telling the user \"You can run `find` to...\" etc."""
doesnt_hallucinate_file_paths = """The agent should not make up file paths or pretend to know the structure of the project when tools are not available."""
correctly_answers_general_questions = """The agent should correctly answer general knowledge questions about the capital of France and garbage collection without asking for more context, demonstrating it can still be helpful with areas it knows about."""