Yehowshua/ZIm
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

danger: Check for changes in prompt files (#31744)

Browse source 
This PR adds a Danger check to remind engineers that any changes to our
various prompts need to be verified against the LLM Worker.

When changes to the prompt files are detected, we will fail the PR with
a message:

<img width="929" alt="Screenshot 2025-05-30 at 8 40 58 AM"
src="https://github.com/user-attachments/assets/79afab4e-e799-45f1-a90e-0fd7c9a73706"
/>

Once the corresponding changes have been made (or no changes to the LLM
Worker have been determined to be necessary), including the indicated
attestation message will convert the errors into informational messages:

<img width="926" alt="Screenshot 2025-05-30 at 8 41 52 AM"
src="https://github.com/user-attachments/assets/ff51c17a-7a76-46a7-b468-a7d864d480c3"
/>

Release Notes:

- N/A
This commit is contained in:
Marshall Bowers 2025-05-30 09:46:41 -04:00 • committed by GitHub
parent 6bb4b5fa64
commit 310ea43048
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
5 changed files with 52 additions and 13 deletions
Download patch file Download diff file Expand all files Collapse all files

38
script/danger/dangerfile.ts
View file

@ -1,4 +1,4 @@
import { danger, message, warn } from "danger";
import { danger, message, warn, fail } from "danger";
const { prHygiene } = require("danger-plugin-pr-hygiene");
prHygiene({
@ -57,3 +57,39 @@ if (includesIssueUrl) {
].join("\n"),
);
}
const PROMPT_PATHS = [
"assets/prompts/content_prompt.hbs",
"assets/prompts/terminal_assistant_prompt.hbs",
"crates/agent/src/prompts/stale_files_prompt_header.txt",
"crates/agent/src/prompts/summarize_thread_detailed_prompt.txt",
"crates/agent/src/prompts/summarize_thread_prompt.txt",
"crates/assistant_tools/src/templates/create_file_prompt.hbs",
"crates/assistant_tools/src/templates/edit_file_prompt.hbs",
"crates/git_ui/src/commit_message_prompt.txt",
];
const PROMPT_CHANGE_ATTESTATION = "I have ensured the LLM Worker works with these prompt changes.";
const modifiedPrompts = danger.git.modified_files.filter((file) =>
PROMPT_PATHS.some((promptPath) => file.includes(promptPath)),
);
for (const promptPath of modifiedPrompts) {
if (body.includes(PROMPT_CHANGE_ATTESTATION)) {
message(
[
`This PR contains changes to "${promptPath}".`,
"The author has attested the LLM Worker works with the changes to this prompt.",
].join("\n"),
);
} else {
fail(
[
`Modifying the "${promptPath}" prompt may require corresponding changes in the LLM Worker.`,
"If you are ensure what this entails, talk to @maxdeviant or another AI team member.",
`Once you have made the changes—or determined that none are necessary—add "${PROMPT_CHANGE_ATTESTATION}" to the PR description.`,
].join("\n"),
);
}
}