Implement Anthropic prompt caching (#16274)

Release Notes:

- Adds support for Prompt Caching in Anthropic. For models that support
it this can dramatically lower cost while improving performance.
This commit is contained in:
Roy Williams 2024-08-15 23:21:06 -04:00 committed by GitHub
parent 09b6e3f2a6
commit 46fb917e02
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
11 changed files with 338 additions and 70 deletions

View file

@ -784,6 +784,7 @@ impl PromptLibrary {
messages: vec![LanguageModelRequestMessage {
role: Role::System,
content: vec![body.to_string().into()],
cache: false,
}],
stop: Vec::new(),
temperature: 1.,