Yehowshua/ZIm
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Revert tokenizer for custom OpenAI models (#17660)

Browse source 
Fix for custom openai models tokenizer settings.
This commit is contained in:
Peter Tripp 2024-09-10 15:38:27 -04:00 • committed by GitHub
parent a23e381096
commit bd1ff476b9
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
1 changed files with 5 additions and 1 deletions
Download patch file Download diff file Expand all files Collapse all files

6
crates/language_model/src/provider/open_ai.rs
View file

@ -370,7 +370,11 @@ pub fn count_open_ai_tokens(
})
.collect::<Vec<_>>();
tiktoken_rs::num_tokens_from_messages(model.id(), &messages)
if let open_ai::Model::Custom { .. } = model {
tiktoken_rs::num_tokens_from_messages("gpt-4", &messages)
} else {
tiktoken_rs::num_tokens_from_messages(model.id(), &messages)
}
})
.boxed()
}