language_models: Update tiktoken-rs to support newer models (#30951)

I was able to get this fix in upstream, so now we can have simpler code
paths for our model selection.

I also added a test to catch if this would cause a bug again in the
future.

Release Notes:

- N/A
This commit is contained in:
Ben Brandt 2025-05-19 13:40:36 +02:00 committed by GitHub
parent 2b6dab9197
commit 57424e4743
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
3 changed files with 54 additions and 14 deletions

View file

@ -555,7 +555,7 @@ sysinfo = "0.31.0"
take-until = "0.2.0"
tempfile = "3.20.0"
thiserror = "2.0.12"
tiktoken-rs = "0.6.0"
tiktoken-rs = "0.7.0"
time = { version = "0.3", features = [
"macros",
"parsing",