language_models: Update tiktoken-rs to support newer models (#30951)
I was able to get this fix in upstream, so now we can have simpler code paths for our model selection. I also added a test to catch if this would cause a bug again in the future. Release Notes: - N/A
This commit is contained in:
parent
2b6dab9197
commit
57424e4743
3 changed files with 54 additions and 14 deletions
|
@ -555,7 +555,7 @@ sysinfo = "0.31.0"
|
|||
take-until = "0.2.0"
|
||||
tempfile = "3.20.0"
|
||||
thiserror = "2.0.12"
|
||||
tiktoken-rs = "0.6.0"
|
||||
tiktoken-rs = "0.7.0"
|
||||
time = { version = "0.3", features = [
|
||||
"macros",
|
||||
"parsing",
|
||||
|
|
Loading…
Add table
Add a link
Reference in a new issue