3a3d3c05e8
tiktoken_rs is a bit behind (and even upstream tiktoken doesn't have all of these models) We were incorrectly using the cl100k tokenizer for some models that actually use the o200k tokenizers. So that is updated. I also made the match arms specific so that we do a better job of catching whether or not tiktoken-rs accurately supports new models we add in. I will also do a PR upstream to see if we can move some of this logic back out if tiktoken better supports the newer models. Release Notes: - Improved tokenizer support for openai models. |
||
|---|---|---|
| .. | ||
| src | ||
| Cargo.toml | ||
| LICENSE-GPL | ||