ZIm/crates/language_models
Roshan Padaki af461f8165
assistant: Use GPT 4 tokenizer for o3-mini (#24068)
Sorry to dump an unsolicited PR for a hot feature! I'm sure someone else
was taking a look at this.

I noticed that token counting was disabled and I was getting error logs
of the form `[2025-01-31T22:59:01-05:00 ERROR assistant_context_editor]
No tokenizer found for model o3-mini` when using the new model. To fix
the issue, this PR registers the `gpt-4` tokenizer for this model.

Release Notes:

- openai: Fixed Assistant token counts for `o3-mini` models
2025-02-01 12:08:44 -05:00
..
src assistant: Use GPT 4 tokenizer for o3-mini (#24068) 2025-02-01 12:08:44 -05:00
Cargo.toml Add DeepSeek support (#23551) 2025-01-27 13:40:59 -05:00
LICENSE-GPL Add language_models crate to house language model providers (#20945) 2024-11-20 18:49:34 -05:00