Token
AI
The basic unit of text processed by a language model, typically a word or subword. Language models work with tokens rather than individual characters, and token limits determine how much text a model can process.
AI
The basic unit of text processed by a language model, typically a word or subword. Language models work with tokens rather than individual characters, and token limits determine how much text a model can process.