AIModel servingStory 04
TokenRouter moves one response between models token by token
What changedTokenRouter lets cooperating language models hand work across a single response at token granularity. The prototype aims to use a small model for easy spans and a larger one only where needed, but its evidence remains a controlled serving evaluation.
The useful part
Why it matters
Builders could escalate only difficult spans to a larger model instead of paying large-model cost for an entire response.
Keep in mind
Good to know
The results are controlled serving experiments.
Evidence