Project: Model Router

Most requests hitting your LLM endpoint are easy. Someone wants a string reformatted, a short question answered, a sentence rewritten. A small model handles those indistinguishably from a frontier model, at a fraction of the price. The problem is that you cannot tell which request is which until after you have answered it.

Unless you ask first. This is intent routing pointed at your own cost structure: a Choice over what the request is, a Score over how hard it is, and a dispatch table that picks the model.

from typesafe_sdk import Choice, Score, TypeSafeClient

client = TypeSafeClient(model="jev-1.13")

ROUTER = {
    "task": Choice(
        instructions="What kind of work does this request ask for",
        criteria={
            "lookup": "Retrieve or restate a specific fact from the provided context",
            "transform": "Reformat, translate, summarize, or rewrite supplied text",
            "generate": "Produce substantial new content such as an essay or a draft",
            "reason": "Work through a problem with multiple dependent steps",
            "code": "Write, review, or debug source code",
        },
    ),
    "complexity": Score(
        instructions="How much capability does answering this well require",
        criteria=[
            "Trivial: a small model handles this perfectly",
            "Moderate: some nuance, but well within a small model",
            "Hard: needs careful handling or domain knowledge",
            "Very hard: multi-step reasoning or high cost of being wrong",
        ],
    ),
}