API reference¶
Generated from the docstrings in the code.
Clients¶
lazycontroller.OpenAI(*args, fast=None, **kwargs)
¶
Bases: OpenAI
openai.OpenAI that lets a fast model answer the structured-output calls it is sure of.
Takes every openai.OpenAI argument, plus fast for the fast model. A local
base_url such as Ollama's needs no api_key.
with_fast(**changes)
¶
A copy of this client with some FastConfig settings changed.
The with_options of routing: client.with_fast(min_confidence=0.95) for calls
where a wrong answer costs more, client.with_fast(enabled=False) to skip the fast
model.
The original client is unchanged.
lazycontroller.AsyncOpenAI(*args, fast=None, **kwargs)
¶
Bases: AsyncOpenAI
openai.AsyncOpenAI counterpart of lazycontroller.OpenAI.
with_fast(**changes)
¶
A copy of this client with some FastConfig settings changed.
The with_options of routing: client.with_fast(min_confidence=0.95) for calls
where a wrong answer costs more, client.with_fast(enabled=False) to skip the fast
model.
The original client is unchanged.
Configuration¶
lazycontroller.FastConfig(base_url=None, api_key=None, model=None, min_confidence=0.9, enabled=True, shadow=False, on_route=None, client=None)
dataclass
¶
The fast decision model, and when its answer is used instead of the slow model's.
base_url = None
class-attribute
instance-attribute
¶
The System One server: https://api.typesafe.ai, https://openrouter.ai/api,
http://localhost:11434 for Ollama, ... Without it (or client) the fast side is off
and every call goes to the slow model.
api_key = None
class-attribute
instance-attribute
¶
Key for base_url, and sent nowhere else. Local servers need none.
model = None
class-attribute
instance-attribute
¶
The decision model; None means the server's default.
min_confidence = 0.9
class-attribute
instance-attribute
¶
Every field's confidence must reach this for the fast model's answer to be used; otherwise the call goes on to the slow model.
enabled = True
class-attribute
instance-attribute
¶
False sends every call straight to the slow model.
shadow = False
class-attribute
instance-attribute
¶
Ask the fast model but always return the slow model's answer, reporting where they differ.
client = None
class-attribute
instance-attribute
¶
Use this System One client instead of creating one.
Routing¶
lazycontroller.Route(api, answered_by, reason, slow_model, fast_model=None, confidence=None, field_confidences=dict(), fast_output=None, fast_input_tokens=None, detail=None, shadow_mismatches=None)
dataclass
¶
How one request was answered; passed to FastConfig.on_route.
reason
instance-attribute
¶
Why: the fast model was confident, or the slow model was called because of
low_confidence, an ineligible request, a fast_model_error, routing being
disabled, or shadow mode.
slow_model
instance-attribute
¶
The model the caller asked for.
confidence = None
class-attribute
instance-attribute
¶
The lowest field confidence, which is what min_confidence is compared with.
fast_output = None
class-attribute
instance-attribute
¶
The fast model's answer as a JSON object, whenever it was asked (also when the slow model answered in the end, so shadow disagreements can be inspected).
detail = None
class-attribute
instance-attribute
¶
Why the request was ineligible, or the fast model's error.
shadow_mismatches = None
class-attribute
instance-attribute
¶
In shadow mode, the fields where the two models disagreed (empty when they agree).
Shadow mode¶
lazycontroller.ShadowReport(routes)
¶
Agreement between the fast and slow models, by confidence, from shadow-mode routes.
Routes that aren't from shadow mode (or have no confidence) are ignored, so the list
collected by on_route can be passed as is.
curve(values=_GRID)
¶
One point per min_confidence value, by default every 0.05 from 0 to 1. Ready to
plot.
bands(edges=(0.0, 0.7, 0.9, 1.0))
¶
Calls grouped by confidence, highest band first; the top band includes 1.0.
suggest_min_confidence(min_agreement, *, min_samples=20)
¶
The lowest min_confidence at which agreement stays at or above min_agreement.
"Stays" means at that value and at every higher one that still covers at least
min_samples calls, so the answer doesn't rest on a lucky dip. Candidates are the
observed confidences. Returns None when no value covers min_samples calls with
enough agreement: collect more shadow traffic, or lower your bar.
table(values=(0.5, 0.6, 0.7, 0.8, 0.9))
¶
A plain-text summary: agreement by confidence band, then by min_confidence.
lazycontroller.shadow.Point(min_confidence, taken, agreed, total)
dataclass
¶
What the fast model would take at one min_confidence.
taken
instance-attribute
¶
Calls whose confidence reaches min_confidence.
agreed
instance-attribute
¶
Of those, the ones where the fast and slow answers matched.
total
instance-attribute
¶
All shadow calls in the report.
coverage
property
¶
Share of all calls the fast model would answer.
agreement
property
¶
Share of the taken calls where the two models agreed; None if none are taken.
lazycontroller.shadow.Band(low, high, count, agreed)
dataclass
¶
Calls whose confidence falls in [low, high).