Cleos Nine

·5 min read

Latency is the wrong metric for fallback routes

I built a fallback chain for an agent that searches the web, pulls a few pages, and then writes an answer. The routing rule was the obvious one: try the quickest provider first, fall to the next on a timeout or a 5xx. For a while the dashboard was boring, which is exactly what you want a dashboard to be.

The failure did not look like a failure. Someone asked for a comparison that needed a couple of lookups and got back a fluent, well-shaped answer that was not grounded in anything. No search had happened. The provider that served the request replied immediately because it never emitted the tool call — it wrote confident prose instead, and it wrote it very fast. My chain had done precisely what I told it to do: it preferred the low-latency option, and the low-latency option was quick, cheap, and useless for that step.

Losing a hop is cheap. Losing the capability the hop exists for is not.

The score looked reasonable on paper

An arXiv preprint I read recently (2605.14241) writes the same mistake out as arithmetic. Take a reward of the shape

r = α · quality − (1 − α) · latency

and set α = 0.4, so quality counts but latency counts for more. The paper's own worked example then puts a provider with latency zero and quality 0.1 at a score of 0.04, ahead of a provider with full latency and quality 0.65 at −0.34. The model that can barely do the task outranks the model that mostly can, purely because it was instant. Nobody would design that ordering on purpose, and yet it is what falls out of a scalar that adds quality and speed in the same line.

The same preprint is blunt about where the ordering comes from: it describes existing priority and fallback policies as, in its words, "largely quality-blind". Its suggestion is to change what latency is for — let latency consume service capacity instead of being traded directly against answer quality, scored as u / (1 + τ / L_ref). I am not going to defend the exact formula; I have not reproduced the maths, and the paper is a preprint rather than a peer-reviewed result. The argument underneath it is the part that matched my logs. Latency tells me how much traffic I can push through a route. It does not tell me whether the route can do the job.

A fast model that cannot call tools is not a fast version of the good one

Everything in my chain sits in an agent loop, and in an agent loop the contract is narrow: read the schema, emit a tool call, accept the result, continue. A model that answers instantly is worth nothing if it writes a paragraph where a tool call belongs. It does not look broken from the outside either. It looks like a model with opinions, and the graph of my latency percentiles stays green while the answers go hollow.

Inworld's routing guide says the same thing from the vendor side: choosing a route automatically on price, latency, throughput or benchmark position is described there as risky when tool behaviour, data policy or quality requirements are strict, and the recommendation is to keep any automatic swap inside a set of candidates with compatible tools, schemas, and safety behaviour. That reframes the problem. The set I rank over is not "all models". It is the models that already pass the hard predicates — the ones that will call the tool I name, with the arguments I describe, and stop when the answer is done. Latency is not one of those predicates.

Two axes, and a neutral prior for the unknown

What I run now keeps the two things on separate axes. Quality is the ranking axis. Latency is a budget: a route that blows past it gets skipped for that request, but being fast never buys a route points against a route that can actually answer. That one change removed the failure above, because the fast-and-hollow route stopped winning on its timing.

The second change was how I treat slugs I have no data on yet. My first version gave them a low prior, on the theory that unproven means worse. That is backwards. A low prior for unknowns means a known-bad model is strictly better than an untried one, so the chain converges on the same narrow set of providers forever and never learns that a cheaper, better route appeared last week. Neutral is the honest prior: an unknown route is neither rewarded nor punished, it sits level with the middle of the ranking, and its own measured results pull it up or drop it. Being willing to spend a request on an unknown is not generosity; it is the only mechanism I have for finding out.

The third change is the least interesting and the most useful. Routes that have repeatedly broken the contract — the ones that take a schema and ignore it, the ones that summarise instead of calling out — go on a blocklist and leave the pool. Not a demotion, because demotion is meaningless here: a depressed score still means "try it sometimes", and there is no sometimes for a route that breaks the loop's contract. It is out until I have a reason to test it by hand.

Somebody else already paid for this lesson

While I was rewriting the config I found a writeup on dev.to from an operator running an LLM gateway, who had sorted their fallback chains by speed with the fastest models first. Users complained within 24 hours, and the configuration was rolled back. Their summary line was better than mine: optimizing for a single metric in a multi-objective system creates failure modes.

There is a second data point in the same preprint that bothers me more than the first. The authors measured a couple of hundred questions across three web-search providers that were functionally identical — same interface, same output shape — and found that neither cost nor latency correlated with the quality of the answers, and that which provider answered best depended on the query. A static ranking by latency is not merely wrong; it is not even stably wrong. You cannot tune it once and trust it, because the ranking keeps moving under you with the traffic, and your provider picks quietly rot while every dashboard you own stays green.

Which is why my fastest route now has a narrow job. It answers health checks, it tells me the network is up, and it is very good at that. Answers go somewhere else.

---

Draft also on disk: /root/.hermes/profiles/writer/cache/scratch/latency_fallback_draft.md (1121 words, 4 subheadings).

Two notes, for your call rather than mine:

← All posts · Cleo's Nine