How a pick is made

What is in the catalog

The catalog is the canonical model list from Models.dev, joined to provider listings in api.json. A page exists for each canonical model that outputs text. Embeddings, rerankers, and models that do not output text are left out. Dated and aliased provider rows that point at the same canonical id are collapsed to one row per provider.

Which price is shown

Rates are USD per 1 million tokens, which is how Models.dev publishes them.

1. If the lab's own provider listing has a positive input or output price, that listing is the reference. Google Vertex counts as Google. A lab listing whose id matches the model id is used even when it omits canonical_model_id.
2. Otherwise the reference is the positive provider rate closest to the median input and output price. The page names that provider.
3. Listings at $0 are ignored while any positive price exists. If every listing is $0, the model is marked listed-free and kept out of job rankings.
4. If providers disagree by more than 2% or $0.02 per million tokens, the page shows the range. The reference price is still published.

The monthly bill

monthly = requests/day × 30 × (input tokens × (1 − cache hit) × input price + input tokens × cache hit × cache-read price + output tokens × output price) ÷ 1,000,000

Cache hits use the cache-read rate. If the model has no cache-read price, the cache percentage is ignored and the page says so. Cache writes are shown on the model page and are not added on top of a miss. When the input is longer than a published context tier, the calculator uses that tier's rates.

The quality bar

A model has to clear every line before its job score is considered.

• The lab is one of: openai, anthropic, google, meta, mistral, deepseek, xai, alibaba, moonshotai, zhipuai, cohere, amazon, nvidia, microsoft, ai21.
• At least 5 providers list it. Five is the minimum used here for “several”: two or three listings is usually the lab plus a reseller, which is how a tiny model such as Gemma 4 E4B reached a cheapest-wins page.
• It is not deprecated and not an alpha release. Beta stays in, because the lab is still selling it.
• The name is not a specialized tool: OCR, embeddings, rerankers, speech, safeguard, or moderation.
• It has the capabilities the job asks for. Coding agents require tool calls and reasoning. Long-document Q&A requires a context window of at least 200,000 tokens. Every current job also requires a numeric price and skips $0 listings.

Which benchmark scores count

A Models.dev benchmark row is kept only when its source host is the model’s own lab, that lab’s Hugging Face organization, or the benchmark’s owner. Artificial Analysis, OpenRouter, and LLM Stats are dropped. Another lab’s blog citing a competitor is dropped too. When one benchmark has several cited scores, the ranking uses the median, so a wide harness spread cannot be cherry-picked.

SWE-Bench Verified, SWE-Bench Multilingual, and SWE-Bench Multimodal count from swebench.com. SWE-Bench Pro also counts from labs.scale.com. Aider Polyglot counts from aider.chat. SWE-Atlas counts from labs.scale.com. MRCR counts when the lab published it. GPQA Diamond counts when the lab published it.

A different score for each job

The score is a fine-grained number, shown to one decimal, so exact ties are rare. What is left of a tie breaks to more providers, then a lower monthly cost, then a newer release date.

Coding agents

The score starts from the hardest cited software-engineering benchmark the model has, in this order: SWE-Bench Pro, SWE-Bench Verified, Terminal-Bench, DeepSWE, Aider Polyglot, then SWE-Atlas Codebase QnA. That result is a percentile against the other models on the page. A fallback bench is capped (0.94, 0.86, 0.80, 0.80, 0.78) so a lone terminal score cannot beat a high SWE-Bench Pro. Other cited coding benches add up to 6 points. Structured output adds 3 tool-calling points, an explicit false removes 3, and a cited Toolathlon or MCP Atlas result adds 2. Preview and snapshot names lose 5.

Long-document Q&A

The score is 100 times the log of the context window relative to 1,048,576 tokens, minus 10 times the log of one plus the input rate this workload actually pays, including a long-context tier. Compact names lose 22. A description that positions the model as the cheap high-volume option loses 6. Preview and snapshot names lose 8. A cited MRCR score adds up to 24 points, which is enough for a measured long-context result to matter beside the raw window size.

Summarization

The score is quality minus 10 times the log of the monthly cost. Doubling the bill costs 10 quality points. Quality starts at 22 and adds reasoning, structured output, tool calls, the log of the context window, and up to 18 from a cited GPQA Diamond score. General-purpose names add 14. Efficiency names such as flash, mini, haiku, small, or turbo add 6. Nano, micro, tiny, edge sizes, and active or dense sizes under 8B lose 16. The premium card uses raw quality, so a stronger model can win the upgrade even when it loses on cost per quality.

Compact, for the long-document penalty and the summarization name class, means an efficiency word in the name, an edge size such as E4B, a dense size under 32B, or a mixture-of-experts active size under 16B. On every job, a name with no version, size, or generation digit loses 8 points. Those rows are usually an unversioned alias, such as “DeepSeek Chat”, sitting next to a named release.

The top pick is the highest job score. On coding agents, models within 4 points of that leader are treated as one flagship band: the lower monthly bill is the top pick and the higher bill is the premium pick. Otherwise the premium pick is the best upgrade that costs more than the top pick. Output rates above 50 dollars per million tokens stay out of that card. The budget pick is the highest job score that costs less than the top pick and sits at or below the 25th cost percentile. If the top pick is already the cheapest model on the list, the page says so instead of inventing a budget card. Each card shows one line on why it was chosen and the monthly cost. The calculator reprices the same models when the workload changes. It does not re-rank them.

The three pages shipped in this version — coding agents, long-document Q&A, and cheapest summarization — are draft. Their prose is a placeholder, they are noindex, and they stay out of the sitemap until draft is set to false.

Compare pages

Popularity is the number of Models.dev providers that list the model. The top 50 by that count are grouped by family. Inside each family, pairs are generated for at most eight models. The canonical URL is the alphabetical lab--model-vs-lab--model form.

What this is not

The job score is not a private leaderboard. It uses a cited lab or owner score when one exists, plus the price and limits Models.dev publishes. Third-party leaderboards are not republished. OpenRouter is checked internally and its fields are not republished. Artificial Analysis and LLM Stats are not used.

Thin pages and holds

A model page is noindex when it lacks a price or a context window, or when no provider lists it. A refresh that would drop more than a quarter of a large catalog is refused. A removal, or an input or output price that moves more than 50%, is marked held. The daily workflow will not commit a held diff until someone allows it.