Inkling’s first real test is not the headline benchmark

Inkling’s first real test is not the headline benchmark

4 min read

Mira Murati’s Thinking Machines Lab finally has a model in the market, and the useful question is not whether Inkling wins a review headline. It is whether its benchmark strength survives price, latency, licensing, and boring production tests.

TL;DR: Inkling looks like a serious debut from Thinking Machines Lab, but builders should treat the impressive MCP score as a starting signal, not a deployment answer.

What did Thinking Machines Lab actually ship?

Mira Murati’s Thinking Machines Lab has been quiet for roughly two years. Now its debut model, Inkling, is out and available through OpenRouter, according to Decrypt’s “Mira Murati’s Inkling AI Model Review: Best Open-Source Model in the West.”

That matters for two reasons.

First, Murati is not a random founder with a model card. She was OpenAI’s CTO and one of the more visible operators behind ChatGPT’s early product era. So when her lab ships, people pay attention.

Second, availability through OpenRouter changes the adoption path. A model does not need a bespoke enterprise sales cycle to get tested anymore. If it is on a common routing layer, builders can put it next to Claude, Gemini, GPT, Mistral, Qwen, Llama variants, and whatever they already use. That makes the first wave of judgment much less theatrical and much more useful.

Run the same prompts. Same temperature. Same tools. Same eval harness. Same latency budget. Same retry policy. Then see what breaks.

The Decrypt review’s headline claim is strong: “Best Open-Source Model in the West.” I would separate the interesting part from the loaded part. The interesting part is that Inkling apparently posts a genuinely impressive MCP score. The loaded part is “open-source,” because access through OpenRouter is not the same thing as full freedom to inspect, modify, host, and redistribute a model.

If the weights, license, training disclosure, and commercial terms are not clear, treat “open-source” as a claim to verify, not a feature to assume.

two model deployment paths splitting from one model, one path through a hosted router and one path into self-hosted infr

Does a high MCP score mean Inkling is ready for production?

Not by itself.

A benchmark can tell you a model deserves attention. It cannot tell you the model deserves your production traffic. That gap is where teams get burned.

Decrypt says Inkling’s MCP score is genuinely impressive. Good. That suggests the model may be strong on the specific tasks behind that score. But a builder needs to know what the score is actually measuring. Tool use? Multi-step reasoning? Coding? Context handling? Instruction following? Agentic workflows? The acronym matters less than the coverage.

Most apps fail in the boring edges: malformed JSON, refusal weirdness, citation drift, subtle instruction misses, long-context decay, tool-call loops, latency spikes, cost overruns, and safety behavior that changes when a prompt includes real customer data.

That is why I would not frame Inkling as “beats X” or “doesn’t beat Y” yet. I would frame it as: this model has earned a slot in the eval queue.

The better early test is not a public leaderboard. It is your private failure set. Pull 100 to 500 cases where your current model has disappointed you. Include the ugly prompts. Messy user inputs. Ambiguous requests. Tool calls with missing fields. Long documents with one buried fact. Support tickets where tone matters. Code changes where one wrong import ruins the patch.

Then run Inkling against your incumbent. Do blind review where possible. Measure pass rate, cost, latency, and correction burden. The last one matters. A model that is 5% better on answer quality but requires twice as many retries may be worse in the app.

Where does price-performance get tricky?

This is the part I like most in the Decrypt framing: the price-to-performance math is “more complicated.”

That is usually true. Cheap tokens are not always cheap systems. Expensive models are not always expensive if they finish the job in one pass.

For Inkling, the practical question is not only “what is the price per token on OpenRouter?” It is also how often the model needs retries, how many tokens it burns while reasoning or recovering, whether it follows tool schemas cleanly, and whether you need a second model to check its work.

OpenRouter also adds a distribution layer. That is convenient. It may also affect latency, routing behavior, logging posture, and availability. None of that is bad by default. It just belongs in the real cost model.

I would test Inkling first in contained workflows: internal research assistants, coding review helpers, structured extraction, eval generation, and agent prototypes where the blast radius is low. If it performs well there, move toward customer-facing use with logging, fallback models, and hard output validation.

For a builder, the play is simple: add Inkling to your eval bench this week, not your production default. Verify the license before calling it open-source, test it against your own failure cases, and compare total task cost rather than token price. The catch most readers miss is that “best model” is usually the wrong category. The useful category is “best model for this workflow, under this budget, with these failure costs.”