Budget vs Frontier: Our Test Data
What we measured when DeepSeek V4.1 Flash and Grok 4.6 were given the same job.
Test run 18 September 2026. Published 18 September 2026 by IDFS AI LLC.
Why this page exists
We publish articles that cite their sources, and an automated check fetches every cited URL before anything ships. That check does its job well enough that it blocked us from citing our own measurements, because a number with no public source behind it is exactly what the check is built to catch.
So here are the numbers, on a page anyone can open. This is first-party data from a single test run, not peer-reviewed research, and we have tried to be equally clear about what it does and does not show.
The test
On 18 September 2026 we gave two models an identical assignment: research, draft, and self-check three articles against the same written contract. The brief was byte-for-byte the same for both. Afterwards, an automated check that we control, not the model, independently re-verified each article by fetching every cited URL over the network and recomputing each article's own claim-support arithmetic.
- DeepSeek V4.1 Flash, accessed through OpenRouter, reasoning effort set to low.
- Grok 4.6 (xAI), via its own command-line agent.
- Both ran concurrently, on the same machine, against the same brief.
What we measured
| Measure | DeepSeek V4.1 Flash | Grok 4.6 |
|---|---|---|
| Articles completed | 3 of 3 | 3 of 3 |
| Passed all checks on first attempt | Yes | Yes |
| Wall clock for the batch | 27.1 minutes | 26.2 minutes |
| Words per article | 1,564 to 1,785 | 2,753 to 2,973 |
| External citations per article | 7 to 9 | 10 to 12 |
| Input tokens (batch) | 2,141,220 | not exposed by the tool |
| Input served from cache | 71.7% | not exposed by the tool |
| Output tokens (batch) | 25,336 | not exposed by the tool |
| Measured cost for the batch | $0.336, about $0.11 per article | subscription, not metered per token |
The wall-clock difference is 3.4%. Both runs included live web research and network verification of every source, which accounts for a large share of the elapsed time in both cases, so wall clock here is not a clean measure of raw model speed.
The behaviour worth reporting
During its run, DeepSeek V4.1 Flash flagged a statistic in its own draft, determined that the source was paywalled and could not be verified, and removed the number rather than keeping it. On another article it discarded four candidate sources that were dead or blocked to automated retrieval instead of citing them. We did not prompt either behaviour specifically; the brief required every claim to be supported, and the model chose deletion over an unsupported figure.
The counterpoint, which matters more than the result
On Artificial Analysis's AA-Omniscience index, DeepSeek V4.1 Flash scores -5 against Grok 4.6's +29. A negative score means the model produces more incorrect answers than correct ones. Its hallucination rate on that index, meaning how often it guesses rather than admitting it does not know, is 96.5% against 34.3%.
Those figures describe closed-book behaviour: answering from memory with nothing to look at. Our test never asked either model to do that. Every claim had to be retrieved, cited, and verified against a live source. That is the whole finding, and it is the part worth taking away. A cheap model inside enforced retrieval with every citation checked is viable. The same model trusted to answer from memory is not.
Limitations
- One run each. Three articles per model, a single batch, one day. This is an observation, not a pass rate.
- Token counts on this model vary a lot between runs. In separate testing the same trivial task ranged from roughly 800 to 36,000 tokens. Treat single-run token figures with caution, including ours.
- The two tools are not identical harnesses. Each model ran in the agent framework built for it, so some of the difference is tooling rather than model.
- Cost is not like-for-like. DeepSeek was billed per token; Grok ran on a subscription that does not meter per-token usage, so we have not published a cost comparison and are not implying one.
- Published list prices for context: DeepSeek V4.1 Flash at $0.15 per million input tokens and $0.60 per million output, off-peak, doubling during published peak hours; Grok 4.6 at $2.00 and $6.00. Prices as published on 18 September 2026 and subject to change.
- Quality was not scored. Passing our checks means the sourcing contract was met. It does not mean an editor judged one set of articles better than the other.