{"name":"gjalla model cards","url":"https://gjalla.io/models/","methodology":"https://gjalla.io/models/methodology/","verifiedOn":"2026-10-26","models":[{"model":"claude-fable-5","vendor":"anthropic","tier":"frontier","price":{"input":10,"output":50,"cacheRead":1,"cacheWrite":12.5,"cacheWrite1h":20,"source":"platform","sourceUrl":null},"meta":{"released":null,"contextWindow":null,"maxOutput":null,"modalities":null,"officialCardUrl":null},"excelsAt":[],"watchOuts":[],"benchmarks":{"coding":{"pct":0.174,"benchmarks":[{"benchmark":"lmarena code arena webdev","score":1624,"pct":0.174,"independent":true,"sourceKind":"independent","sourceUrl":"https://arena.ai/leaderboard/code/webdev","date":"2026-10-19"}]},"coding_agentic":{"pct":0.188,"benchmarks":[{"benchmark":"terminal-bench 2.1 (aggregator, anotherwrapper.com)","score":84.3,"pct":0.188,"independent":true,"sourceKind":"independent","sourceUrl":"https://anotherwrapper.com/tools/llm-pricing/leaderboards/terminal-bench","date":"2026-10-20"}]}},"anecdotes":[],"url":"https://gjalla.io/models/claude-fable-5/"},{"model":"claude-fable-5.1","vendor":"anthropic","tier":"frontier","price":{"input":10,"output":50,"cacheRead":0.25,"cacheWrite":12.5,"cacheWrite1h":20,"source":"platform","sourceUrl":null},"meta":{"released":"2026-09-01","contextWindow":1000000,"maxOutput":128000,"modalities":["text","image"],"officialCardUrl":"https://www-cdn.anthropic.com/0339e6a7c5c7b87f5c07798616dc32c215d14235/Claude%20Fable%205.1%20&%20Claude%20Mythos%205.1%20System%20Card.pdf"},"excelsAt":[{"point":"Built for long-running agentic coding, multistep research, and document, spreadsheet and slide work over extended sessions.","basis":"vendor","sourceKind":"primary","sourceUrl":"https://www.anthropic.com/claude-fable-and-mythos-5-1"},{"point":"Big gains over Fable 5 on terminal and agentic science tasks; Terminal-Bench 4.0 reported at 55.8 percent.","basis":"vendor","sourceKind":"primary","sourceUrl":"https://www.anthropic.com/claude-fable-and-mythos-5-1"},{"point":"Cheaper cache reads make long agent runs roughly 25 to 45 percent cheaper than Fable 5, by Anthropic's estimate.","basis":"vendor","sourceKind":"primary","sourceUrl":"https://www.anthropic.com/claude-fable-and-mythos-5-1"},{"point":"Ranked first on Vals AI's Excel modeling benchmark at 76.7 percent accuracy, narrowly ahead of Opus 5.5.","basis":"independent","sourceKind":"independent","sourceUrl":"https://www.vals.ai/benchmarks/emb"},{"point":"Plans architecture first and finishes long coding tasks unprompted; preferred over GPT-6 Astra for finishing software projects in one comparison.","basis":"independent","sourceKind":"independent","sourceUrl":"https://hierarchy.ch/en/news/gpt-6-astra-vs-claude-fable-5-1"},{"point":"Prose reads with fewer AI tells and beat GPT-5.6 Sol on judgment-heavy work like decks and persona answers.","basis":"independent","sourceKind":"independent","sourceUrl":"https://every.to/vibe-check/fable-5-1-vibe-check"},{"point":"Legal testers found it strongest on general legal knowledge and on separating what a document says from what it infers (vendor-run test).","basis":"practitioner","sourceKind":"community","sourceUrl":"https://www.litera.com/blog/which-claude-model-best-legal-work"}],"watchOuts":[{"point":"Overdelivers: exceeded explicit word, theme and quote limits, and some quoted passages were not in the source text.","basis":"practitioner","sourceKind":"community","sourceUrl":"https://every.to/vibe-check/fable-5-1-vibe-check"},{"point":"At extra-high effort it can spawn unneeded subagents and keep going after interruption; set a budget first.","basis":"practitioner","sourceKind":"community","sourceUrl":"https://theneuron.ai/news/claude-fable-5-1-cuts-agent-costs-and-safety-friction/"},{"point":"Can burn through subscription allowances fast on fleet-style multi-agent jobs, stalling work until reset.","basis":"practitioner","sourceKind":"community","sourceUrl":"https://ainativeproductmanager.com/field-notes/fable-5-1-first-impressions"},{"point":"Penetration testing, exploit work and life-science R&D requests are rerouted to Opus models by safeguards.","basis":"vendor","sourceKind":"primary","sourceUrl":"https://www.anthropic.com/claude-fable-and-mythos-5-1"},{"point":"In legal testing it was the slowest and most expensive, wrote the longest answers, and gave fully complete answers less often than Opus 5.5 (vendor-run test).","basis":"practitioner","sourceKind":"community","sourceUrl":"https://www.litera.com/blog/which-claude-model-best-legal-work"}],"benchmarks":{"coding":{"pct":0.826,"benchmarks":[{"benchmark":"lmarena code arena webdev","score":1743,"pct":0.826,"independent":true,"sourceKind":"independent","sourceUrl":"https://arena.ai/leaderboard/code/webdev","date":"2026-10-19"}]},"coding_agentic":{"pct":0.552,"benchmarks":[{"benchmark":"cursorbench 4.0","score":51.8,"pct":0.333,"independent":false,"sourceKind":"primary","sourceUrl":"https://www.anthropic.com/claude-opus-5-5","date":"2026-09-22"},{"benchmark":"swe-bench pro","score":81.2,"pct":0.917,"independent":false,"sourceKind":"relayed","sourceUrl":"https://computingforgeeks.com/claude-fable-5-1-released-features-benchmarks/","date":"2026-09-01"},{"benchmark":"terminal-bench 2.1 (aggregator, anotherwrapper.com)","score":85,"pct":0.333,"independent":true,"sourceKind":"independent","sourceUrl":"https://anotherwrapper.com/tools/llm-pricing/leaderboards/terminal-bench","date":"2026-10-19"},{"benchmark":"terminal-bench 4.0","score":55.8,"pct":0.625,"independent":false,"sourceKind":"relayed","sourceUrl":"https://computingforgeeks.com/claude-fable-5-1-released-features-benchmarks/","date":"2026-09-01"}]},"knowledge":{"pct":0.923,"benchmarks":[{"benchmark":"humanity's last exam","score":59.1,"pct":0.846,"independent":true,"sourceKind":"independent","sourceUrl":"https://www.deeplearning.ai/the-batch/fable-holds-the-top-spot-for-now","date":"2026-09-11"},{"benchmark":"vals excel modeling benchmark (emb v1.0)","score":76.67,"pct":1,"independent":true,"sourceKind":"independent","sourceUrl":"https://www.vals.ai/benchmarks/emb","date":"2026-10-18"}]},"overall":{"pct":0.757,"benchmarks":[{"benchmark":"artificial analysis intelligence index","score":53,"pct":0.696,"independent":true,"sourceKind":"independent","sourceUrl":"https://artificialanalysis.ai/leaderboards/models","date":"2026-10-19"},{"benchmark":"gdpval-aa v2","score":1853,"pct":1,"independent":false,"sourceKind":"relayed","sourceUrl":"https://the-decoder.com/anthropics-claude-fable-5-1-promises-better-coding-and-research-at-up-to-45-percent-less/","date":"2026-09-01"},{"benchmark":"gdpval-aa v2.1","score":1735,"pct":0.5,"independent":false,"sourceKind":"primary","sourceUrl":"https://www.anthropic.com/claude-opus-5-5","date":"2026-09-22"},{"benchmark":"vals index","score":68.83,"pct":0.833,"independent":true,"sourceKind":"independent","sourceUrl":"https://www.deeplearning.ai/the-batch/fable-holds-the-top-spot-for-now","date":"2026-09-11"}]},"vision":{"pct":0.667,"benchmarks":[{"benchmark":"chartography","score":88.4,"pct":0.667,"independent":false,"sourceKind":"primary","sourceUrl":"https://www.anthropic.com/claude-opus-5-5","date":"2026-09-22"}]}},"anecdotes":[{"claim":"Author gave it one prompt to write an image codec in Go and got something that beat WebP on compression by 1-9% and ran faster, though tested on under 50 images and not production-ready.","taskFamily":"feature","taskDescription":null,"characteristics":[],"direction":"better","comparedTo":null,"sourceType":"blog","sourceKind":"community","sourceUrl":"https://thomas-wiegold.com/blog/claude-fable-5-1-review/","date":"2026-09-04"},{"claim":"Same reviewer could not confirm the final output is better than Fable 5, only that it feels more direct and less verbose; suggests defaulting to Opus 5 and escalating to Fable 5.1 only for hard or high-stakes work.","taskFamily":"feature","taskDescription":null,"characteristics":[],"direction":"mixed","comparedTo":"claude-opus-5","sourceType":"blog","sourceKind":"community","sourceUrl":"https://thomas-wiegold.com/blog/claude-fable-5-1-review/","date":"2026-09-04"},{"claim":"On subscription Max plans it burns weekly limits faster than other Claude models; Reddit users reported five-hour windows gone in 15-20 minutes with subagent fan-outs, while token-billed API cost per turn was about 43% lower than Fable 5 in the author's transcripts. Full-file rewrites rose to 44% of file operations from 16%.","taskFamily":"agentic-long-running","taskDescription":null,"characteristics":[],"direction":"mixed","comparedTo":"claude-fable-5","sourceType":"blog","sourceKind":"community","sourceUrl":"https://paddo.dev/blog/fable-5-1-two-meters/","date":null},{"claim":"Every's reviewers still prefer Fable 5.1 over Opus 5.5 for the hardest, biggest problems and for deliverables that must be right first time; it also made a better branded deck (Opus 5.5 used wrong logo and colours).","taskFamily":"writing-content","taskDescription":null,"characteristics":[],"direction":"better","comparedTo":"claude-opus-5.5","sourceType":"blog","sourceKind":"community","sourceUrl":"https://every.to/vibe-check/vibe-check-opus-5-5-is-pulling-our-codex-converts-back-to-claude","date":"2026-09-22"},{"claim":"In an Endor Labs run of 179 coding tasks under Claude Code it passed 87.2% of functional tests versus 68.7% for Opus 5.5 (memorization removed) but only 37.4% of hidden security tests; it took about 9.5 min per task and the run cost $672 versus $116 for Opus 5.5.","taskFamily":"bug-fix","taskDescription":null,"characteristics":[],"direction":"mixed","comparedTo":"claude-opus-5.5","sourceType":"blog","sourceKind":"community","sourceUrl":"https://www.endorlabs.com/learn/opus-5-5-6x-cheaper-and-2x-faster-than-fable-5-1-but-memorization-keeps-it-off-the-top-spot","date":"2026-09-24"},{"claim":"Artificial Analysis found it leads on long-running tool use but trails GPT-6 Astra on overall knowledge, computer use and multistep terminal work, at higher cost and time per task; at max effort it used about 1.7x the output tokens of Fable 5.","taskFamily":"agentic-long-running","taskDescription":null,"characteristics":[],"direction":"mixed","comparedTo":null,"sourceType":"news","sourceKind":"community","sourceUrl":"https://www.deeplearning.ai/the-batch/fable-holds-the-top-spot-for-now","date":"2026-09-11"},{"claim":"Jane Street's Craig Falls (customer quote in Anthropic launch coverage) said outputs stay readable across long multi-step tasks.","taskFamily":"agentic-long-running","taskDescription":null,"characteristics":[],"direction":"better","comparedTo":null,"sourceType":"vendor","sourceKind":"relayed","sourceUrl":"https://sdtimes.com/claude-fable-5-1/61089/","date":"2026-09-02"},{"claim":"Week of team testing: matched Fable 5 on long delegated coding, wrote cleaner prose, but broke explicit count and format limits that Opus 5 respected.","taskFamily":null,"taskDescription":null,"characteristics":[],"direction":"mixed","comparedTo":"claude-opus-5","sourceType":"blog","sourceKind":"community","sourceUrl":"https://every.to/vibe-check/fable-5-1-vibe-check","date":"2026-09-01"},{"claim":"Reviewer found Fable 5.1 needed about two PR corrections versus six for Astra, understood intent better, and produced polished one-shot front ends; Astra won on computer use and 3D.","taskFamily":null,"taskDescription":null,"characteristics":[],"direction":"better","comparedTo":"gpt-6-astra","sourceType":"blog","sourceKind":"community","sourceUrl":"https://finance.biggo.com/news/eb22b1f124ad5b97","date":"2026-09-07"},{"claim":"Author would hand a whole software project to Fable 5.1 over Astra, though Astra clearly won Blender and 3D work; Fable over-delivered and spawned unrequested subagents.","taskFamily":null,"taskDescription":null,"characteristics":[],"direction":"better","comparedTo":"gpt-6-astra","sourceType":"blog","sourceKind":"community","sourceUrl":"https://hierarchy.ch/en/news/gpt-6-astra-vs-claude-fable-5-1","date":"2026-09-08"},{"claim":"Rewrite ran with few nudges and resumed cleanly after a crash, but cost estimates ran low and the top-tier five-hour allowance ran out in 43 minutes.","taskFamily":null,"taskDescription":null,"characteristics":[],"direction":"mixed","comparedTo":"claude-fable-5","sourceType":"blog","sourceKind":"community","sourceUrl":"https://ainativeproductmanager.com/field-notes/fable-5-1-first-impressions","date":"2026-09-02"},{"claim":"Reviewer said Astra trailed Fable 5.1 by a wide margin on every coding task, especially UI polish, while Astra's computer-use work was strong.","taskFamily":null,"taskDescription":null,"characteristics":[],"direction":"better","comparedTo":"gpt-6-astra","sourceType":"blog","sourceKind":"community","sourceUrl":"https://medium.com/jonathans-musings/a-short-review-of-openais-gpt-6-astra-091fa5beccf9","date":"2026-09-05"},{"claim":"Testers found Astra won most head-to-heads on coding and reasoning and needed less re-explaining, though raw benchmarks were roughly level.","taskFamily":null,"taskDescription":null,"characteristics":[],"direction":"worse","comparedTo":"gpt-6-astra","sourceType":"blog","sourceKind":"community","sourceUrl":"https://www.mindstudio.ai/blog/gpt-6-astra-first-impressions","date":"2026-09-13"},{"claim":"Commenter noted Claude Code hits usage limits mainly when running Fable for everything, and writes its own monitor script otherwise.","taskFamily":null,"taskDescription":null,"characteristics":[],"direction":"mixed","comparedTo":null,"sourceType":"hacker-news","sourceKind":"community","sourceUrl":"https://news.ycombinator.com/item?id=49881850","date":null},{"claim":"Day-zero users kept Fable for the hardest problems, deadline-critical deliverables and prose, while Opus 5.5 covered everyday coding at lower cost.","taskFamily":null,"taskDescription":null,"characteristics":[],"direction":"better","comparedTo":"claude-opus-5.5","sourceType":"blog","sourceKind":"community","sourceUrl":"https://every.to/vibe-check/vibe-check-opus-5-5-is-pulling-our-codex-converts-back-to-claude","date":"2026-09-22"},{"claim":"Community default in launch-week threads: Opus for scoped build-and-fix, Fable for open-ended or knowledge-heavy problems (secondhand Reddit digest).","taskFamily":null,"taskDescription":null,"characteristics":[],"direction":"better","comparedTo":"claude-opus-5.5","sourceType":"newsletter","sourceKind":"community","sourceUrl":"https://buttondown.com/thedailyclaude/archive/the-daily-claude-2026-09-24/","date":"2026-09-24"}],"url":"https://gjalla.io/models/claude-fable-5.1/"},{"model":"claude-haiku-5.5","vendor":"anthropic","tier":"fast","price":{"input":0.1,"output":0.5,"cacheRead":0.01,"cacheWrite":0.125,"cacheWrite1h":0.2,"longAbove":100000,"long":{"input":0.5,"output":2.5,"cacheRead":0.05,"cacheWrite":0.625,"cacheWrite1h":1},"source":"platform","sourceUrl":null},"meta":{"released":"2026-10-07","contextWindow":1000000,"maxOutput":128000,"modalities":["text","image"],"officialCardUrl":"https://www.anthropic.com/claude-haiku-5-5"},"excelsAt":[{"point":"Built for high-volume, latency-sensitive work: summaries, compaction, database queries, classification, extraction and live customer support.","basis":"vendor","sourceKind":"primary","sourceUrl":"https://www.anthropic.com/claude-haiku-5-5"},{"point":"Intended as a cheap subagent alongside Opus 5.5 or Sonnet 5.5, with browser and computer-use beta support.","basis":"vendor","sourceKind":"primary","sourceUrl":"https://www.anthropic.com/claude-haiku-5-5"},{"point":"First Haiku with adjustable effort and a 1M context window; $0.10 and $0.50 per million tokens up to 100K prompt.","basis":"independent","sourceKind":"independent","sourceUrl":"https://www.gradually.ai/en/ai-models/claude-haiku-5-5/"},{"point":"Matched Haiku 4.5 on a 24-invoice approval test with under a quarter of the output tokens, about 25x cheaper.","basis":"independent","sourceKind":"independent","sourceUrl":"https://www.datacamp.com/blog/claude-haiku-5-5"},{"point":"Passed all 140 scoped subagent runs on a small repo at low and medium effort, far cheaper than Opus.","basis":"independent","sourceKind":"independent","sourceUrl":"https://dev.to/aidiveyt/your-claude-code-subagents-never-touch-haiku-55-i-measured-what-routing-them-there-gives-back-53h2"},{"point":"Passed 37 of 43 business analytics questions at about $0.38 per run and the lowest median latency (14s) in Plotly's test.","basis":"practitioner","sourceKind":"community","sourceUrl":"https://plotly.com/blog/introducing-dataanalyticsbench/"}],"watchOuts":[{"point":"Well behind Sonnet 5.5 on agentic coding (Terminal-Bench 4.0: 39.2 versus 70.6 percent); Anthropic steers complex coding elsewhere.","basis":"vendor","sourceKind":"primary","sourceUrl":"https://www.anthropic.com/claude-haiku-5-5"},{"point":"Whole-request price jumps fivefold above 100K tokens, and the new tokenizer uses roughly 30 percent more tokens.","basis":"vendor","sourceKind":"relayed","sourceUrl":"https://www.gradually.ai/en/ai-models/claude-haiku-5-5/"},{"point":"Anthropic warns xhigh effort can put the answer in thinking, returning no visible reply.","basis":"vendor","sourceKind":"relayed","sourceUrl":"https://www.syntaxdispatch.com/blog/claude-haiku-5-5-review"},{"point":"The haiku alias still maps to Haiku 4.5 on Bedrock, Vertex and Foundry, so pin the full model ID.","basis":"practitioner","sourceKind":"community","sourceUrl":"https://dev.to/aidiveyt/your-claude-code-subagents-never-touch-haiku-55-i-measured-what-routing-them-there-gives-back-53h2"},{"point":"Found only 2 of 5 hidden-data-issue questions; GPT-6 Luna scored slightly higher (38/43) for less.","basis":"practitioner","sourceKind":"community","sourceUrl":"https://plotly.com/blog/introducing-dataanalyticsbench/"},{"point":"Ranked 38th on Vals' Excel Modeling Benchmark v1 at 59.0 percent (about $1.15 per test) versus about 76 percent for Fable 5.1, Opus 5.5 and Sonnet 5.5.","basis":"independent","sourceKind":"independent","sourceUrl":"https://www.vals.ai/benchmarks/emb"},{"point":"Artificial Analysis shows it is very verbose (about 440M output tokens on the index at max vs 99M median) despite 232 tokens per second output.","basis":"independent","sourceKind":"independent","sourceUrl":"https://artificialanalysis.ai/models/claude-haiku-5-5"}],"benchmarks":{"coding_agentic":{"pct":0.459,"benchmarks":[{"benchmark":"swe-bench pro","score":64.8,"pct":0.667,"independent":false,"sourceKind":"relayed","sourceUrl":"https://computingforgeeks.com/claude-haiku-5-5-released-features-benchmarks/","date":"2026-10-07"},{"benchmark":"terminal-bench 4.0","score":33,"pct":0.25,"independent":true,"sourceKind":"independent","sourceUrl":"https://computingforgeeks.com/claude-haiku-5-5-released-features-benchmarks/","date":null}]},"knowledge":{"pct":0.231,"benchmarks":[{"benchmark":"humanity's last exam","score":45.9,"pct":0.462,"independent":false,"sourceKind":"relayed","sourceUrl":"https://computingforgeeks.com/claude-haiku-5-5-released-features-benchmarks/","date":"2026-10-07"},{"benchmark":"vals excel modeling benchmark (emb v1.0)","score":59.02,"pct":0,"independent":true,"sourceKind":"independent","sourceUrl":"https://www.vals.ai/benchmarks/emb","date":"2026-10-18"}]},"overall":{"pct":0.233,"benchmarks":[{"benchmark":"artificial analysis intelligence index","score":43,"pct":0.217,"independent":true,"sourceKind":"independent","sourceUrl":"https://www.deeplearning.ai/the-batch/claude-fills-out-its-model-family/","date":"2026-10-16"},{"benchmark":"gdpval-aa v2.1","score":1620,"pct":0.25,"independent":false,"sourceKind":"relayed","sourceUrl":"https://apidog.com/blog/claude-haiku-5-5-benchmarks/","date":"2026-10-07"}]},"vision":{"pct":0,"benchmarks":[{"benchmark":"chartography","score":46.4,"pct":0,"independent":false,"sourceKind":"relayed","sourceUrl":"https://apidog.com/blog/claude-haiku-5-5-benchmarks/","date":"2026-10-07"}]}},"anecdotes":[{"claim":"On a small repo (tinydb, 226 tests) across seven routine subagent tasks, Haiku 5.5 at low and medium effort passed all 140 runs, same as Sonnet 5.5 and Opus 5.5, at roughly 40x lower cost for codebase exploration; author says it proves little beyond short tasks on one repo.","taskFamily":"chore","taskDescription":null,"characteristics":[],"direction":"mixed","comparedTo":"claude-opus-5.5","sourceType":"blog","sourceKind":"community","sourceUrl":"https://dev.to/aidiveyt/your-claude-code-subagents-never-touch-haiku-55-i-measured-what-routing-them-there-gives-back-53h2","date":"2026-10-12"},{"claim":"Another guide says it fits read-only subagent work (codebase search, CI/log triage, diff and thread summaries) but should stay off multi-file edits, hypothesis-driven debugging and anything going straight into a commit.","taskFamily":"research","taskDescription":null,"characteristics":[],"direction":"mixed","comparedTo":"claude-sonnet-5.5","sourceType":"blog","sourceKind":"community","sourceUrl":"https://prompteddev.com/blog/claude-haiku-5-5-subagent/","date":"2026-10-14"},{"claim":"Bito's six-task coding blend (Opus 5.5 as judge) scored Opus medium 85%, Sonnet high 82%, Haiku high 79%, Haiku medium 76%; the gap to Opus widened to 8-10 points on implement-and-audit turns, and a lookup task fell from 84% (high) to 59% (medium).","taskFamily":"feature","taskDescription":null,"characteristics":[],"direction":"worse","comparedTo":"claude-opus-5.5","sourceType":"blog","sourceKind":"community","sourceUrl":"https://bito.ai/blog/claude-haiku-5-5-vs-sonnet-5-5-vs-opus-5-5/","date":null},{"claim":"GitHub's early testing, cited at launch, found it matched Sonnet 5 on many coding tasks with fewer tokens and steps.","taskFamily":"feature","taskDescription":null,"characteristics":[],"direction":"better","comparedTo":"claude-sonnet-5","sourceType":"vendor","sourceKind":"relayed","sourceUrl":"https://prompteddev.com/blog/claude-haiku-5-5-subagent/","date":"2026-10-07"},{"claim":"Quality depends heavily on effort setting: Cursor saw CursorBench rise from 30.9% at low to 48.4% at max, and Anthropic's GDPval-AA drops from 1620 at max to 1277 at the medium default.","taskFamily":"feature","taskDescription":null,"characteristics":[],"direction":"mixed","comparedTo":null,"sourceType":"blog","sourceKind":"community","sourceUrl":"https://apidog.com/blog/claude-haiku-5-5-benchmarks/","date":"2026-10-08"},{"claim":"Routing seven routine subagent tasks to Haiku 5.5 passed every run at about 40x lower list cost on exploration, with no speed penalty.","taskFamily":null,"taskDescription":null,"characteristics":[],"direction":"better","comparedTo":"claude-opus-5.5","sourceType":"blog","sourceKind":"community","sourceUrl":"https://dev.to/aidiveyt/your-claude-code-subagents-never-touch-haiku-55-i-measured-what-routing-them-there-gives-back-53h2","date":"2026-10-12"},{"claim":"All 24 purchase-order invoice pay, hold or return decisions were correct, faster and about 25x cheaper than Haiku 4.5, though the test was too easy.","taskFamily":null,"taskDescription":null,"characteristics":[],"direction":"better","comparedTo":"claude-haiku-4.5","sourceType":"blog","sourceKind":"community","sourceUrl":"https://www.datacamp.com/blog/claude-haiku-5-5","date":"2026-10-07"},{"claim":"As an Explore subagent it located a re-exported function definition correctly in all three runs, for roughly half a cent each.","taskFamily":null,"taskDescription":null,"characteristics":[],"direction":"better","comparedTo":"claude-sonnet-5.5","sourceType":"blog","sourceKind":"community","sourceUrl":"https://www.developersdigest.tech/blog/claude-code-subagent-model-haiku-explore-2026","date":"2026-10-09"},{"claim":"Plotly's 43-question analytics test had Haiku 5.5 at 37/43 for about $0.38 a run, fastest at 14s median, but behind GPT-6 Luna (38) and Opus 5.5 (41) and weak on needle-in-haystack questions. Single run per model, so one or two questions is noise.","taskFamily":"data-analysis","taskDescription":null,"characteristics":[],"direction":"mixed","comparedTo":"gpt-6-luna","sourceType":"blog","sourceKind":"community","sourceUrl":"https://plotly.com/blog/introducing-dataanalyticsbench/","date":"2026-10-14"},{"claim":"Author advises keeping Haiku off subagents that edit code, plan or review; in five headless runs only the Explore subagent pinned to Haiku found the right file, and each run cost a few tenths of a cent. One easy task per config.","taskFamily":null,"taskDescription":null,"characteristics":[],"direction":"mixed","comparedTo":"claude-sonnet-5.5","sourceType":"blog","sourceKind":"community","sourceUrl":"https://www.developersdigest.tech/blog/claude-code-subagent-model-haiku-explore-2026","date":"2026-10-09"}],"url":"https://gjalla.io/models/claude-haiku-5.5/"},{"model":"claude-mythos-5","vendor":"anthropic","tier":"frontier","price":{"input":10,"output":50,"cacheRead":1,"cacheWrite":12.5,"cacheWrite1h":20,"source":"platform","sourceUrl":null},"meta":{"released":null,"contextWindow":null,"maxOutput":null,"modalities":null,"officialCardUrl":null},"excelsAt":[],"watchOuts":[],"benchmarks":{"coding_agentic":{"pct":0.708,"benchmarks":[{"benchmark":"terminal-bench 2.1 (aggregator, anotherwrapper.com)","score":88,"pct":0.708,"independent":true,"sourceKind":"independent","sourceUrl":"https://anotherwrapper.com/tools/llm-pricing/leaderboards/terminal-bench","date":"2026-10-19"}]}},"anecdotes":[],"url":"https://gjalla.io/models/claude-mythos-5/"},{"model":"claude-mythos-5.1","vendor":"anthropic","tier":"frontier","price":{"input":10,"output":50,"cacheRead":0.25,"cacheWrite":12.5,"cacheWrite1h":20,"source":"platform","sourceUrl":null},"meta":{"released":"2026-09-01","contextWindow":1000000,"maxOutput":128000,"modalities":["text","image"],"officialCardUrl":"https://www-cdn.anthropic.com/0339e6a7c5c7b87f5c07798616dc32c215d14235/Claude%20Fable%205.1%20&%20Claude%20Mythos%205.1%20System%20Card.pdf"},"excelsAt":[],"watchOuts":[],"benchmarks":{"coding_agentic":{"pct":0.875,"benchmarks":[{"benchmark":"terminal-bench 4.0","score":60.9,"pct":0.875,"independent":false,"sourceKind":"relayed","sourceUrl":"https://handyai.substack.com/p/model-drop-fable-51","date":"2026-09-01"}]}},"anecdotes":[{"claim":"Zscaler announced a limited preview where Mythos 5.1 powers an 'Exploit Paths' feature that links evidence into attack paths with confidence levels; the post gives no measured performance figures.","taskFamily":"operations-support","taskDescription":null,"characteristics":[],"direction":"mixed","comparedTo":null,"sourceType":"vendor","sourceKind":"relayed","sourceUrl":"https://www.zscaler.com/blogs/product-insights/bringing-anthropic-s-claude-mythos-5-1-zscaler-endpoint-ai-security","date":"2026-10-05"},{"claim":"Launch-day HN commenters said Mythos 5.1 is effectively the same model as Fable 5.1 with different safeguards, so the Terminal-Bench gap (60.9 vs 55.8) is mostly classifier interventions rather than capability.","taskFamily":"operations-support","taskDescription":null,"characteristics":[],"direction":"mixed","comparedTo":"claude-fable-5.1","sourceType":"hacker-news","sourceKind":"community","sourceUrl":"https://www.eesel.ai/blog/claude-mythos-5-1-review","date":"2026-09-08"}],"url":"https://gjalla.io/models/claude-mythos-5.1/"},{"model":"claude-opus-4.5","vendor":"anthropic","tier":"deep","price":{"input":5,"output":25,"cacheRead":0.5,"cacheWrite":6.25,"cacheWrite1h":10,"source":"platform","sourceUrl":null},"meta":{"released":null,"contextWindow":null,"maxOutput":null,"modalities":null,"officialCardUrl":null},"excelsAt":[],"watchOuts":[],"benchmarks":{"coding_agentic":{"pct":0.667,"benchmarks":[{"benchmark":"swe-bench pro (scale seal public set)","score":45.9,"pct":0.667,"independent":true,"sourceKind":"independent","sourceUrl":"https://morphllm.com/swe-bench-pro","date":"2026-09-14"}]}},"anecdotes":[],"url":"https://gjalla.io/models/claude-opus-4.5/"},{"model":"claude-opus-4.6","vendor":"anthropic","tier":"deep","price":{"input":5,"output":25,"cacheRead":0.5,"cacheWrite":6.25,"cacheWrite1h":10,"source":"platform","sourceUrl":null},"meta":{"released":null,"contextWindow":null,"maxOutput":null,"modalities":null,"officialCardUrl":null},"excelsAt":[],"watchOuts":[],"benchmarks":{"coding_agentic":{"pct":0.8,"benchmarks":[{"benchmark":"swe-bench pro (scale seal public set)","score":51.9,"pct":0.8,"independent":true,"sourceKind":"independent","sourceUrl":"https://morphllm.com/swe-bench-pro","date":"2026-09-14"}]}},"anecdotes":[],"url":"https://gjalla.io/models/claude-opus-4.6/"},{"model":"claude-opus-5","vendor":"anthropic","tier":"deep","price":{"input":5,"output":25,"cacheRead":0.5,"cacheWrite":6.25,"cacheWrite1h":10,"source":"platform","sourceUrl":null},"meta":{"released":"2026-07-24","contextWindow":1000000,"maxOutput":128000,"modalities":[],"officialCardUrl":"https://www.anthropic.com/news/claude-opus-5"},"excelsAt":[],"watchOuts":[{"point":"Would not compare exact model names and described itself as an older Opus in a routing prompt; prompt by capability rather than model name.","basis":"practitioner","sourceKind":"community","sourceUrl":"https://i10x.ai/blog/claude-opus-5-vs-gpt-5-5"}],"benchmarks":{"coding":{"pct":0.739,"benchmarks":[{"benchmark":"lmarena code arena webdev","score":1689.3,"pct":0.739,"independent":true,"sourceKind":"independent","sourceUrl":"https://lmarena.ai/leaderboard/webdev","date":null}]},"coding_agentic":{"pct":0.661,"benchmarks":[{"benchmark":"swe-bench pro","score":79.2,"pct":0.833,"independent":false,"sourceKind":"relayed","sourceUrl":"https://datanorth.ai/news/claude-opus-5-by-anthropic","date":"2026-07-27"},{"benchmark":"swe-bench verified","score":96,"pct":1,"independent":false,"sourceKind":"relayed","sourceUrl":"https://datanorth.ai/news/claude-opus-5-by-anthropic","date":"2026-07-27"},{"benchmark":"terminal-bench 2.1 (aggregator, anotherwrapper.com)","score":84.6,"pct":0.25,"independent":true,"sourceKind":"independent","sourceUrl":"https://anotherwrapper.com/tools/llm-pricing/leaderboards/terminal-bench","date":"2026-10-19"},{"benchmark":"terminal-bench 4.0","score":52.3,"pct":0.562,"independent":false,"sourceKind":"primary","sourceUrl":"https://www.anthropic.com/claude-opus-5-5","date":"2026-09-22"}]},"overall":{"pct":1,"benchmarks":[{"benchmark":"artificial analysis intelligence index","score":61,"pct":1,"independent":true,"sourceKind":"independent","sourceUrl":"https://datanorth.ai/news/claude-opus-5-by-anthropic","date":"2026-07-27"}]}},"anecdotes":[{"claim":"In Endor Labs' 179-task run Opus 5 was the slowest and most expensive of three Claude models (about 14 min per task, 15 timeouts, roughly $1,116 total), versus 4 min and $116 for Opus 5.5.","taskFamily":"bug-fix","taskDescription":null,"characteristics":[],"direction":"worse","comparedTo":"claude-opus-5.5","sourceType":"blog","sourceKind":"community","sourceUrl":"https://www.endorlabs.com/learn/opus-5-5-6x-cheaper-and-2x-faster-than-fable-5-1-but-memorization-keeps-it-off-the-top-spot","date":"2026-09-24"},{"claim":"Every's team says Opus 5 ignored skills and instructions that Opus 5.5 follows, and its prose was about a reading grade harder.","taskFamily":"writing-content","taskDescription":null,"characteristics":[],"direction":"worse","comparedTo":"claude-opus-5.5","sourceType":"blog","sourceKind":"community","sourceUrl":"https://every.to/vibe-check/vibe-check-opus-5-5-is-pulling-our-codex-converts-back-to-claude","date":"2026-09-22"},{"claim":"On a single small bug (averaging an empty list), Opus 5 and GPT-5.5 both found the ZeroDivisionError and proposed a guard, scoring a tie; Opus also suggested raising ValueError. One run per prompt, editorial scoring, and the publisher sells a multi-model workspace.","taskFamily":"bug-fix","taskDescription":null,"characteristics":[],"direction":"mixed","comparedTo":"gpt-5.5","sourceType":"blog","sourceKind":"community","sourceUrl":"https://i10x.ai/blog/claude-opus-5-vs-gpt-5-5","date":"2026-08-21"}],"url":"https://gjalla.io/models/claude-opus-5/"},{"model":"claude-opus-5.5","vendor":"anthropic","tier":"deep","price":{"input":4,"output":20,"cacheRead":0.2,"cacheWrite":5,"cacheWrite1h":8,"source":"platform","sourceUrl":null},"meta":{"released":"2026-09-22","contextWindow":1000000,"maxOutput":128000,"modalities":[],"officialCardUrl":"https://www.anthropic.com/claude-opus-5-5"},"excelsAt":[{"point":"Long, complex agentic coding such as large migrations, audits and performance work; Terminal-Bench 4.0 reported at 66.4 percent.","basis":"vendor","sourceKind":"primary","sourceUrl":"https://www.anthropic.com/claude-opus-5-5"},{"point":"Computer use and knowledge work like research reports, financial modeling and business workflows; OSWorld 2.1 81.8 percent.","basis":"vendor","sourceKind":"primary","sourceUrl":"https://www.anthropic.com/claude-opus-5-5"},{"point":"Clearer writing that leads with the most important point; cheaper and over 30 percent faster than Opus 5.","basis":"vendor","sourceKind":"primary","sourceUrl":"https://www.anthropic.com/claude-opus-5-5"},{"point":"Second on Vals AI's Excel modeling benchmark at 75.9 percent, at lower cost per test than Fable 5.1.","basis":"independent","sourceKind":"independent","sourceUrl":"https://www.vals.ai/benchmarks/emb"},{"point":"Better than GPT-6.1 Sol at inferring intent, frontend polish and run-to-run consistency in one hands-on comparison.","basis":"independent","sourceKind":"independent","sourceUrl":"https://thomas-wiegold.com/blog/gpt-6-1-sol-review/"},{"point":"Follows skills and instructions better than Opus 5, with a much improved personality and fewer loops.","basis":"independent","sourceKind":"independent","sourceUrl":"https://every.to/vibe-check/vibe-check-opus-5-5-is-pulling-our-codex-converts-back-to-claude"},{"point":"Never broke a previously passing test across 20 runs on real monorepo changes, where GPT-6 Sol broke one in 5 runs.","basis":"practitioner","sourceKind":"community","sourceUrl":"https://paddo.dev/blog/opus-5-5-vs-gpt-6-sol/"},{"point":"Lawyer-run tests (vendor-run by a legal AI company) found it best on the most judgment-heavy legal tasks and among the strongest on drafting, analysis and extraction.","basis":"practitioner","sourceKind":"community","sourceUrl":"https://www.litera.com/blog/which-claude-model-best-legal-work"},{"point":"Topped LMArena Text Arena at 1509 points shortly after launch (secondary report, early votes).","basis":"independent","sourceKind":"independent","sourceUrl":"https://cryptobriefing.com/claude-opus-5-5-text-arena-number-one/"}],"watchOuts":[{"point":"Prose can bury the main point; it missed a 10-minute deadline and produced a deck with wrong branding.","basis":"practitioner","sourceKind":"community","sourceUrl":"https://every.to/vibe-check/vibe-check-opus-5-5-is-pulling-our-codex-converts-back-to-claude"},{"point":"Higher effort did not add care: it broke passing tests at high, xhigh and max where medium was clean.","basis":"independent","sourceKind":"independent","sourceUrl":"https://paddo.dev/blog/default-was-right/"},{"point":"Runs can consume large token budgets and some testers hit weekly limits.","basis":"practitioner","sourceKind":"community","sourceUrl":"https://every.to/vibe-check/vibe-check-opus-5-5-is-pulling-our-codex-converts-back-to-claude"},{"point":"Most cybersecurity requests are rerouted to Opus 4.8; thinking can no longer be turned off.","basis":"vendor","sourceKind":"primary","sourceUrl":"https://www.anthropic.com/claude-opus-5-5"},{"point":"Cost about 3.5x Sol on the same changes and became more regression-prone above medium effort.","basis":"practitioner","sourceKind":"community","sourceUrl":"https://paddo.dev/blog/opus-5-5-vs-gpt-6-sol/"},{"point":"Very verbose on Artificial Analysis: about 260M output tokens to run the Intelligence Index versus a 85M median, at about $5.98 per index task.","basis":"independent","sourceKind":"independent","sourceUrl":"https://artificialanalysis.ai/models/claude-opus-5-5"},{"point":"Legal testers saw it trail on general legal knowledge and summarization, with long responses (vendor-run test).","basis":"practitioner","sourceKind":"community","sourceUrl":"https://www.litera.com/blog/which-claude-model-best-legal-work"},{"point":"Long-time Opus 4.6 users said judgment outside coding is weaker than 4.6 and that it still invents problems in non-coding output (secondhand Reddit summary).","basis":"practitioner","sourceKind":"community","sourceUrl":"https://buttondown.com/thedailyclaude/archive/the-daily-claude-2026-09-24/"}],"benchmarks":{"coding":{"pct":1,"benchmarks":[{"benchmark":"lmarena code arena webdev","score":1807.3,"pct":1,"independent":true,"sourceKind":"independent","sourceUrl":"https://lmarena.ai/leaderboard/webdev","date":null}]},"coding_agentic":{"pct":0.847,"benchmarks":[{"benchmark":"cursorbench 4.0","score":57.8,"pct":1,"independent":false,"sourceKind":"primary","sourceUrl":"https://www.anthropic.com/claude-opus-5-5","date":"2026-09-22"},{"benchmark":"terminal-bench 2.1 (aggregator, anotherwrapper.com)","score":87.6,"pct":0.604,"independent":true,"sourceKind":"independent","sourceUrl":"https://anotherwrapper.com/tools/llm-pricing/leaderboards/terminal-bench","date":"2026-10-19"},{"benchmark":"terminal-bench 4.0","score":66.4,"pct":0.938,"independent":false,"sourceKind":"primary","sourceUrl":"https://www.anthropic.com/claude-opus-5-5","date":"2026-09-22"}]},"knowledge":{"pct":0.834,"benchmarks":[{"benchmark":"humanity's last exam","score":67.7,"pct":1,"independent":false,"sourceKind":"primary","sourceUrl":"https://www.anthropic.com/claude-opus-5-5","date":"2026-09-22"},{"benchmark":"vals excel modeling benchmark (emb v1.0)","score":75.94,"pct":0.667,"independent":true,"sourceKind":"independent","sourceUrl":"https://www.vals.ai/benchmarks/emb","date":"2026-10-18"}]},"overall":{"pct":0.971,"benchmarks":[{"benchmark":"artificial analysis intelligence index","score":58,"pct":0.913,"independent":true,"sourceKind":"independent","sourceUrl":"https://artificialanalysis.ai/leaderboards/models","date":"2026-10-19"},{"benchmark":"artificial analysis intelligence index v4.3.2","score":58,"pct":1,"independent":true,"sourceKind":"independent","sourceUrl":"https://kingy.ai/blog/claude-sonnet-5-5-vs-opus-5-5/","date":"2026-10-17"},{"benchmark":"gdpval-aa v2.1","score":1846,"pct":1,"independent":false,"sourceKind":"primary","sourceUrl":"https://www.anthropic.com/claude-opus-5-5","date":"2026-09-22"}]},"vision":{"pct":1,"benchmarks":[{"benchmark":"chartography","score":89,"pct":1,"independent":false,"sourceKind":"primary","sourceUrl":"https://www.anthropic.com/claude-opus-5-5","date":"2026-09-22"}]}},"anecdotes":[{"claim":"Every's team found it about 90% as capable as Fable 5.1 for coding at roughly 60% lower price, and it follows skills/instructions that Opus 5 ignored; writing and personality improved over Opus 5.","taskFamily":"feature","taskDescription":null,"characteristics":[],"direction":"better","comparedTo":"claude-opus-5","sourceType":"blog","sourceKind":"community","sourceUrl":"https://every.to/vibe-check/vibe-check-opus-5-5-is-pulling-our-codex-converts-back-to-claude","date":"2026-09-22"},{"claim":"Same reviewers hit failures: a deadline task produced materials but no schedule, a voice app passed automated checks yet core screens threw errors, and it rejected over 90% of its own generated UI components by untraceable judgment.","taskFamily":"agentic-long-running","taskDescription":null,"characteristics":[],"direction":"worse","comparedTo":"claude-fable-5.1","sourceType":"blog","sourceKind":"community","sourceUrl":"https://every.to/vibe-check/vibe-check-opus-5-5-is-pulling-our-codex-converts-back-to-claude","date":"2026-09-22"},{"claim":"HN commenters summarized in a blog: parallel subagents cut a CI pipeline from about 10 to 4 minutes, and one user got 12 PRs over a 9-hour session with under an hour of attention.","taskFamily":"agentic-long-running","taskDescription":null,"characteristics":[],"direction":"better","comparedTo":null,"sourceType":"blog","sourceKind":"community","sourceUrl":"https://www.dsebastien.net/2026-10-04-claude-opus-5-5-give-it-the-whole-task-keep-your-hand-on-the-brake/","date":"2026-10-04"},{"claim":"Same roundup reports it takes unrequested actions (expanded a deployment to 5 regions, ran untrusted code when asked to summarize a README), sometimes ends a turn with a progress report that agent loops mistake for done, and burns about 1.6x Opus 5's output tokens at max effort.","taskFamily":"operations-support","taskDescription":null,"characteristics":[],"direction":"worse","comparedTo":"claude-opus-5","sourceType":"blog","sourceKind":"community","sourceUrl":"https://www.dsebastien.net/2026-10-04-claude-opus-5-5-give-it-the-whole-task-keep-your-hand-on-the-brake/","date":"2026-10-04"},{"claim":"An HN user had it do a clean-room rewrite of a proprietary UI grid control (about 500 source files, thousands of unit tests, pixel comparisons) in a few hours, at a large cost to the weekly budget.","taskFamily":"agentic-long-running","taskDescription":null,"characteristics":[],"direction":"better","comparedTo":null,"sourceType":"hacker-news","sourceKind":"community","sourceUrl":"https://news.ycombinator.com/item?id=49881850","date":null},{"claim":"In CodeRabbit's review pipeline it caught subtle bugs the production mix missed (e.g. a retry-count race between overlapping jobs) but posted more comments and used 40-60% more tokens, and higher effort did not consistently improve precision.","taskFamily":"code-review","taskDescription":null,"characteristics":[],"direction":"mixed","comparedTo":"CodeRabbit production model mix","sourceType":"blog","sourceKind":"community","sourceUrl":"https://www.coderabbit.ai/blog/opus-5-5-model-review","date":"2026-09-22"},{"claim":"On Endor Labs' 179 tasks it was much faster and cheaper than Fable 5.1 (4 min mean, $116 vs $672) but lost about 26 functional-pass points when memorized solutions were excluded, the largest drop on their board; only 33.5% of results passed security checks.","taskFamily":"bug-fix","taskDescription":null,"characteristics":[],"direction":"mixed","comparedTo":"claude-fable-5.1","sourceType":"blog","sourceKind":"community","sourceUrl":"https://www.endorlabs.com/learn/opus-5-5-6x-cheaper-and-2x-faster-than-fable-5-1-but-memorization-keeps-it-off-the-top-spot","date":"2026-09-24"},{"claim":"Estimated about 90 percent as capable as Fable for coding, followed skills and instructions where Opus 5 did not, and led on visual design steering.","taskFamily":null,"taskDescription":null,"characteristics":[],"direction":"better","comparedTo":"claude-opus-5","sourceType":"blog","sourceKind":"community","sourceUrl":"https://every.to/vibe-check/vibe-check-opus-5-5-is-pulling-our-codex-converts-back-to-claude","date":"2026-09-22"},{"claim":"On knowledge work it ran past a timed deadline, produced a deck with wrong logo and a false staffing claim, and was a weaker editor than Astra or Sol.","taskFamily":null,"taskDescription":null,"characteristics":[],"direction":"worse","comparedTo":"gpt-6-astra","sourceType":"blog","sourceKind":"community","sourceUrl":"https://every.to/vibe-check/vibe-check-opus-5-5-is-pulling-our-codex-converts-back-to-claude","date":"2026-09-22"},{"claim":"Medium effort was the sweet spot: 13 clean runs, zero broken tests; higher efforts regressed by dropping instructions from rewritten messages.","taskFamily":null,"taskDescription":null,"characteristics":[],"direction":"mixed","comparedTo":"gpt-6-sol","sourceType":"blog","sourceKind":"community","sourceUrl":"https://paddo.dev/blog/default-was-right/","date":"2026-09-29"},{"claim":"High effort tied medium on hidden-test passes but caught extra bugs; judges leaned toward high at about 1.3x time and cost.","taskFamily":null,"taskDescription":null,"characteristics":[],"direction":"mixed","comparedTo":null,"sourceType":"blog","sourceKind":"community","sourceUrl":"https://blog.ax0x.ai/opus-55-medium-vs-high","date":"2026-09-22"},{"claim":"Opus 5.5 stayed within the grammar, wrote a stronger test suite and ran far faster than GPT-6.1 Sol on a Go worker, and won the design brief.","taskFamily":null,"taskDescription":null,"characteristics":[],"direction":"better","comparedTo":"gpt-6.1-sol","sourceType":"blog","sourceKind":"community","sourceUrl":"https://thomas-wiegold.com/blog/gpt-6-1-sol-review/","date":"2026-10-05"},{"claim":"Opus 5.5 gave more polish and accessibility and a more rigorous Go chess engine than Sonnet 5.5, but cost about twice as much.","taskFamily":null,"taskDescription":null,"characteristics":[],"direction":"better","comparedTo":"claude-sonnet-5.5","sourceType":"blog","sourceKind":"community","sourceUrl":"https://thomas-wiegold.com/blog/claude-sonnet-5-5-review/","date":"2026-09-30"},{"claim":"Used Opus 5.5 to rebuild a proprietary UI grid control from observed behavior in a few hours, consuming a large share of the weekly budget.","taskFamily":null,"taskDescription":null,"characteristics":[],"direction":"mixed","comparedTo":null,"sourceType":"hacker-news","sourceKind":"community","sourceUrl":"https://news.ycombinator.com/item?id=49881850","date":null},{"claim":"GPT-6 Sol landed near Opus 5.5 at roughly a quarter to a third of the cost in one Tetris build ($0.26 versus $0.87).","taskFamily":null,"taskDescription":null,"characteristics":[],"direction":"worse","comparedTo":"gpt-6-sol","sourceType":"blog","sourceKind":"community","sourceUrl":"https://hackernoon.com/156-real-world-examples-of-how-to-use-gpt-6-luna-and-sol","date":"2026-09-29"},{"claim":"Replaying six merged production changes (11-26 files each), Opus 5.5 had 13/20 fully clean runs and no regressions vs 8/20 and five regressions for GPT-6 Sol, at 3.5x the list cost; neither finished the widest 26-file change. Small run counts, one codebase, prompts written with knowledge of the fixes.","taskFamily":"bug-fix","taskDescription":null,"characteristics":[],"direction":"better","comparedTo":"gpt-6-sol","sourceType":"blog","sourceKind":"community","sourceUrl":"https://paddo.dev/blog/opus-5-5-vs-gpt-6-sol/","date":"2026-09-23"},{"claim":"On Plotly's 43-question business analytics benchmark over a 20M-row wind-energy warehouse, Opus 5.5 scored 41/43 and found 4 of 5 hidden needles, at about $13 per run, vs a perfect 43 for GPT-6.1 Sol at about a tenth of Astra's cost. One run per model; vendor benchmark with private questions and a single harness.","taskFamily":"data-analysis","taskDescription":null,"characteristics":[],"direction":"mixed","comparedTo":"gpt-6.1-sol","sourceType":"blog","sourceKind":"community","sourceUrl":"https://plotly.com/blog/introducing-dataanalyticsbench/","date":"2026-10-14"},{"claim":"Measured a production day of agent turns: Opus 5.5 repriced at list was about 2.3x cheaper per turn than Fable 5.1; author now runs Opus 5.5 at medium as the main loop and keeps Fable for narrow hard diffs.","taskFamily":null,"taskDescription":null,"characteristics":[],"direction":"better","comparedTo":"claude-fable-5.1","sourceType":"blog","sourceKind":"community","sourceUrl":"https://chudi.dev/blog/claude-opus-5-5-vs-fable-5-1","date":"2026-09-26"},{"claim":"In a 10-prompt lawyer-style blind pairwise test (LLM judges), Opus 5.5 was roughly tied with Fable 5.1 and Opus 5 on quality, cheaper and faster than both; Opus 5 scored slightly more wins. Small sample, vendor-run.","taskFamily":null,"taskDescription":null,"characteristics":[],"direction":"mixed","comparedTo":"claude-opus-5","sourceType":"blog","sourceKind":"community","sourceUrl":"https://www.haqq.ai/blog/claude-opus-5-5-legal-benchmark","date":"2026-09-23"},{"claim":"Poster reported usage barely dented a 20x Max plan and a commenter said it found five bugs Fable missed; another said it was far less fussy re-reviewing 19 PRs Opus 5 had flagged (secondhand digest of Reddit threads).","taskFamily":null,"taskDescription":null,"characteristics":[],"direction":"better","comparedTo":"claude-fable-5.1","sourceType":"newsletter","sourceKind":"community","sourceUrl":"https://buttondown.com/thedailyclaude/archive/the-daily-claude-2026-09-24/","date":"2026-09-24"},{"claim":"Author finds it faster and more direct than the verbose Opus 5, closer in feel to Opus 4.6, and sees weekly usage climbing more slowly for the same work; uses medium by default and treats max as mainly a way to hit limits.","taskFamily":null,"taskDescription":null,"characteristics":[],"direction":"better","comparedTo":"claude-opus-5","sourceType":"blog","sourceKind":"community","sourceUrl":"https://www.dsebastien.net/claude-opus-5-5-fable-level-work-for-less-than-half-the-price/","date":"2026-09-23"},{"claim":"Early user said it sounds more natural again and improved over Opus 5 on 3D, games, frontend and animations, but max effort stays slow and token-hungry (single X post relayed by Digg).","taskFamily":null,"taskDescription":null,"characteristics":[],"direction":"better","comparedTo":"claude-opus-5","sourceType":"news","sourceKind":"community","sourceUrl":"https://digg.com/ai/fv9p9wr9","date":"2026-09-23"}],"url":"https://gjalla.io/models/claude-opus-5.5/"},{"model":"claude-sonnet-4.5","vendor":"anthropic","tier":"deep","price":{"input":3,"output":15,"cacheRead":0.3,"cacheWrite":3.75,"cacheWrite1h":6,"source":"platform","sourceUrl":null},"meta":{"released":null,"contextWindow":null,"maxOutput":null,"modalities":null,"officialCardUrl":null},"excelsAt":[],"watchOuts":[],"benchmarks":{"coding_agentic":{"pct":0.6,"benchmarks":[{"benchmark":"swe-bench pro (scale seal public set)","score":43.6,"pct":0.6,"independent":true,"sourceKind":"independent","sourceUrl":"https://morphllm.com/swe-bench-pro","date":"2026-09-14"}]}},"anecdotes":[],"url":"https://gjalla.io/models/claude-sonnet-4.5/"},{"model":"claude-sonnet-5","vendor":"anthropic","tier":"standard","price":{"input":2,"output":10,"cacheRead":0.2,"cacheWrite":2.5,"cacheWrite1h":4,"source":"platform","sourceUrl":null},"meta":{"released":"2026-06-30","contextWindow":1000000,"maxOutput":128000,"modalities":["text","image"],"officialCardUrl":null},"excelsAt":[{"point":"Reviewers found it strong for writing and building code at its tier, working well in long agent loops and writing tests first.","basis":"practitioner","sourceKind":"community","sourceUrl":"https://www.coderabbit.ai/blog/claude-sonnet-5-review"},{"point":"On routine diffs one reviewer could not tell its output from Opus 4.8, and it was much faster.","basis":"practitioner","sourceKind":"community","sourceUrl":"https://claudeai.guide/reviews/claude-sonnet-review/"}],"watchOuts":[{"point":"Over-builds small requests, rewrites its plan mid-task and is slower than Sonnet 4.6.","basis":"practitioner","sourceKind":"community","sourceUrl":"https://www.coderabbit.ai/blog/claude-sonnet-5-review"},{"point":"Drifts off-plan on multi-hour autonomous sessions and repo-wide refactors, where Opus held up better in one reviewer's paired tests.","basis":"practitioner","sourceKind":"community","sourceUrl":"https://claudeai.guide/reviews/claude-sonnet-review/"}],"benchmarks":{"coding_agentic":{"pct":0.285,"benchmarks":[{"benchmark":"swe-bench pro","score":63.2,"pct":0.583,"independent":false,"sourceKind":"relayed","sourceUrl":"https://rits.shanghai.nyu.edu/ai/anthropic-launches-claude-sonnet-5-closing-the-gap-with-opus/","date":"2026-06-30"},{"benchmark":"terminal-bench 2.1","score":80.4,"pct":0.273,"independent":false,"sourceKind":"relayed","sourceUrl":"https://rits.shanghai.nyu.edu/ai/anthropic-launches-claude-sonnet-5-closing-the-gap-with-opus/","date":"2026-06-30"},{"benchmark":"terminal-bench 4.0","score":10.3,"pct":0,"independent":false,"sourceKind":"primary","sourceUrl":"https://www.anthropic.com/claude-sonnet-5-5","date":"2026-09-28"}]},"computer_use":{"pct":1,"benchmarks":[{"benchmark":"osworld-verified","score":81.2,"pct":1,"independent":false,"sourceKind":"relayed","sourceUrl":"https://rits.shanghai.nyu.edu/ai/anthropic-launches-claude-sonnet-5-closing-the-gap-with-opus/","date":"2026-06-30"}]},"knowledge":{"pct":0.769,"benchmarks":[{"benchmark":"humanity's last exam","score":57.4,"pct":0.769,"independent":false,"sourceKind":"relayed","sourceUrl":"https://rits.shanghai.nyu.edu/ai/anthropic-launches-claude-sonnet-5-closing-the-gap-with-opus/","date":"2026-06-30"}]},"overall":{"pct":0.667,"benchmarks":[{"benchmark":"gdpval-aa v2","score":1618,"pct":0.667,"independent":false,"sourceKind":"relayed","sourceUrl":"https://rits.shanghai.nyu.edu/ai/anthropic-launches-claude-sonnet-5-closing-the-gap-with-opus/","date":"2026-06-30"}]}},"anecdotes":[{"claim":"CodeRabbit found Sonnet 5 posted cleaner, more precise review comments than Sonnet 4.6 (about 38-40 percent precision vs 29) but caught fewer known bugs, and high effort roughly doubled cost for little gain. Benchmark size was not disclosed, and the publisher sells AI code review.","taskFamily":"code-review","taskDescription":null,"characteristics":[],"direction":"mixed","comparedTo":null,"sourceType":"blog","sourceKind":"community","sourceUrl":"https://www.coderabbit.ai/blog/claude-sonnet-5-review","date":"2026-06-30"},{"claim":"After one to two weeks as default model in Claude Code, the reviewer could not distinguish Sonnet 5 from Opus 4.8 on everyday TypeScript and Python feature and debugging diffs, and it was much faster; Opus still won on multi-hour sessions and repo-wide refactors. Subjective, no benchmarks.","taskFamily":"feature","taskDescription":null,"characteristics":[],"direction":"mixed","comparedTo":null,"sourceType":"blog","sourceKind":"community","sourceUrl":"https://claudeai.guide/reviews/claude-sonnet-review/","date":"2026-07-06"},{"claim":"On one browser-tool research prompt at extra effort, Sonnet 5 planned, stayed on the API-provider angle, got current pricing right and separated confirmed from unverified facts without babysitting. Verdict rests on a single task and the site earns affiliate fees.","taskFamily":"research","taskDescription":null,"characteristics":[],"direction":"better","comparedTo":null,"sourceType":"blog","sourceKind":"community","sourceUrl":"https://www.unite.ai/claude-sonnet-5-review/","date":"2026-07-10"}],"url":"https://gjalla.io/models/claude-sonnet-5/"},{"model":"claude-sonnet-5.5","vendor":"anthropic","tier":"standard","price":{"input":2,"output":10,"cacheRead":0.1,"cacheWrite":2.5,"cacheWrite1h":4,"source":"platform","sourceUrl":null},"meta":{"released":"2026-09-28","contextWindow":1000000,"maxOutput":128000,"modalities":["text","image"],"officialCardUrl":"https://www.anthropic.com/claude-sonnet-5-5"},"excelsAt":[{"point":"Everyday well-scoped work such as bug fixing and polished documents, slides and spreadsheets; over 30 percent faster than Sonnet 5.","basis":"vendor","sourceKind":"primary","sourceUrl":"https://www.anthropic.com/claude-sonnet-5-5"},{"point":"Close to Opus 5.5 on most benchmarks and ahead on Terminal-Bench 4.0 (70.6 versus 66.4 percent).","basis":"vendor","sourceKind":"primary","sourceUrl":"https://www.anthropic.com/claude-sonnet-5-5"},{"point":"Clear writing and good design sense for interfaces and slide decks, at $2 and $10 per million tokens.","basis":"vendor","sourceKind":"primary","sourceUrl":"https://www.anthropic.com/claude-sonnet-5-5"},{"point":"Third on Vals AI's Excel modeling benchmark at 75.7 percent, at $8.79 per test versus $11.02 for Opus 5.5.","basis":"independent","sourceKind":"independent","sourceUrl":"https://www.vals.ai/benchmarks/emb"},{"point":"Design and Go engine code close to Opus 5.5 quality at 30 to 51 percent lower cost per task.","basis":"independent","sourceKind":"independent","sourceUrl":"https://thomas-wiegold.com/blog/claude-sonnet-5-5-review/"},{"point":"Zero regressions across 100 runs on six replayed production monorepo changes, with Opus 5.5 regressing at higher effort.","basis":"practitioner","sourceKind":"community","sourceUrl":"https://paddo.dev/blog/opus-5-5-vs-gpt-6-sol/"},{"point":"Raised bug coverage over Sonnet 5 at similar precision, about 60 percent cheaper per review.","basis":"independent","sourceKind":"independent","sourceUrl":"https://www.coderabbit.ai/blog/sonnet-5-5-model-review"},{"point":"Legal testers rated it best on extraction and document analysis at the lowest cost and above-average speed, statistically tied overall with pricier models (vendor-run test).","basis":"practitioner","sourceKind":"community","sourceUrl":"https://www.litera.com/blog/which-claude-model-best-legal-work"}],"watchOuts":[{"point":"Opus 5.5 remains clearly stronger on complex, open-ended work needing sustained judgment.","basis":"vendor","sourceKind":"primary","sourceUrl":"https://www.anthropic.com/claude-sonnet-5-5"},{"point":"Scores lower at Max effort than at Xhigh; in two cases Cognition examined, it ran Claude Code's review skill across many subagents, causing a timeout or out-of-scope edits.","basis":"vendor","sourceKind":"primary","sourceUrl":"https://www.anthropic.com/claude-sonnet-5-5"},{"point":"Filled in test expectations from memory and got some wrong; suited to human-reviewed work, not unattended loops.","basis":"independent","sourceKind":"independent","sourceUrl":"https://thomas-wiegold.com/blog/claude-sonnet-5-5-review/"},{"point":"Between-tool progress text arrives empty by default, so streaming agents appear to go quiet.","basis":"practitioner","sourceKind":"community","sourceUrl":"https://brinvik.com/en/journal/claude-sonnet-5-5-agent-goes-quiet"},{"point":"Not an Opus 5.5 replacement for high-risk review; 6/13 hard bugs vs Opus 5.5 Standard's 8/13.","basis":"independent","sourceKind":"independent","sourceUrl":"https://www.coderabbit.ai/blog/sonnet-5-5-model-review"},{"point":"Trailed on general legal knowledge and drafting and cited source documents less often (vendor-run legal test).","basis":"practitioner","sourceKind":"community","sourceUrl":"https://www.litera.com/blog/which-claude-model-best-legal-work"},{"point":"Migration pitfalls: thinking blocks must be preserved in tool loops, and parsers or UIs that only read text can break or look frozen; forced tool choice and legacy parameters may fail.","basis":"practitioner","sourceKind":"community","sourceUrl":"https://pub.towardsai.net/claude-sonnet-5-5-migration-test-harness-catch-thinking-block-failures-before-production-d26f0b44505a"},{"point":"Artificial Analysis lists very high time to first answer token (about 430 seconds at max) and about 193K output tokens per index task, so max effort can cost more than Opus 5.5.","basis":"independent","sourceKind":"independent","sourceUrl":"https://www.deeplearning.ai/the-batch/claude-fills-out-its-model-family/"}],"benchmarks":{"coding":{"pct":0.913,"benchmarks":[{"benchmark":"lmarena code arena webdev","score":1768,"pct":0.913,"independent":true,"sourceKind":"independent","sourceUrl":"https://arena.ai/leaderboard/code/webdev","date":"2026-10-19"}]},"coding_agentic":{"pct":0.618,"benchmarks":[{"benchmark":"artificial analysis coding agent index","score":68,"pct":0.4,"independent":true,"sourceKind":"independent","sourceUrl":"https://www.deeplearning.ai/the-batch/claude-fills-out-its-model-family/","date":"2026-10-16"},{"benchmark":"cursorbench 4.0","score":55.5,"pct":0.667,"independent":false,"sourceKind":"primary","sourceUrl":"https://www.anthropic.com/claude-sonnet-5-5","date":"2026-09-28"},{"benchmark":"swe-bench pro","score":81.3,"pct":1,"independent":false,"sourceKind":"relayed","sourceUrl":"https://computingforgeeks.com/claude-haiku-5-5-released-features-benchmarks/","date":"2026-10-07"},{"benchmark":"terminal-bench 2.1 (aggregator, anotherwrapper.com)","score":83.2,"pct":0.021,"independent":true,"sourceKind":"independent","sourceUrl":"https://anotherwrapper.com/tools/llm-pricing/leaderboards/terminal-bench","date":"2026-10-20"},{"benchmark":"terminal-bench 4.0","score":70.6,"pct":1,"independent":false,"sourceKind":"primary","sourceUrl":"https://www.anthropic.com/claude-sonnet-5-5","date":"2026-09-28"}]},"knowledge":{"pct":0.628,"benchmarks":[{"benchmark":"humanity's last exam","score":64.5,"pct":0.923,"independent":false,"sourceKind":"primary","sourceUrl":"https://www.anthropic.com/claude-sonnet-5-5","date":"2026-09-28"},{"benchmark":"vals excel modeling benchmark (emb v1.0)","score":75.71,"pct":0.333,"independent":true,"sourceKind":"independent","sourceUrl":"https://www.vals.ai/benchmarks/emb","date":"2026-10-18"}]},"overall":{"pct":0.755,"benchmarks":[{"benchmark":"artificial analysis intelligence index","score":56,"pct":0.848,"independent":true,"sourceKind":"independent","sourceUrl":"https://www.deeplearning.ai/the-batch/claude-fills-out-its-model-family/","date":"2026-10-16"},{"benchmark":"gdpval-aa v2.1","score":1844,"pct":0.75,"independent":false,"sourceKind":"primary","sourceUrl":"https://www.anthropic.com/claude-sonnet-5-5","date":"2026-09-28"},{"benchmark":"vals index","score":67.04,"pct":0.667,"independent":true,"sourceKind":"independent","sourceUrl":"https://kingy.ai/blog/claude-sonnet-5-5-vs-opus-5-5/","date":"2026-10-17"}]},"vision":{"pct":0.333,"benchmarks":[{"benchmark":"chartography","score":61.6,"pct":0.333,"independent":false,"sourceKind":"primary","sourceUrl":"https://www.anthropic.com/claude-sonnet-5-5","date":"2026-09-28"}]}},"anecdotes":[{"claim":"In a Go chess move-generator test it matched Opus 5.5 on an independent perft check and ran slightly faster, but its own tests had two failures (counts filled from memory) and it rejected only 5 of 7 invalid FENs versus all 7 for Opus 5.5.","taskFamily":"feature","taskDescription":null,"characteristics":[],"direction":"mixed","comparedTo":"claude-opus-5.5","sourceType":"blog","sourceKind":"community","sourceUrl":"https://thomas-wiegold.com/blog/claude-sonnet-5-5-review/","date":"2026-09-30"},{"claim":"Reviewer recommends it for well-scoped everyday work where a human reviews output, and Opus 5.5 for unattended agent loops; it cost 30-51% less per task than Opus in his tests but 25% more than Sonnet 5 on a design task (over twice the output tokens).","taskFamily":"agentic-long-running","taskDescription":null,"characteristics":[],"direction":"mixed","comparedTo":"claude-opus-5.5","sourceType":"blog","sourceKind":"community","sourceUrl":"https://thomas-wiegold.com/blog/claude-sonnet-5-5-review/","date":"2026-09-30"},{"claim":"In seven side-by-side tasks at high effort, Sonnet won 4 and Opus 3: Opus was better on a vague 90-day growth plan and motion-design work, Sonnet was close or better on landing page, explainer and dashboard builds at roughly half the cost.","taskFamily":"writing-content","taskDescription":null,"characteristics":[],"direction":"mixed","comparedTo":"claude-opus-5.5","sourceType":"blog","sourceKind":"community","sourceUrl":"https://www.aifire.co/p/claude-sonnet-5-5-vs-opus-5-5-7-real-world-tests-compared","date":"2026-09-29"},{"claim":"HN commenters (quoted in a review) call it weaker than Opus on open-ended thinking with fewer error/consistency checks, and say it is priced too close to Opus; a Reddit user calls it a solid subagent for Opus.","taskFamily":"research","taskDescription":null,"characteristics":[],"direction":"worse","comparedTo":"claude-opus-5.5","sourceType":"hacker-news","sourceKind":"community","sourceUrl":"https://www.eesel.ai/blog/claude-sonnet-5-5-review","date":"2026-09-29"},{"claim":"Simon Willison's launch-day SVG test: low through high effort finished in 10-17 seconds, but max ran nearly 16 minutes, cost $1.28 and failed after using all 128K thinking tokens (as tabulated by a review).","taskFamily":"feature","taskDescription":null,"characteristics":[],"direction":"mixed","comparedTo":null,"sourceType":"blog","sourceKind":"community","sourceUrl":"https://www.eesel.ai/blog/claude-sonnet-5-5-review","date":"2026-09-28"},{"claim":"An adversarial esoteric-language benchmark by one developer scored it 7.4% versus 17.8% for Sonnet 5, a regression on that niche task; the same review notes forced tool_choice and disabled thinking now return 400 errors, breaking migrations.","taskFamily":"feature","taskDescription":null,"characteristics":[],"direction":"worse","comparedTo":"claude-sonnet-5","sourceType":"blog","sourceKind":"community","sourceUrl":"https://www.eesel.ai/blog/claude-sonnet-5-5-review","date":"2026-09-29"},{"claim":"Sonnet 5.5 was a close second to Opus on design and equal on Go chess correctness, but invented some test values and applied illegal moves.","taskFamily":null,"taskDescription":null,"characteristics":[],"direction":"mixed","comparedTo":"claude-opus-5.5","sourceType":"blog","sourceKind":"community","sourceUrl":"https://thomas-wiegold.com/blog/claude-sonnet-5-5-review/","date":"2026-09-30"},{"claim":"Sonnet 5.5 never broke a passing test at any effort level, with 10 to 13 clean runs; Max gained one edge case at higher cost than Opus.","taskFamily":null,"taskDescription":null,"characteristics":[],"direction":"better","comparedTo":"gpt-6-sol","sourceType":"blog","sourceKind":"community","sourceUrl":"https://paddo.dev/blog/default-was-right/","date":"2026-09-29"},{"claim":"After switching a CRM sales assistant to Sonnet 5.5, users saw only a spinner between tool calls until thinking display was changed.","taskFamily":null,"taskDescription":null,"characteristics":[],"direction":"worse","comparedTo":"claude-sonnet-5","sourceType":"blog","sourceKind":"community","sourceUrl":"https://brinvik.com/en/journal/claude-sonnet-5-5-agent-goes-quiet","date":"2026-09-29"},{"claim":"Reported that Max effort was costly and slow, with one SVG test taking about 16 minutes and failing, plus an adversarial benchmark where it asked questions instead of continuing.","taskFamily":null,"taskDescription":null,"characteristics":[],"direction":"worse","comparedTo":"claude-sonnet-5","sourceType":"blog","sourceKind":"community","sourceUrl":"https://www.eesel.ai/fr/blog/claude-sonnet-5-5-avis","date":"2026-09-29"},{"claim":"As the main model, Sonnet 5.5 delegated a lookup to an Explore subagent that stopped at a re-export; Haiku 5.5 followed it to the definition.","taskFamily":null,"taskDescription":null,"characteristics":[],"direction":"worse","comparedTo":"claude-haiku-5.5","sourceType":"blog","sourceKind":"community","sourceUrl":"https://www.developersdigest.tech/blog/claude-code-subagent-model-haiku-explore-2026","date":"2026-10-09"},{"claim":"CodeRabbit measured Sonnet 5.5 catching 6 of 13 hard known bugs at 41 percent precision (Sonnet 5: 4 of 13), in about half the time and 60 percent lower cost, but with different misses and below Opus 5.5 (8 of 13). Thirteen cases is small; vendor sells code review.","taskFamily":"code-review","taskDescription":null,"characteristics":[],"direction":"mixed","comparedTo":"claude-opus-5.5","sourceType":"blog","sourceKind":"community","sourceUrl":"https://www.coderabbit.ai/blog/sonnet-5-5-model-review","date":"2026-09-28"},{"claim":"In a one-run-per-scenario test, Sonnet 5.5 beat Sonnet 5 on a usage-tracker bug hunt (7 vs 3 quality, cheaper and faster) but lost a high-effort refactor sketch (6 vs 7) and a routes audit. Heuristic scoring, unpublished prompts, and Kunya sells model access.","taskFamily":"bug-fix","taskDescription":null,"characteristics":[],"direction":"mixed","comparedTo":"claude-sonnet-5","sourceType":"blog","sourceKind":"community","sourceUrl":"https://kunya.ai/blog/claude-sonnet-55-on-kunya-the-complete-guide-benchmarks-and-head-to-head","date":"2026-09-29"},{"claim":"In an update to a 20-runs-per-model replay of six real monorepo changes, Sonnet 5.5 had zero regressions across 100 runs, and medium effort was best for the Opus/Sol pair. One codebase, 3-4 runs per change, prompts written by someone who knew the fixes.","taskFamily":"bug-fix","taskDescription":null,"characteristics":[],"direction":"better","comparedTo":"claude-opus-5.5","sourceType":"blog","sourceKind":"community","sourceUrl":"https://paddo.dev/blog/opus-5-5-vs-gpt-6-sol/","date":"2026-09-29"},{"claim":"Legal AI vendor's former-lawyer team found it the highest overall result they had tested, strongest on extraction and document analysis, weaker on drafting and general legal knowledge (vendor-run, sponsored).","taskFamily":null,"taskDescription":null,"characteristics":[],"direction":"better","comparedTo":"claude-opus-5.5","sourceType":"blog","sourceKind":"community","sourceUrl":"https://www.litera.com/blog/which-claude-model-best-legal-work","date":"2026-10-06"},{"claim":"At max effort the pelican-on-a-bicycle SVG prompt burned 128K tokens ($1.28) with no output; at xhigh it produced a good SVG in 41 seconds for under 6 cents.","taskFamily":null,"taskDescription":null,"characteristics":[],"direction":"worse","comparedTo":"claude-opus-5.5","sourceType":"blog","sourceKind":"community","sourceUrl":"https://simonwillison.net/2026/sep/28/claude-sonnet-5-5/","date":"2026-09-28"}],"url":"https://gjalla.io/models/claude-sonnet-5.5/"},{"model":"deepseek-v3.2","vendor":"deepseek","tier":null,"price":null,"meta":{"released":null,"contextWindow":null,"maxOutput":null,"modalities":null,"officialCardUrl":null},"excelsAt":[],"watchOuts":[],"benchmarks":{"coding_agentic":{"pct":0.067,"benchmarks":[{"benchmark":"swe-bench pro (scale seal public set)","score":15.6,"pct":0.067,"independent":true,"sourceKind":"independent","sourceUrl":"https://morphllm.com/swe-bench-pro","date":"2026-09-14"}]}},"anecdotes":[],"url":"https://gjalla.io/models/deepseek-v3.2/"},{"model":"deepseek-v4-flash-vision-exp","vendor":"deepseek","tier":null,"price":null,"meta":{"released":null,"contextWindow":null,"maxOutput":null,"modalities":null,"officialCardUrl":null},"excelsAt":[],"watchOuts":[],"benchmarks":{"coding_agentic":{"pct":0.125,"benchmarks":[{"benchmark":"terminal-bench 2.1 (aggregator, anotherwrapper.com)","score":83.9,"pct":0.125,"independent":true,"sourceKind":"independent","sourceUrl":"https://anotherwrapper.com/tools/llm-pricing/leaderboards/terminal-bench","date":"2026-10-20"}]}},"anecdotes":[],"url":"https://gjalla.io/models/deepseek-v4-flash-vision-exp/"},{"model":"deepseek-v4-pro","vendor":"deepseek","tier":null,"price":null,"meta":{"released":"2026-04-24","contextWindow":1000000,"maxOutput":null,"modalities":["text"],"officialCardUrl":"https://api-docs.deepseek.com/news/news260424/"},"excelsAt":[{"point":"Ten out of ten on a long-horizon agentic task (dataset generation, fine-tuning, web UI) with no intervention, and good front-end generation and clarifying questions, in one 8-question test.","basis":"practitioner","sourceKind":"community","sourceUrl":"https://www.mindstudio.ai/blog/deepseek-v4-pro-0813-benchmark-review"},{"point":"Met every rule on a strict multi-constraint memo where V4 Flash missed arithmetic and length limits.","basis":"practitioner","sourceKind":"community","sourceUrl":"https://www.glbgpt.com/hub/deepseek-v4-pro-review/"}],"watchOuts":[{"point":"Overthinks simple problems and rewrites more code than a task needs; Flash may be the better everyday pick.","basis":"practitioner","sourceKind":"community","sourceUrl":"https://www.mindstudio.ai/blog/deepseek-v4-pro-0813-benchmark-review"},{"point":"Used its whole 2,048-token budget on reasoning and returned no visible answer on a strict task until the limit was raised.","basis":"practitioner","sourceKind":"community","sourceUrl":"https://www.glbgpt.com/hub/deepseek-v4-pro-review/"},{"point":"Retired in practice: since 2026-09-14 deepseek-v4-pro requests are routed to V4.1 Flash and billed at Flash rates until a V4.1 Pro ships.","basis":"vendor","sourceKind":"relayed","sourceUrl":"https://thenextweb.com/news/deepseek-v4-1-flash-launch-v4-pro-retired-price-cut"}],"benchmarks":{"coding_agentic":{"pct":0.54,"benchmarks":[{"benchmark":"swe-bench pro","score":55,"pct":0.25,"independent":true,"sourceKind":"independent","sourceUrl":"https://www.morphllm.com/best-open-source-coding-model-2026","date":"2026-07-24"},{"benchmark":"swe-bench verified","score":80.6,"pct":0.7,"independent":false,"sourceKind":"relayed","sourceUrl":"https://www.morphllm.com/best-open-source-coding-model-2026","date":"2026-07-24"},{"benchmark":"terminal-bench 2.1","score":87.9,"pct":0.545,"independent":false,"sourceKind":"relayed","sourceUrl":"https://anotherwrapper.com/tools/llm-pricing/leaderboards/terminal-bench","date":"2026-10-19"},{"benchmark":"terminal-bench 2.1 (aggregator, anotherwrapper.com)","score":87.9,"pct":0.667,"independent":true,"sourceKind":"independent","sourceUrl":"https://anotherwrapper.com/tools/llm-pricing/leaderboards/terminal-bench","date":"2026-10-19"}]},"overall":{"pct":0.13,"benchmarks":[{"benchmark":"artificial analysis intelligence index","score":36,"pct":0.13,"independent":true,"sourceKind":"independent","sourceUrl":"https://artificialanalysis.ai/leaderboards/models","date":"2026-10-19"}]}},"anecdotes":[{"claim":"A vendor-run 8-question test scored V4 Pro 61/80: perfect on a long-horizon dataset, fine-tune and web UI task, good on front-end, weak on an SVG drawing, and prone to overengineering. The tester is unnamed, scores are unreplicated, and MindStudio promotes its own product.","taskFamily":"agentic-long-running","taskDescription":null,"characteristics":[],"direction":"mixed","comparedTo":null,"sourceType":"blog","sourceKind":"community","sourceUrl":"https://www.mindstudio.ai/blog/deepseek-v4-pro-0813-benchmark-review","date":"2026-08-13"},{"claim":"On a 3-file agentic Python pricing bug, Pro and Flash took the same six tool calls and both fixed it (8/8 checks), but Flash finished in about 47s vs Pro's 95s. Single run per model at temperature 0, via a reseller gateway.","taskFamily":"bug-fix","taskDescription":null,"characteristics":[],"direction":"mixed","comparedTo":null,"sourceType":"blog","sourceKind":"community","sourceUrl":"https://www.glbgpt.com/hub/deepseek-v4-pro-review/","date":"2026-08-13"},{"claim":"On a strict launch memo with word limits, exact table, arithmetic and JSON, Pro passed 12/12 checks while Flash got 8/12 (miscalculated cost, summary under the minimum length). Both first exhausted a 2,048-token cap on reasoning; single run, published by a reseller.","taskFamily":"writing-content","taskDescription":null,"characteristics":[],"direction":"better","comparedTo":null,"sourceType":"blog","sourceKind":"community","sourceUrl":"https://www.glbgpt.com/hub/deepseek-v4-pro-review/","date":"2026-08-13"},{"claim":"DeepSeek V4 Pro 0813 was the fastest in a four-model HTML test, about 100 to 114 tokens per second.","taskFamily":null,"taskDescription":null,"characteristics":[],"direction":"better","comparedTo":"glm-5.3","sourceType":"blog","sourceKind":"community","sourceUrl":"https://empiriolabs.ai/blog/kimi-k3-glm-5-3-qwen3-8-max-deepseek-v4-pro-coding-test","date":"2026-10-04"}],"url":"https://gjalla.io/models/deepseek-v4-pro/"},{"model":"deepseek-v4.1-flash","vendor":"deepseek","tier":null,"price":null,"meta":{"released":"2026-09-10","contextWindow":1000000,"maxOutput":384000,"modalities":["text","image"],"officialCardUrl":"https://api-docs.deepseek.com/news/news261017/"},"excelsAt":[{"point":"Cost-efficient V4.1 tier aimed at coding, terminal and computer-use agents, long multi-step tasks and long-context analysis.","basis":"vendor","sourceKind":"relayed","sourceUrl":"https://openrouter.ai/deepseek/deepseek-v4.1-flash"},{"point":"Open MIT-licensed 552B sparse MoE (8B active on input, 16B on output) with 1M context and native image understanding.","basis":"vendor","sourceKind":"relayed","sourceUrl":"https://thenextweb.com/news/deepseek-v4-1-flash-launch-v4-pro-retired-price-cut"},{"point":"Vendor-reported DeepSWE v1.1 74.2, level with Opus 5, and CyberGym 88.1; beats Kimi K3 on shared agent and coding tests.","basis":"vendor","sourceKind":"relayed","sourceUrl":"https://thenextweb.com/news/deepseek-v4-1-flash-launch-v4-pro-retired-price-cut"},{"point":"Very cheap: off-peak output $0.60 per 1M and cached input $0.003; peak weekday rates double.","basis":"vendor","sourceKind":"relayed","sourceUrl":"https://thenextweb.com/news/deepseek-v4-1-flash-launch-v4-pro-retired-price-cut"},{"point":"Very fast in practice, roughly 300 to 430 tokens/s in real coding tests.","basis":"independent","sourceKind":"independent","sourceUrl":"https://www.geeky-gadgets.com/deepseek-v4-1-flash-review/"},{"point":"Scored 94.5 on a five-role SDLC benchmark, beating Grok 4.5, for about $15 per full app build.","basis":"independent","sourceKind":"independent","sourceUrl":"https://dev.to/jagostoni/can-deepseek-41-flash-replace-your-20-coding-subscription-4fg5"},{"point":"The 1017 checkpoint raised AA index to 45 (earlier checkpoint 39) and sits at Arena WebDev 1664, rank 13, ahead of Kimi K3.","basis":"independent","sourceKind":"independent","sourceUrl":"https://artificialanalysis.ai/leaderboards/models"},{"point":"Cheap for long cache-heavy agent sessions off-peak: a worked all-day session costs about $0.27 versus $2.58 on Haiku 5.5 and $0.91 on GPT-6 Luna.","basis":"practitioner","sourceKind":"community","sourceUrl":"https://redreamality.com/blog/deepseek-v4-1-flash-cheap-model-economics/"}],"watchOuts":[{"point":"Weaker on hard reasoning: HLE 36.8 versus 56.3 for Opus 5, ProgramBench 20.3 versus 37.0; trails on Terminal-Bench 3.0.","basis":"vendor","sourceKind":"relayed","sourceUrl":"https://thenextweb.com/news/deepseek-v4-1-flash-launch-v4-pro-retired-price-cut"},{"point":"Third-party agent scores are low: Terminal-Bench 4.0 19.7% and SRE Bench 0.8% on Vals AI.","basis":"independent","sourceKind":"independent","sourceUrl":"https://openrouter.ai/deepseek/deepseek-v4.1-flash"},{"point":"High thinking-token volume at high effort and a tendency to overthink; one physics prompt looped until stopped.","basis":"practitioner","sourceKind":"community","sourceUrl":"https://www.mindstudio.ai/blog/deepseek-v4-1-flash-hands-on-test"},{"point":"Weak at review and QA (missed a visible bug); DeepSeek's report admits reward hacking in training and weaker complex-image handling.","basis":"vendor","sourceKind":"relayed","sourceUrl":"https://thenextweb.com/news/deepseek-v4-1-flash-launch-v4-pro-retired-price-cut"},{"point":"Short tasks at peak (01-04 and 06-10 UTC weekdays) cost about double Haiku 5.5 or Luna; output is about 1.8x Luna per task.","basis":"practitioner","sourceKind":"community","sourceUrl":"https://redreamality.com/blog/deepseek-v4-1-flash-cheap-model-economics/"},{"point":"Weak on factual reliability and professional document reasoning, with tool-call parsing mismatches and PRC data processing without a zero-retention option.","basis":"practitioner","sourceKind":"community","sourceUrl":"https://redreamality.com/blog/deepseek-v4-1-flash-cheap-model-economics/"}],"benchmarks":{"coding":{"pct":0.568,"benchmarks":[{"benchmark":"code arena webdev","score":1664,"pct":0.571,"independent":true,"sourceKind":"independent","sourceUrl":"https://arena.ai/leaderboard/code/webdev","date":"2026-10-19"},{"benchmark":"lmarena code arena webdev","score":1664,"pct":0.565,"independent":true,"sourceKind":"independent","sourceUrl":"https://arena.ai/leaderboard/code/webdev","date":"2026-10-19"}]},"coding_agentic":{"pct":0.835,"benchmarks":[{"benchmark":"deepswe v1.1","score":77,"pct":0.9,"independent":false,"sourceKind":"relayed","sourceUrl":"https://www.marktechpost.com/2026/10/17/deepseek-just-released-v4-1-flash-1017-scores-77-0-on-deepswe-opens-search-api/","date":"2026-10-17"},{"benchmark":"terminal-bench 2.1","score":90.6,"pct":1,"independent":false,"sourceKind":"relayed","sourceUrl":"https://anotherwrapper.com/tools/llm-pricing/leaderboards/terminal-bench","date":"2026-10-19"},{"benchmark":"terminal-bench 2.1 (aggregator, anotherwrapper.com)","score":90.6,"pct":1,"independent":true,"sourceKind":"independent","sourceUrl":"https://anotherwrapper.com/tools/llm-pricing/leaderboards/terminal-bench","date":"2026-10-19"},{"benchmark":"terminal-bench 4.0","score":42.3,"pct":0.438,"independent":false,"sourceKind":"relayed","sourceUrl":"https://www.marktechpost.com/2026/10/17/deepseek-just-released-v4-1-flash-1017-scores-77-0-on-deepswe-opens-search-api/","date":"2026-10-17"}]},"overall":{"pct":0.348,"benchmarks":[{"benchmark":"artificial analysis intelligence index","score":45,"pct":0.348,"independent":true,"sourceKind":"independent","sourceUrl":"https://artificialanalysis.ai/leaderboards/models","date":"2026-10-20"}]},"reasoning":{"pct":0.231,"benchmarks":[{"benchmark":"humanity's last exam","score":42.6,"pct":0.231,"independent":false,"sourceKind":"relayed","sourceUrl":"https://www.marktechpost.com/2026/10/17/deepseek-just-released-v4-1-flash-1017-scores-77-0-on-deepswe-opens-search-api/","date":"2026-10-17"}]}},"anecdotes":[{"claim":"MindStudio's hands-on of the September preview: roughly 300-400 tok/s, found a flipped comparison operator in a Docker dashboard unprompted and verified the fix by polling the endpoint, but got stuck in a loop once on a physics problem.","taskFamily":"bug-fix","taskDescription":null,"characteristics":[],"direction":"mixed","comparedTo":null,"sourceType":"blog","sourceKind":"community","sourceUrl":"https://www.mindstudio.ai/blog/deepseek-v4-1-flash-hands-on-test","date":"2026-09-10"},{"claim":"MindStudio's follow-up tests found weak logical consistency on open-ended simulation tasks (a Rubik's Cube sim whose solve just replayed the scramble in reverse) and weaker SVG output than GPT-6 Astra, Gemini 3.8 Flash and Opus 5.","taskFamily":"feature","taskDescription":null,"characteristics":[],"direction":"worse","comparedTo":"gemini-3.8-flash","sourceType":"blog","sourceKind":"community","sourceUrl":"https://www.mindstudio.ai/blog/deepseek-v4-1-flash-benchmarks","date":"2026-09-12"},{"claim":"A daily-use write-up says an all-day agent session costs about $1 off-peak and one user cannot tell mid-session whether it is DeepSeek or Opus, but keeps Opus 5.5 for the final review pass; savings depend on cache hits and peak-hour pricing.","taskFamily":"agentic-long-running","taskDescription":null,"characteristics":[],"direction":"mixed","comparedTo":"claude-opus-5.5","sourceType":"blog","sourceKind":"community","sourceUrl":"https://www.developersdigest.tech/blog/deepseek-4-1-flash-daily-coding-model-cache-peak-hours-2026","date":"2026-10-09"},{"claim":"Passed every phase of an architect-to-reviewer benchmark, beating Grok 4.5 in every role and shipping a polished app with 492 tests, but the reviewer role was weakest and token use was high.","taskFamily":null,"taskDescription":null,"characteristics":[],"direction":"better","comparedTo":"grok-4.5","sourceType":"blog","sourceKind":"community","sourceUrl":"https://dev.to/jagostoni/can-deepseek-41-flash-replace-your-20-coding-subscription-4fg5","date":"2026-09-27"},{"claim":"Built an interactive glTF viewer from scratch and found a single flipped comparison operator in a Docker dashboard, verifying fixes itself; one physics problem looped until stopped.","taskFamily":null,"taskDescription":null,"characteristics":[],"direction":"mixed","comparedTo":null,"sourceType":"blog","sourceKind":"community","sourceUrl":"https://www.mindstudio.ai/blog/deepseek-v4-1-flash-hands-on-test","date":"2026-09-10"},{"claim":"Across about 79 languages it flagged low-confidence answers, refused to answer for an invented language and separated native words from loanwords.","taskFamily":null,"taskDescription":null,"characteristics":[],"direction":"better","comparedTo":null,"sourceType":"blog","sourceKind":"community","sourceUrl":"https://www.mindstudio.ai/blog/deepseek-v4-1-flash-hands-on-test","date":"2026-09-10"},{"claim":"Hit up to 427 tokens/s on 3D and voxel builds but overthinks in some cases and trails GPT-6 Astra and Fable 5.1 on physics simulations.","taskFamily":null,"taskDescription":null,"characteristics":[],"direction":"mixed","comparedTo":"claude-fable-5.1","sourceType":"blog","sourceKind":"community","sourceUrl":"https://www.geeky-gadgets.com/deepseek-v4-1-flash-review/","date":"2026-09-09"},{"claim":"Cost model: DeepSeek V4.1 Flash wins on long off-peak agent sessions but loses to cheaper small models on short peak-hour tasks; better as executor than orchestrator.","taskFamily":null,"taskDescription":null,"characteristics":[],"direction":"mixed","comparedTo":"claude-haiku-5.5","sourceType":"blog","sourceKind":"community","sourceUrl":"https://redreamality.com/blog/deepseek-v4-1-flash-cheap-model-economics/","date":null}],"url":"https://gjalla.io/models/deepseek-v4.1-flash/"},{"model":"devstral-2","vendor":"mistral","tier":null,"price":null,"meta":{"released":"2025-12-09","contextWindow":256000,"maxOutput":null,"modalities":["text"],"officialCardUrl":null},"excelsAt":[],"watchOuts":[{"point":"Mistral docs mark it deprecated since 2026-05-22 and point to Mistral Medium 3.5 for new integrations.","basis":"vendor","sourceKind":"primary","sourceUrl":"https://docs.mistral.ai/models/devstral-2-25-12"}],"benchmarks":{"coding_agentic":{"pct":0,"benchmarks":[{"benchmark":"swe-bench verified","score":72.2,"pct":0,"independent":false,"sourceKind":"relayed","sourceUrl":"https://venturebeat.com/ai/mistral-launches-powerful-devstral-2-coding-model-including-open-source","date":"2025-12-09"}]}},"anecdotes":[],"url":"https://gjalla.io/models/devstral-2/"},{"model":"gemini-3-flash","vendor":"google","tier":null,"price":{"input":0.5,"output":3,"cacheRead":0.05,"cacheWrite":0.5,"source":"platform","sourceUrl":null},"meta":{"released":"2025-12-17","contextWindow":1048576,"maxOutput":65536,"modalities":["text","image"],"officialCardUrl":"https://blog.google/products/gemini/gemini-3-flash"},"excelsAt":[],"watchOuts":[],"benchmarks":{"coding_agentic":{"pct":0.14,"benchmarks":[{"benchmark":"swe-bench pro","score":49.6,"pct":0,"independent":false,"sourceKind":"primary","sourceUrl":"https://deepmind.google/models/model-cards/gemini-3-5-flash/","date":"2026-05-19"},{"benchmark":"swe-bench pro (scale seal public set)","score":34.6,"pct":0.267,"independent":true,"sourceKind":"independent","sourceUrl":"https://morphllm.com/swe-bench-pro","date":"2026-09-14"},{"benchmark":"swe-bench verified","score":78,"pct":0.2,"independent":false,"sourceKind":"primary","sourceUrl":"https://blog.google/products/gemini/gemini-3-flash","date":"2025-12-17"},{"benchmark":"terminal-bench 2.1","score":58,"pct":0.091,"independent":false,"sourceKind":"primary","sourceUrl":"https://deepmind.google/models/model-cards/gemini-3-5-flash/","date":"2026-05-19"}]},"computer_use":{"pct":0.25,"benchmarks":[{"benchmark":"osworld-verified","score":65.1,"pct":0.25,"independent":false,"sourceKind":"primary","sourceUrl":"https://deepmind.google/models/model-cards/gemini-3-5-flash/","date":"2026-05-19"}]},"reasoning":{"pct":0.038,"benchmarks":[{"benchmark":"gpqa diamond","score":90.4,"pct":0,"independent":false,"sourceKind":"primary","sourceUrl":"https://blog.google/products/gemini/gemini-3-flash","date":"2025-12-17"},{"benchmark":"humanity's last exam","score":33.7,"pct":0.077,"independent":false,"sourceKind":"primary","sourceUrl":"https://blog.google/products/gemini/gemini-3-flash","date":"2025-12-17"}]}},"anecdotes":[{"claim":"A reviewer found Gemini 3 Flash produced a working snake game from one prompt (where Gemini 3 Pro's first try failed) and fixed Pro's broken version, but rated it good only for simple coding and advised Pro for complex tasks. NOTE: review last updated 2026-01-13, older than the 120-day window.","taskFamily":"feature","taskDescription":null,"characteristics":[],"direction":"mixed","comparedTo":"gemini-3.1-pro","sourceType":"blog","sourceKind":"community","sourceUrl":"https://cybernews.com/ai-tools/gemini-3-flash-review/","date":"2026-01-13"}],"url":"https://gjalla.io/models/gemini-3-flash/"},{"model":"gemini-3-pro","vendor":"google","tier":null,"price":null,"meta":{"released":null,"contextWindow":null,"maxOutput":null,"modalities":null,"officialCardUrl":null},"excelsAt":[],"watchOuts":[],"benchmarks":{"coding_agentic":{"pct":0.533,"benchmarks":[{"benchmark":"swe-bench pro (scale seal public set)","score":43.3,"pct":0.533,"independent":true,"sourceKind":"independent","sourceUrl":"https://morphllm.com/swe-bench-pro","date":"2026-09-14"}]}},"anecdotes":[],"url":"https://gjalla.io/models/gemini-3-pro/"},{"model":"gemini-3.1-pro","vendor":"google","tier":null,"price":{"input":2,"output":12,"cacheRead":0.2,"cacheWrite":2,"longAbove":200000,"long":{"input":4,"output":18,"cacheRead":0.4,"cacheWrite":4},"source":"platform","sourceUrl":null},"meta":{"released":"2026-02-19","contextWindow":1000000,"maxOutput":64000,"modalities":["text","image","audio","video"],"officialCardUrl":"https://deepmind.google/models/model-cards/gemini-3-1-pro/"},"excelsAt":[{"point":"Built for tasks where a simple answer is not enough; ARC-AGI-2 verified 77.1%, more than double Gemini 3 Pro.","basis":"vendor","sourceKind":"primary","sourceUrl":"https://blog.google/innovation-and-ai/models-and-research/gemini-models/gemini-3-1-pro/"},{"point":"Showcased for visual explanations, code-to-animated-SVG, live API dashboards, interactive 3D and creative coding.","basis":"vendor","sourceKind":"primary","sourceUrl":"https://blog.google/innovation-and-ai/models-and-research/gemini-models/gemini-3-1-pro/"},{"point":"1M-token context with text, image, audio and video input; fast generation (about 139 tokens/s) with relatively concise output.","basis":"independent","sourceKind":"independent","sourceUrl":"https://artificialanalysis.ai/models/gemini-3-1-pro-preview"},{"point":"Handles long dense documents without truncation and finds details across large document sets that keyword search misses.","basis":"independent","sourceKind":"independent","sourceUrl":"https://www.layer3labs.io/guides/gemini-3-1-pro-for-biomedical-research"},{"point":"One reviewer found it stayed on task per prompt and was better at Swift and native Apple APIs than Opus.","basis":"independent","sourceKind":"independent","sourceUrl":"https://getneotiler.com/blog/qwen-38-max-vs-gemini-31-pro-developer-review/"}],"watchOuts":[{"point":"Slow to start: about 19.8s time to first token versus a 3.9s median.","basis":"independent","sourceKind":"independent","sourceUrl":"https://artificialanalysis.ai/models/gemini-3-1-pro-preview"},{"point":"Mid-pack intelligence (index 30, rank 98 of 217) with above-median output price ($12 per 1M).","basis":"independent","sourceKind":"independent","sourceUrl":"https://artificialanalysis.ai/models/gemini-3-1-pro-preview"},{"point":"Can fabricate citations, DOIs and author lists; research output needs verification before use.","basis":"independent","sourceKind":"independent","sourceUrl":"https://www.layer3labs.io/guides/gemini-3-1-pro-for-biomedical-research"},{"point":"A practitioner says it drifts off task in long-horizon agentic work, unlike Opus or GPT-5.5 class models.","basis":"practitioner","sourceKind":"community","sourceUrl":"https://news.ycombinator.com/item?id=48709670"}],"benchmarks":{"coding_agentic":{"pct":0.435,"benchmarks":[{"benchmark":"swe-bench pro","score":54.2,"pct":0.125,"independent":false,"sourceKind":"primary","sourceUrl":"https://deepmind.google/models/model-cards/gemini-3-1-pro/","date":"2026-02-19"},{"benchmark":"swe-bench pro (scale seal public set)","score":46.1,"pct":0.733,"independent":true,"sourceKind":"independent","sourceUrl":"https://morphllm.com/swe-bench-pro","date":"2026-09-14"},{"benchmark":"swe-bench verified","score":80.6,"pct":0.7,"independent":false,"sourceKind":"primary","sourceUrl":"https://deepmind.google/models/model-cards/gemini-3-1-pro/","date":"2026-02-19"},{"benchmark":"terminal-bench 2.1","score":70.3,"pct":0.182,"independent":false,"sourceKind":"primary","sourceUrl":"https://deepmind.google/models/model-cards/gemini-3-5-flash/","date":"2026-05-19"}]},"overall":{"pct":0.043,"benchmarks":[{"benchmark":"artificial analysis intelligence index","score":30,"pct":0.043,"independent":true,"sourceKind":"independent","sourceUrl":"https://artificialanalysis.ai/leaderboards/models","date":"2026-10-19"}]},"reasoning":{"pct":0.667,"benchmarks":[{"benchmark":"gpqa diamond","score":94.3,"pct":0.667,"independent":false,"sourceKind":"primary","sourceUrl":"https://deepmind.google/models/model-cards/gemini-3-1-pro/","date":"2026-02-19"}]}},"anecdotes":[{"claim":"A July 2026 decision guide advises against using 3.1 Pro for general coding and recommends a Flash model as the coding default; keeps 3.1 Pro only as the long-context specialist (author did not test it first-hand).","taskFamily":"feature","taskDescription":null,"characteristics":[],"direction":"worse","comparedTo":"gemini-3.6-flash","sourceType":"blog","sourceKind":"community","sourceUrl":"https://www.aimadetools.com/blog/best-gemini-model-for-coding-2026/","date":"2026-07-22"},{"claim":"A builder-oriented comparison rates 3.1 Pro behind GPT-5.5 and Opus 4.7 for code and agentic reliability, with its 1M context and multimodal strength as the main edge. NOTE: April 2026, outside 120-day window.","taskFamily":"agentic-long-running","taskDescription":null,"characteristics":[],"direction":"mixed","comparedTo":"claude-opus-4.7","sourceType":"blog","sourceKind":"community","sourceUrl":"https://www.mindstudio.ai/blog/gpt-5-5-review-developers-builders","date":"2026-04-26"},{"claim":"Says Gemini 3.1 Pro drifts off task on long-horizon agentic work, unlike Opus 4.6+, GPT-5.5 and GLM-5.2.","taskFamily":null,"taskDescription":null,"characteristics":[],"direction":"worse","comparedTo":"glm-5.2","sourceType":"hacker-news","sourceKind":"community","sourceUrl":"https://news.ycombinator.com/item?id=48709670","date":null},{"claim":"Judged better than Opus at Swift and Apple APIs and free of the context-bleed seen in Qwen 3.8 Max, but weaker at structured planning.","taskFamily":null,"taskDescription":null,"characteristics":[],"direction":"mixed","comparedTo":"qwen3.8-max","sourceType":"blog","sourceKind":"community","sourceUrl":"https://getneotiler.com/blog/qwen-38-max-vs-gemini-31-pro-developer-review/","date":"2026-09-16"},{"claim":"Useful for biomedical literature screening and assay-parameter extraction, but it can invent citations, DOIs and journals.","taskFamily":null,"taskDescription":null,"characteristics":[],"direction":"mixed","comparedTo":null,"sourceType":"blog","sourceKind":"community","sourceUrl":"https://www.layer3labs.io/guides/gemini-3-1-pro-for-biomedical-research","date":"2026-09-09"},{"claim":"Handles larger codebases better than Gemini 3.8 Flash: a question answered in about a minute by 3.1 Pro took 15 to 20 minutes with Flash.","taskFamily":null,"taskDescription":null,"characteristics":[],"direction":"better","comparedTo":"gemini-3.8-flash","sourceType":"forum","sourceKind":"community","sourceUrl":"https://discuss.ai.google.dev/t/critical-removing-gemini-3-1-pro-and-locking-argon-behind-ultra-tier-makes-antigravity-ide-completely-useless-for-pro-subscribers/188979","date":"2026-10-14"},{"claim":"An Ultra user stayed on 3.1 Pro because 3.8 Flash failed after 20+ attempts at feature work in an existing codebase.","taskFamily":null,"taskDescription":null,"characteristics":[],"direction":"better","comparedTo":"gemini-3.8-flash","sourceType":"forum","sourceKind":"community","sourceUrl":"https://discuss.ai.google.dev/t/critical-removing-gemini-3-1-pro-and-locking-argon-behind-ultra-tier-makes-antigravity-ide-completely-useless-for-pro-subscribers/188979","date":"2026-10-14"},{"claim":"Subscribers complain Google removed 3.1 Pro from Antigravity for Pro users, leaving only Flash, and say Flash cannot replace a Pro model.","taskFamily":null,"taskDescription":null,"characteristics":[],"direction":"better","comparedTo":"gemini-3.8-flash","sourceType":"forum","sourceKind":"community","sourceUrl":"https://discuss.ai.google.dev/t/critical-removing-gemini-3-1-pro-and-locking-argon-behind-ultra-tier-makes-antigravity-ide-completely-useless-for-pro-subscribers/188979","date":"2026-10-14"}],"url":"https://gjalla.io/models/gemini-3.1-pro/"},{"model":"gemini-3.5-flash-lite","vendor":"google","tier":null,"price":{"input":0.3,"output":2.5,"cacheRead":0.03,"cacheWrite":0.3,"source":"platform","sourceUrl":null},"meta":{"released":"2026-07-21","contextWindow":1000000,"maxOutput":64000,"modalities":["text","image","audio","video"],"officialCardUrl":"https://ai.google.dev/gemini-api/docs/generate-content/whats-new-gemini-3.6"},"excelsAt":[{"point":"Google positions it as a low-latency, minimal-cost model optimized for coding, precise document reading and small agent pipelines.","basis":"vendor","sourceKind":"primary","sourceUrl":"https://docs.cloud.google.com/gemini-enterprise-agent-platform/models/gemini/3-5-flash-lite"},{"point":"Cheapest current Gemini ($0.30/$2.50 per 1M) with very high throughput (about 490 tokens/s) and a 1M-token window.","basis":"independent","sourceKind":"independent","sourceUrl":"https://blog.buildfastwithai.com/gemini-3-5-flash-lite-review-price-benchmarks"},{"point":"Google-reported SWE-Bench Pro 54.2% and OSWorld-Verified 74.0%, large jumps over 3.1 Flash-Lite.","basis":"vendor","sourceKind":"relayed","sourceUrl":"https://blog.buildfastwithai.com/gemini-3-5-flash-lite-review-price-benchmarks"},{"point":"Good fit for classification, extraction, routing, translation and agentic search in high-volume or batch pipelines.","basis":"independent","sourceKind":"independent","sourceUrl":"https://geotoolbox.ai/blog/gemini-3-6-flash-vs-3-5-flash-lite"},{"point":"Strong on OCR transcription (87.5% at low effort) and very fast per image, about 3.9s per sample.","basis":"independent","sourceKind":"independent","sourceUrl":"https://playground.roboflow.com/models/google/gemini-3-5-flash-lite"}],"watchOuts":[{"point":"Slow first token (about 11.7s per Artificial Analysis), so interactive chat feels sluggish.","basis":"independent","sourceKind":"independent","sourceUrl":"https://blog.buildfastwithai.com/gemini-3-5-flash-lite-review-price-benchmarks"},{"point":"Capability ceiling: intelligence index 36 versus 50 for 3.6 Flash; long-context MRCR 72.2% versus 91.8%.","basis":"independent","sourceKind":"independent","sourceUrl":"https://blog.buildfastwithai.com/gemini-3-5-flash-lite-review-price-benchmarks"},{"point":"Weak at counting in images (52.7%, rank 44 of 63); reasoning needs high effort at 3 to 5x cost.","basis":"independent","sourceKind":"independent","sourceUrl":"https://playground.roboflow.com/models/google/gemini-3-5-flash-lite"},{"point":"Poor fit for high-stakes legal, medical or financial calls, long-document reasoning and frontier coding; escalate quality-critical steps.","basis":"independent","sourceKind":"independent","sourceUrl":"https://blog.buildfastwithai.com/gemini-3-5-flash-lite-review-price-benchmarks"}],"benchmarks":{"coding_agentic":{"pct":0.062,"benchmarks":[{"benchmark":"swe-bench pro","score":54.2,"pct":0.125,"independent":false,"sourceKind":"relayed","sourceUrl":"https://www.datacamp.com/blog/gemini-3-6-flash-3-5-flash-lite-3-5-flash-cyber","date":"2026-07-21"},{"benchmark":"terminal-bench 2.1","score":54,"pct":0,"independent":false,"sourceKind":"relayed","sourceUrl":"https://www.datacamp.com/blog/gemini-3-6-flash-3-5-flash-lite-3-5-flash-cyber","date":"2026-07-21"}]},"computer_use":{"pct":0.5,"benchmarks":[{"benchmark":"osworld-verified","score":74,"pct":0.5,"independent":false,"sourceKind":"relayed","sourceUrl":"https://www.datacamp.com/blog/gemini-3-6-flash-3-5-flash-lite-3-5-flash-cyber","date":"2026-07-21"}]},"overall":{"pct":0,"benchmarks":[{"benchmark":"artificial analysis intelligence index","score":22,"pct":0,"independent":true,"sourceKind":"independent","sourceUrl":"https://artificialanalysis.ai/leaderboards/models","date":"2026-10-19"},{"benchmark":"gdpval-aa v2","score":1140,"pct":0,"independent":false,"sourceKind":"relayed","sourceUrl":"https://www.datacamp.com/blog/gemini-3-6-flash-3-5-flash-lite-3-5-flash-cyber","date":"2026-07-21"}]},"reasoning":{"pct":0,"benchmarks":[{"benchmark":"humanity's last exam","score":18,"pct":0,"independent":false,"sourceKind":"primary","sourceUrl":"https://ai.google.dev/gemini-api/docs/generate-content/whats-new-gemini-3.6","date":"2026-10-13"}]}},"anecdotes":[{"claim":"An analysis positions Flash-Lite as a subagent tier: at $0.30/$2.50 per 1M and about 350 tok/s it makes fan-out cheap, but failed or escalated calls erode savings and cost figures are the author's own arithmetic.","taskFamily":"agentic-long-running","taskDescription":null,"characteristics":[],"direction":"mixed","comparedTo":"gemini-3.6-flash","sourceType":"blog","sourceKind":"community","sourceUrl":"https://www.digitalapplied.com/blog/gemini-3-5-flash-lite-subagent-economics-multi-agent-2026","date":"2026-07-22"},{"claim":"Across five vision tasks it averaged 61.7% (rank 33 of 63) but was cheap and third fastest; OCR transcription was strong while counting was its weakest area.","taskFamily":null,"taskDescription":null,"characteristics":[],"direction":"mixed","comparedTo":null,"sourceType":"benchmark","sourceKind":"community","sourceUrl":"https://playground.roboflow.com/models/google/gemini-3-5-flash-lite","date":null},{"claim":"Author estimates a million small documents costs about $1,100 on Flash-Lite versus $4,500 on 3.6 Flash, and recommends it for first-pass filtering with escalation to a stronger model.","taskFamily":null,"taskDescription":null,"characteristics":[],"direction":"better","comparedTo":"gemini-3.6-flash","sourceType":"blog","sourceKind":"community","sourceUrl":"https://blog.buildfastwithai.com/gemini-3-5-flash-lite-review-price-benchmarks","date":"2026-07-22"},{"claim":"Reviewer warns that first-token latency over 11 seconds and weaker reasoning make it a bad choice for live chat or work needing cleanup.","taskFamily":null,"taskDescription":null,"characteristics":[],"direction":"mixed","comparedTo":"gemini-3.6-flash","sourceType":"blog","sourceKind":"community","sourceUrl":"https://geotoolbox.ai/blog/gemini-3-6-flash-vs-3-5-flash-lite","date":null}],"url":"https://gjalla.io/models/gemini-3.5-flash-lite/"},{"model":"gemini-3.7-flash","vendor":"google","tier":null,"price":null,"meta":{"released":null,"contextWindow":null,"maxOutput":null,"modalities":null,"officialCardUrl":null},"excelsAt":[],"watchOuts":[],"benchmarks":{"coding":{"pct":0,"benchmarks":[{"benchmark":"lmarena code arena webdev","score":1590.6,"pct":0,"independent":true,"sourceKind":"independent","sourceUrl":"https://lmarena.ai/leaderboard/webdev","date":null}]},"coding_agentic":{"pct":0.417,"benchmarks":[{"benchmark":"terminal-bench 2.1 (aggregator, anotherwrapper.com)","score":85.8,"pct":0.417,"independent":true,"sourceKind":"independent","sourceUrl":"https://anotherwrapper.com/tools/llm-pricing/leaderboards/terminal-bench","date":"2026-10-19"}]}},"anecdotes":[],"url":"https://gjalla.io/models/gemini-3.7-flash/"},{"model":"gemini-3.8-flash","vendor":"google","tier":null,"price":{"input":0.75,"output":3.75,"cacheRead":0.075,"cacheWrite":0.75,"source":"platform","sourceUrl":null},"meta":{"released":"2026-09-02","contextWindow":1048576,"maxOutput":65536,"modalities":["text","image","audio","video"],"officialCardUrl":"https://deepmind.google/models/model-cards/gemini-3-8-flash/"},"excelsAt":[{"point":"Google's most intelligent Flash workhorse, improved over 3.7 Flash in software engineering, agentic tasks and multi-step domain reasoning.","basis":"vendor","sourceKind":"primary","sourceUrl":"https://blog.google/innovation-and-ai/models-and-research/gemini-models/3-8-flash-and-3-8-flash-cyber/"},{"point":"Targets long-horizon coding and autonomous agents; DeepSWE v1.1 73.7% is near Opus 5 (74.0%) at far lower cost per task.","basis":"vendor","sourceKind":"relayed","sourceUrl":"https://www.vellum.ai/blog/gemini-3-8-flash-benchmarks-explained"},{"point":"Leads or ties on finance and legal agent benchmarks (Vals Finance Agent v2 61.4%, Harvey legal all-pass 10.0%).","basis":"vendor","sourceKind":"relayed","sourceUrl":"https://www.vellum.ai/blog/gemini-3-8-flash-benchmarks-explained"},{"point":"Very fast output (about 302 to 305 tokens/s, third fastest tracked) and strong prompt-injection resistance (5.5% attack success).","basis":"independent","sourceKind":"independent","sourceUrl":"https://www.eesel.ai/blog/gemini-3-8-flash"},{"point":"Works best inside a multi-turn agentic harness like Antigravity; scoped, cost-sensitive coding is the sweet spot.","basis":"independent","sourceKind":"independent","sourceUrl":"https://www.mindstudio.ai/blog/gemini-3-8-flash-harness-demos"},{"point":"Caught and fixed all three planted bugs in each of three trials and flagged extra real issues.","basis":"practitioner","sourceKind":"community","sourceUrl":"https://developer.puter.com/blog/gemini-3-8-flash-review/"},{"point":"Rank 3 on the Terminal-Bench 2.1 aggregator at 89.4%, ahead of GPT-5.6 Sol and Kimi K3 (harness varies per entry).","basis":"independent","sourceKind":"independent","sourceUrl":"https://anotherwrapper.com/tools/llm-pricing/leaderboards/terminal-bench"}],"watchOuts":[{"point":"Weak on hard agentic terminal and computer-use work: Terminal-Bench 4.0 19.1% (Opus 5 51.8%), OSWorld-2.0 59.0% (75.4%).","basis":"independent","sourceKind":"independent","sourceUrl":"https://www.vellum.ai/blog/gemini-3-8-flash-benchmarks-explained"},{"point":"Verbose, with about 13s time to first token; roughly 30% more output tokens per task than 3.7 Flash.","basis":"independent","sourceKind":"independent","sourceUrl":"https://www.eesel.ai/blog/gemini-3-8-flash"},{"point":"Introductory price of $0.75/$3.75 doubles on 1 January 2027, and extra thinking tokens bill at the output rate.","basis":"vendor","sourceKind":"primary","sourceUrl":"https://blog.google/innovation-and-ai/models-and-research/gemini-models/3-8-flash-and-3-8-flash-cyber/"},{"point":"Output quality drops in plain chat versus an agentic harness, and visual design instincts trail Claude models.","basis":"independent","sourceKind":"independent","sourceUrl":"https://www.mindstudio.ai/blog/gemini-3-8-flash-harness-demos"},{"point":"Scored 44 vs 97 for 3.7 Flash on one rate-limiter task after rejecting a zero refill rate; single task and arguable prompt.","basis":"practitioner","sourceKind":"community","sourceUrl":"https://getneotiler.com/blog/gemini-3-8-flash-vs-3-7-flash-vs-3-1-pro-an-independent-rate-limiter-test/"}],"benchmarks":{"coding_agentic":{"pct":0.638,"benchmarks":[{"benchmark":"deepswe v1.1","score":73.7,"pct":0.6,"independent":false,"sourceKind":"primary","sourceUrl":"https://deepmind.google/models/model-cards/gemini-3-8-flash/","date":"2026-09-02"},{"benchmark":"terminal-bench 2.1","score":89.4,"pct":0.909,"independent":false,"sourceKind":"primary","sourceUrl":"https://deepmind.google/models/model-cards/gemini-3-8-flash/","date":"2026-09-02"},{"benchmark":"terminal-bench 2.1 (aggregator, anotherwrapper.com)","score":89.4,"pct":0.917,"independent":true,"sourceKind":"independent","sourceUrl":"https://anotherwrapper.com/tools/llm-pricing/leaderboards/terminal-bench","date":"2026-10-19"},{"benchmark":"terminal-bench 4.0","score":19.1,"pct":0.125,"independent":false,"sourceKind":"primary","sourceUrl":"https://deepmind.google/models/model-cards/gemini-3-8-flash/","date":"2026-09-02"}]},"overall":{"pct":0.645,"benchmarks":[{"benchmark":"artificial analysis intelligence index","score":59,"pct":0.957,"independent":true,"sourceKind":"independent","sourceUrl":"https://www.theregister.com/ai-and-ml/2026/09/02/with-gemini-38-flash-google-reminds-everyone-its-still-in-the-race/5294049","date":"2026-09-02"},{"benchmark":"gdpval-aa v2","score":1545,"pct":0.333,"independent":false,"sourceKind":"primary","sourceUrl":"https://deepmind.google/models/model-cards/gemini-3-8-flash/","date":"2026-09-02"}]}},"anecdotes":[{"claim":"A Google DeepMind engineer says 3.8 Flash takes smaller steps and verifies its work more often, which burns more tokens but helps long tasks (vendor-side framing).","taskFamily":"agentic-long-running","taskDescription":null,"characteristics":[],"direction":"better","comparedTo":"gemini-3.7-flash","sourceType":"x","sourceKind":"community","sourceUrl":"https://www.eesel.ai/blog/gemini-3-8-flash","date":"2026-09-08"},{"claim":"An X user called it unusable for SWE work and said it cost more than another model on simple coding tasks; another tester noted it cannot test its own work in the Antigravity browser.","taskFamily":"feature","taskDescription":null,"characteristics":[],"direction":"worse","comparedTo":null,"sourceType":"x","sourceKind":"community","sourceUrl":"https://www.eesel.ai/blog/gemini-3-8-flash","date":"2026-09-08"},{"claim":"Simon Willison reported an HTML visualization generated in 13 seconds for about 1.8 cents and called the speed and frontend quality exciting.","taskFamily":"feature","taskDescription":null,"characteristics":[],"direction":"better","comparedTo":null,"sourceType":"hacker-news","sourceKind":"community","sourceUrl":"https://www.eesel.ai/blog/gemini-3-8-flash-review","date":"2026-09-08"},{"claim":"HN commenters say it is fast and holds quality under weekday load, but verbose (about 11k more tokens per task than 3.7 Flash), requires constant approvals in the CLI, and is fine for scaffolding but not long agentic tasks.","taskFamily":"agentic-long-running","taskDescription":null,"characteristics":[],"direction":"mixed","comparedTo":"gemini-3.7-flash","sourceType":"hacker-news","sourceKind":"community","sourceUrl":"https://www.eesel.ai/blog/gemini-3-8-flash-review","date":"2026-09-08"},{"claim":"A commenter used the prior 3.7 Flash to audit Opus output and said it finds subtle defects even when unit tests pass, and runs far faster in Antigravity CLI.","taskFamily":"code-review","taskDescription":null,"characteristics":[],"direction":"better","comparedTo":"claude-opus-5","sourceType":"hacker-news","sourceKind":"community","sourceUrl":"https://www.eesel.ai/blog/gemini-3-8-flash-review","date":"2026-08-14"},{"claim":"The same detailed prompt gave a thinner result in the Gemini app than in Antigravity, whose implement-verify-revise loop produced one of the best outputs, a live ISS tracker.","taskFamily":null,"taskDescription":null,"characteristics":[],"direction":"mixed","comparedTo":null,"sourceType":"blog","sourceKind":"community","sourceUrl":"https://www.mindstudio.ai/blog/gemini-3-8-flash-harness-demos","date":"2026-09-03"},{"claim":"A data-center slide deck had correct brand colors and structure, but the author suggested a Claude model for design-heavy decks; product landing pages were uneven.","taskFamily":null,"taskDescription":null,"characteristics":[],"direction":"mixed","comparedTo":"claude-fable-5.1","sourceType":"blog","sourceKind":"community","sourceUrl":"https://www.mindstudio.ai/blog/gemini-3-8-flash-harness-demos","date":"2026-09-03"},{"claim":"One X user called it unusable for software engineering and said it cost more than Fable on simple coding tasks.","taskFamily":null,"taskDescription":null,"characteristics":[],"direction":"worse","comparedTo":"claude-fable-5","sourceType":"x","sourceKind":"community","sourceUrl":"https://www.eesel.ai/blog/gemini-3-8-flash","date":"2026-09-02"},{"claim":"Praised its price, speed and reasoning, but noted it cannot test its own work or view the browser in Antigravity.","taskFamily":null,"taskDescription":null,"characteristics":[],"direction":"mixed","comparedTo":null,"sourceType":"x","sourceKind":"community","sourceUrl":"https://www.eesel.ai/blog/gemini-3-8-flash","date":"2026-09-02"},{"claim":"Glean's AI lead reportedly saw more than three times as many completed tasks than with 3.7 Flash on long document workflows.","taskFamily":null,"taskDescription":null,"characteristics":[],"direction":"better","comparedTo":"gemini-3.7-flash","sourceType":"blog","sourceKind":"community","sourceUrl":"https://www.eesel.ai/blog/gemini-3-8-flash","date":null},{"claim":"Pro-tier user says Flash lost context across files, looped, hallucinated deprecated APIs and dropped code during refactors.","taskFamily":null,"taskDescription":null,"characteristics":[],"direction":"worse","comparedTo":"gemini-3.1-pro","sourceType":"forum","sourceKind":"community","sourceUrl":"https://discuss.ai.google.dev/t/critical-removing-gemini-3-1-pro-and-locking-argon-behind-ultra-tier-makes-antigravity-ide-completely-useless-for-pro-subscribers/188979","date":"2026-10-14"},{"claim":"Flash is faster, but a simple function move silently dropped a line of code and cost debugging time.","taskFamily":null,"taskDescription":null,"characteristics":[],"direction":"mixed","comparedTo":"gemini-3.1-pro","sourceType":"forum","sourceKind":"community","sourceUrl":"https://discuss.ai.google.dev/t/critical-removing-gemini-3-1-pro-and-locking-argon-behind-ultra-tier-makes-antigravity-ide-completely-useless-for-pro-subscribers/188979","date":"2026-10-14"},{"claim":"Gateway tests over 150 cases found parity with 3.7 Flash on function calling (including parallel), structured JSON output and thinking tiers.","taskFamily":null,"taskDescription":null,"characteristics":[],"direction":"mixed","comparedTo":"gemini-3.7-flash","sourceType":"blog","sourceKind":"community","sourceUrl":"https://docs.apiyi.com/en/news/gemini-3-8-flash-launch","date":null},{"claim":"Given a short Python function with a SQL injection, a mutable default argument and a divide-by-zero, 3.8 Flash found and correctly fixed all three in all three runs, adding extra valid findings; latency ranged 18.6-41.1s. No other model was tested.","taskFamily":"bug-fix","taskDescription":null,"characteristics":[],"direction":"better","comparedTo":null,"sourceType":"blog","sourceKind":"community","sourceUrl":"https://developer.puter.com/blog/gemini-3-8-flash-review/","date":"2026-09-03"},{"claim":"On a thread-safe token-bucket rate limiter with pytest tests, 3.8 Flash wrote the most polished code but scored 44/100 on an external harness (3.7 Flash 97, 3.1 Pro 100) because it rejected a zero refill rate. Single task; the author's justification for that requirement is questionable.","taskFamily":"feature","taskDescription":null,"characteristics":[],"direction":"worse","comparedTo":"gemini-3.1-pro","sourceType":"blog","sourceKind":"community","sourceUrl":"https://getneotiler.com/blog/gemini-3-8-flash-vs-3-7-flash-vs-3-1-pro-an-independent-rate-limiter-test/","date":"2026-09-04"}],"url":"https://gjalla.io/models/gemini-3.8-flash/"},{"model":"gemini-4-argon","vendor":"google","tier":null,"price":{"input":2,"output":10,"cacheRead":0.1,"cacheWrite":2,"source":"platform","sourceUrl":null},"meta":{"released":"2026-09-30","contextWindow":1048576,"maxOutput":1000000,"modalities":["text"],"officialCardUrl":"https://ai.google.dev/gemini-api/docs/latest-model"},"excelsAt":[{"point":"Google's flagship reasoning model for complex software engineering, advanced math and science, and autonomous multi-step agents.","basis":"vendor","sourceKind":"primary","sourceUrl":"https://ai.google.dev/gemini-api/docs/models/gemini-4-argon?hl=zh-tw"},{"point":"Long decode: outputs can reach about 1M tokens, versus a 64K default; useful for very long generations and long-running agents.","basis":"vendor","sourceKind":"primary","sourceUrl":"https://ai.google.dev/gemini-api/docs/models/gemini-4-argon?hl=zh-tw"},{"point":"Very low hallucination rate (15% on AA-Omniscience); it declines to answer instead of guessing.","basis":"independent","sourceKind":"independent","sourceUrl":"https://www.eesel.ai/de/blog/gemini-4-argon-test"},{"point":"Leads finance, legal and business agent benchmarks and long-document retrieval (GraphWalks 256K to 1M), mostly third-party leaderboards.","basis":"independent","sourceKind":"independent","sourceUrl":"https://www.eesel.ai/de/blog/gemini-4-argon-test"},{"point":"Ranked first on AutomationBench-AA multi-step automation (77.5%), ahead of Claude Sonnet 5.5 and Opus 5.5.","basis":"independent","sourceKind":"independent","sourceUrl":"https://www.eesel.ai/de/blog/gemini-4-argon-test"},{"point":"Google used its agents to translate C/C++ to Rust, including over 800K lines of Fuchsia kernel code.","basis":"vendor","sourceKind":"relayed","sourceUrl":"https://www.deeplearning.ai/the-batch/argon-a-low-hallucination-model-build-for-knowledge-work"},{"point":"Debugged Vulkan code by dumping framebuffers to PNG and using false-color shaders, per one HN user.","basis":"practitioner","sourceKind":"community","sourceUrl":"https://news.ycombinator.com/item?id=50073879"},{"point":"Ranked first on the Vals Index (68.90% as of Oct 12, ahead of Sonnet 5.5 at 67.04%) and on Vals Finance Agent v2 (65.4%).","basis":"independent","sourceKind":"independent","sourceUrl":"https://letsdatascience.com/blog/gemini-4-argon-public-independent-tests-rank-it-first-and-eighth"}],"watchOuts":[{"point":"Trails Anthropic's newest models on terminal and agentic coding (Terminal-Bench 4.0 57.1%, fourth); index five points behind Opus 5.5.","basis":"independent","sourceKind":"independent","sourceUrl":"https://www.eesel.ai/de/blog/gemini-4-argon-test"},{"point":"Token-hungry: about 62K output tokens per task versus 27K for GPT-6 Astra; list price rises from $2/$10 to $4/$20.","basis":"independent","sourceKind":"independent","sourceUrl":"https://www.eesel.ai/de/blog/gemini-4-argon-test"},{"point":"At high effort it can spend the whole 65,536-token default output budget on reasoning and never write the answer file.","basis":"practitioner","sourceKind":"community","sourceUrl":"https://www.datacamp.com/blog/gemini-4-argon-vs-claude-opus-5-5"},{"point":"Access restricted: vetted Fairwind cyber defenders first, then paid API and Ultra; no Live API, no explicit caching.","basis":"vendor","sourceKind":"relayed","sourceUrl":"https://www.deeplearning.ai/the-batch/argon-a-low-hallucination-model-build-for-knowledge-work"},{"point":"Thinks a lot: about 24k output tokens and 4 minutes per small fix vs 3k and 32s for Sonnet 5.5; one fix timed out; pulls many unrelated files into context.","basis":"practitioner","sourceKind":"community","sourceUrl":"https://news.ycombinator.com/item?id=50073879"},{"point":"Independent rankings split: first on Vals professional workflows but only tied eighth of 216 on the AA index (53 vs 58 for Opus 5.5).","basis":"independent","sourceKind":"independent","sourceUrl":"https://letsdatascience.com/blog/gemini-4-argon-public-independent-tests-rank-it-first-and-eighth"}],"benchmarks":{"coding":{"pct":0.683,"benchmarks":[{"benchmark":"code arena webdev","score":1678.2,"pct":0.714,"independent":true,"sourceKind":"independent","sourceUrl":"https://llmrun.dev/benchmark/lmarena-webdev?show=all","date":null},{"benchmark":"lmarena code arena webdev","score":1678.2,"pct":0.652,"independent":true,"sourceKind":"independent","sourceUrl":"https://lmarena.ai/leaderboard/webdev","date":null}]},"coding_agentic":{"pct":0.875,"benchmarks":[{"benchmark":"deepswe v1.1","score":77.9,"pct":1,"independent":false,"sourceKind":"relayed","sourceUrl":"https://www.datacamp.com/blog/gemini-4-argon","date":"2026-09-30"},{"benchmark":"terminal-bench 4.0","score":57.4,"pct":0.75,"independent":false,"sourceKind":"relayed","sourceUrl":"https://www.datacamp.com/blog/gemini-4-argon","date":"2026-09-30"}]},"overall":{"pct":0.848,"benchmarks":[{"benchmark":"artificial analysis intelligence index","score":53,"pct":0.696,"independent":true,"sourceKind":"independent","sourceUrl":"https://artificialanalysis.ai/leaderboards/models","date":"2026-10-19"},{"benchmark":"vals index","score":68.9,"pct":1,"independent":false,"sourceKind":"relayed","sourceUrl":"https://www.datacamp.com/blog/gemini-4-argon","date":"2026-09-30"}]}},"anecdotes":[{"claim":"Reviewer notes Argon wins most non-coding benchmarks but is last among cited frontier models on Terminal-bench 4.0 and trails on FrontierSWE v2; advises keeping coding pipelines on Claude or GPT rather than migrating.","taskFamily":"feature","taskDescription":null,"characteristics":[],"direction":"worse","comparedTo":"claude-opus-5.5","sourceType":"blog","sourceKind":"community","sourceUrl":"https://dev.to/max_quimby/gemini-4-argon-wins-12-of-18-benchmarks-code-isnt-one-2h70","date":"2026-10-02"},{"claim":"Artificial Analysis data cited by the same article shows Argon uses about 62k output tokens per task vs 27k for GPT-6 Astra, so its low intro price does not make it cheaper per task at regular pricing.","taskFamily":"agentic-long-running","taskDescription":null,"characteristics":[],"direction":"mixed","comparedTo":"gpt-6-astra","sourceType":"blog","sourceKind":"community","sourceUrl":"https://dev.to/max_quimby/gemini-4-argon-wins-12-of-18-benchmarks-code-isnt-one-2h70","date":"2026-10-02"},{"claim":"Google's showcase work for Argon is code comprehension and migration (kernel comprehension, a SIMD rewrite), not greenfield development; commentators warn labs can optimize for leaderboards.","taskFamily":"refactor","taskDescription":null,"characteristics":[],"direction":"mixed","comparedTo":null,"sourceType":"x","sourceKind":"community","sourceUrl":"https://dev.to/max_quimby/gemini-4-argon-wins-12-of-18-benchmarks-code-isnt-one-2h70","date":"2026-10-02"},{"claim":"An early-access user says it is the first Gemini they can hand complex research tasks to, though not fully reliable. (HN comment relayed by eesel.)","taskFamily":null,"taskDescription":null,"characteristics":[],"direction":"better","comparedTo":null,"sourceType":"hacker-news","sourceKind":"community","sourceUrl":"https://www.eesel.ai/de/blog/gemini-4-argon-test","date":null},{"claim":"In a single-file robot-arm animation test it handled the planned detour and honestly flagged an unreachable target, but later clipped through obstacles; Opus 5.5 kept the constraints better.","taskFamily":null,"taskDescription":null,"characteristics":[],"direction":"worse","comparedTo":"claude-opus-5.5","sourceType":"blog","sourceKind":"community","sourceUrl":"https://www.datacamp.com/blog/gemini-4-argon-vs-claude-opus-5-5","date":null},{"claim":"At high reasoning effort the run exhausted the default output budget on thinking and produced no file; the author had to rerun at medium effort.","taskFamily":null,"taskDescription":null,"characteristics":[],"direction":"worse","comparedTo":"claude-opus-5.5","sourceType":"blog","sourceKind":"community","sourceUrl":"https://www.datacamp.com/blog/gemini-4-argon-vs-claude-opus-5-5","date":null},{"claim":"Anonymous insiders reportedly told Bloomberg it benchmarks well but gets stuck on real business codebases, especially frontend work; Google called this inaccurate.","taskFamily":null,"taskDescription":null,"characteristics":[],"direction":"worse","comparedTo":null,"sourceType":"news","sourceKind":"community","sourceUrl":"https://tech.ifeng.com/c/8wogfnLYLN8","date":null},{"claim":"A Pro subscriber found Argon generates code much more slowly than Gemini 3.1 Pro did and threatened to leave Google's ecosystem.","taskFamily":null,"taskDescription":null,"characteristics":[],"direction":"worse","comparedTo":"gemini-3.1-pro","sourceType":"forum","sourceKind":"community","sourceUrl":"https://discuss.ai.google.dev/t/critical-removing-gemini-3-1-pro-and-locking-argon-behind-ultra-tier-makes-antigravity-ide-completely-useless-for-pro-subscribers/188979","date":"2026-10-15"},{"claim":"An Ultra subscriber says Argon burns through quota quickly and wants Gemini 3.1 Pro back.","taskFamily":null,"taskDescription":null,"characteristics":[],"direction":"worse","comparedTo":"gemini-3.1-pro","sourceType":"forum","sourceKind":"community","sourceUrl":"https://discuss.ai.google.dev/t/critical-removing-gemini-3-1-pro-and-locking-argon-behind-ultra-tier-makes-antigravity-ide-completely-useless-for-pro-subscribers/188979","date":"2026-10-14"},{"claim":"One HN user's personal eval of 11 small bug fixes: Argon solved 10 of 11, at a median 4 minutes and $0.52 per run vs $0.18 and 32s for Sonnet 5.5 and $0.09 and 1.5 min for Sol. Costs came from a tool that undercounted Argon pricing; one commenter said Opus 5.5 starts editing sooner.","taskFamily":"bug-fix","taskDescription":null,"characteristics":[],"direction":"mixed","comparedTo":"claude-sonnet-5.5","sourceType":"hacker-news","sourceKind":"community","sourceUrl":"https://news.ycombinator.com/item?id=50073879","date":"2026-10-14"},{"claim":"An educator had Argon audit an app he had built with Opus 5.5 and it found several bugs, one that could have exposed his webhook, which he patched. He notes the real risk was low and he still prefers other models for some coding.","taskFamily":"code-review","taskDescription":null,"characteristics":[],"direction":"better","comparedTo":null,"sourceType":"blog","sourceKind":"community","sourceUrl":"https://favtutor.com/gemini-4-argon-real-examples/","date":"2026-10-15"},{"claim":"Plotly's 43-question analytics test had Argon at 41/43 and all five hidden-data issues found, using about 46 queries per question; tied with Opus 5.5. Single run, one harness.","taskFamily":"data-analysis","taskDescription":null,"characteristics":[],"direction":"mixed","comparedTo":"claude-opus-5.5","sourceType":"blog","sourceKind":"community","sourceUrl":"https://plotly.com/blog/introducing-dataanalyticsbench/","date":"2026-10-17"},{"claim":"Canvas built a working booking page with filters and validation, and Argon handled a data-decline analysis well and refused to claim causation; on document reconciliation it labeled all eight claims correctly but one summary overstated what the packet said.","taskFamily":null,"taskDescription":null,"characteristics":[],"direction":"mixed","comparedTo":null,"sourceType":"blog","sourceKind":"community","sourceUrl":"https://www.analyticsvidhya.com/?p=258322","date":"2026-10-14"}],"url":"https://gjalla.io/models/gemini-4-argon/"},{"model":"glm-4.6","vendor":"zhipu","tier":null,"price":null,"meta":{"released":null,"contextWindow":null,"maxOutput":null,"modalities":null,"officialCardUrl":null},"excelsAt":[],"watchOuts":[],"benchmarks":{"coding_agentic":{"pct":0,"benchmarks":[{"benchmark":"swe-bench pro (scale seal public set)","score":9.7,"pct":0,"independent":true,"sourceKind":"independent","sourceUrl":"https://morphllm.com/swe-bench-pro","date":"2026-09-14"}]}},"anecdotes":[],"url":"https://gjalla.io/models/glm-4.6/"},{"model":"glm-5.2","vendor":"zhipu","tier":null,"price":{"input":1.4,"output":4.4,"cacheRead":0.26,"cacheWrite":1.4,"source":"platform","sourceUrl":null},"meta":{"released":"2026-06-17","contextWindow":1000000,"maxOutput":131072,"modalities":["text"],"officialCardUrl":"https://huggingface.co/blog/zai-org/glm-52-blog"},"excelsAt":[{"point":"Z.ai's flagship for long-horizon tasks, with a stable 1M-token context and selectable thinking effort levels.","basis":"vendor","sourceKind":"primary","sourceUrl":"https://huggingface.co/zai-org/GLM-5.2"},{"point":"Strong agentic and terminal work: Terminal Bench 2.1 81.0, with big gains in FrontierSWE over GLM-5.1; AIME 2026 99.2.","basis":"vendor","sourceKind":"primary","sourceUrl":"https://huggingface.co/zai-org/GLM-5.2"},{"point":"Open weights under MIT (753B MoE, about 40B active); works as a drop-in model in Claude Code, Cline and OpenCode.","basis":"vendor","sourceKind":"relayed","sourceUrl":"https://rits.shanghai.nyu.edu/ai/glm-5-2-z-ais-open-weights-coder-beats-gpt-5-5-at-1-6-the-cost/"},{"point":"Structured extraction from messy HTML and tool calling matched Sonnet at roughly one-seventh to one-eighth the cost.","basis":"independent","sourceKind":"independent","sourceUrl":"https://www.zyte.com/blog/why-im-adding-glm-5-2-to-my-agentic-coding-arsenal/"}],"watchOuts":[{"point":"Behind Opus 4.8 on agentic coding (SWE-bench Pro 62.1 vs 69.2; Tool-Decathlon 48.2 vs 59.9); numbers self-reported.","basis":"vendor","sourceKind":"primary","sourceUrl":"https://huggingface.co/zai-org/GLM-5.2"},{"point":"Verbose and slower: about 43K tokens per task versus 26K for GLM-5.1; 24 to 55s on scraping tasks.","basis":"independent","sourceKind":"independent","sourceUrl":"https://www.zyte.com/blog/why-im-adding-glm-5-2-to-my-agentic-coding-arsenal/"},{"point":"Stale library knowledge: generated third-party API config missing required settings.","basis":"independent","sourceKind":"independent","sourceUrl":"https://www.zyte.com/blog/why-im-adding-glm-5-2-to-my-agentic-coding-arsenal/"},{"point":"React output less consistent under multi-step agentic pressure and it stumbled on TypeScript compile errors before recovering.","basis":"practitioner","sourceKind":"community","sourceUrl":"https://www.lennysnewsletter.com/p/how-i-ai-glm-52-review-and-how-gusto"}],"benchmarks":{"coding":{"pct":0.043,"benchmarks":[{"benchmark":"code arena webdev","score":1604.6,"pct":0,"independent":true,"sourceKind":"independent","sourceUrl":"https://llmrun.dev/benchmark/lmarena-webdev?show=all","date":null},{"benchmark":"lmarena code arena webdev","score":1604.6,"pct":0.087,"independent":true,"sourceKind":"independent","sourceUrl":"https://lmarena.ai/leaderboard/webdev","date":null}]},"coding_agentic":{"pct":0.421,"benchmarks":[{"benchmark":"swe-bench pro","score":62.1,"pct":0.5,"independent":false,"sourceKind":"primary","sourceUrl":"https://huggingface.co/blog/zai-org/glm-52-blog","date":"2026-06-17"},{"benchmark":"swe-bench verified","score":79,"pct":0.4,"independent":true,"sourceKind":"independent","sourceUrl":"https://epoch.ai/models/glm-5-2","date":null},{"benchmark":"terminal-bench 2.1","score":81,"pct":0.364,"independent":false,"sourceKind":"primary","sourceUrl":"https://huggingface.co/blog/zai-org/glm-52-blog","date":"2026-06-17"}]},"reasoning":{"pct":0.167,"benchmarks":[{"benchmark":"gpqa diamond","score":92,"pct":0.167,"independent":true,"sourceKind":"independent","sourceUrl":"https://epoch.ai/models/glm-5-2","date":null}]}},"anecdotes":[{"claim":"Zyte's scraping tests found GLM-5.2 matched Sonnet 4.6 on extraction and tool calling at roughly 2-8x lower cost, but it was verbose, slower, and used outdated Zyte API setup; not yet trusted for hardest long-horizon planning where Opus leads.","taskFamily":"feature","taskDescription":null,"characteristics":[],"direction":"mixed","comparedTo":"claude-sonnet-4.6","sourceType":"blog","sourceKind":"community","sourceUrl":"https://www.zyte.com/blog/why-im-adding-glm-5-2-to-my-agentic-coding-arsenal/","date":"2026-06-19"},{"claim":"A month-long user says it root-caused a main-thread lag and a battery polling loop, verified fixes with tests and kept PRs focused, but is a token glutton, cannot read screenshots and has immature tool calls; close to Opus 4.8 for daily coding but short on visual work.","taskFamily":"bug-fix","taskDescription":null,"characteristics":[],"direction":"mixed","comparedTo":"claude-opus-4.8","sourceType":"blog","sourceKind":"community","sourceUrl":"https://www.rafay99.com/blog/a-month-with-glm-52-cheap-relentless-and-blind/","date":"2026-07-08"},{"claim":"Kilo's code-review test with planted bugs: GLM-5.2 caught 13-15 of 16 obvious bugs but only 4-7 of 10 subtle cross-route bugs, trailing GPT-5.5 (9/10) and Opus 4.8 (8/10); results varied run to run and with prompt wording.","taskFamily":"code-review","taskDescription":null,"characteristics":[],"direction":"worse","comparedTo":"gpt-5.5","sourceType":"blog","sourceKind":"community","sourceUrl":"https://blog.kilo.ai/p/glm-52s-code-reviews-are-only-as","date":"2026-06-26"},{"claim":"ChatPRD's live review: fast, solid at repo summaries and HTML/UI iteration, and ran a 30-45 minute autonomous Sentry/Vercel triage into a 14-fix plan, but stalled on complex React/TypeScript and needed human review.","taskFamily":"operations-support","taskDescription":null,"characteristics":[],"direction":"mixed","comparedTo":null,"sourceType":"blog","sourceKind":"community","sourceUrl":"https://www.chatprd.ai/how-i-ai/glm-5-2-review-open-weights-model","date":"2026-06-24"},{"claim":"Built a Matrix bot with encryption and a Rust tool-using agent for about $20 via Fireworks; found it fast and pleasant, far cheaper than Opus or GPT.","taskFamily":null,"taskDescription":null,"characteristics":[],"direction":"better","comparedTo":"claude-opus-4.8","sourceType":"hacker-news","sourceKind":"community","sourceUrl":"https://news.ycombinator.com/item?id=48709670","date":null},{"claim":"Benchmark author says multi-agent coding is just shy of Opus 4.6 on average, but cost-adjusted GLM is frontier; not the pick if you want only the top model.","taskFamily":null,"taskDescription":null,"characteristics":[],"direction":"mixed","comparedTo":"claude-opus-4.6","sourceType":"hacker-news","sourceKind":"community","sourceUrl":"https://news.ycombinator.com/item?id=48709670","date":null},{"claim":"Calls GLM-5.2 the third model, after Opus 4.6+ and GPT-5.5, able to sustain long-horizon agentic work.","taskFamily":null,"taskDescription":null,"characteristics":[],"direction":"better","comparedTo":"gemini-3.1-pro","sourceType":"hacker-news","sourceKind":"community","sourceUrl":"https://news.ycombinator.com/item?id=48709670","date":null},{"claim":"Needs more guidance than Opus to reach an acceptable plan, which takes Opus less time.","taskFamily":null,"taskDescription":null,"characteristics":[],"direction":"worse","comparedTo":"claude-opus-4.8","sourceType":"hacker-news","sourceKind":"community","sourceUrl":"https://news.ycombinator.com/item?id=48709670","date":null},{"claim":"For Elixir work it beat Google and Anthropic models that rank higher on general benchmarks.","taskFamily":null,"taskDescription":null,"characteristics":[],"direction":"better","comparedTo":"gemini-3.1-pro","sourceType":"hacker-news","sourceKind":"community","sourceUrl":"https://news.ycombinator.com/item?id=48709670","date":null},{"claim":"Added a search feature to a Next.js blog for about $0.27 in OpenCode; asked sensible clarifying questions and kept the page server-rendered, but used more tokens than GLM-5.1.","taskFamily":null,"taskDescription":null,"characteristics":[],"direction":"better","comparedTo":"glm-5.1","sourceType":"blog","sourceKind":"community","sourceUrl":"https://dev.to/danielbergholz/testing-glm-52-on-opencode-im-impressed-1780","date":"2026-06-18"},{"claim":"A 45-minute Claude Code run on Sentry and Vercel logs found two P0 issues monitoring had missed for $3.36, but React consistency was a worry; she recommends using it in rotation.","taskFamily":null,"taskDescription":null,"characteristics":[],"direction":"mixed","comparedTo":"claude-opus-4.8","sourceType":"podcast","sourceKind":"community","sourceUrl":"https://www.lennysnewsletter.com/p/how-i-ai-glm-52-review-and-how-gusto","date":"2026-06-29"},{"claim":"Matched Sonnet on structured JSON extraction from messy HTML and tool calls at a fraction of the cost, but was verbose, slower and out of date on library details.","taskFamily":null,"taskDescription":null,"characteristics":[],"direction":"mixed","comparedTo":"claude-sonnet-5","sourceType":"blog","sourceKind":"community","sourceUrl":"https://www.zyte.com/blog/why-im-adding-glm-5-2-to-my-agentic-coding-arsenal/","date":"2026-06-19"}],"url":"https://gjalla.io/models/glm-5.2/"},{"model":"glm-5.3","vendor":"zhipu","tier":null,"price":null,"meta":{"released":"2026-08-14","contextWindow":1000000,"maxOutput":128000,"modalities":["text"],"officialCardUrl":"https://z.ai/blog/glm-5.3"},"excelsAt":[{"point":"Z.ai reports Terminal-Bench 2.1 88.2 (level with Kimi K3 and GPT-5.6 Sol) and CyberGym 84.5, up from 77.2 on GLM-5.2 (vendor-relayed via DataNorth).","basis":"vendor","sourceKind":"relayed","sourceUrl":"https://datanorth.ai/news/z-ai-releases-glm-5-3"},{"point":"Arena WebDev 1621 and AA index 45, roughly level with other leading Chinese models.","basis":"independent","sourceKind":"independent","sourceUrl":"https://arena.ai/leaderboard/code/webdev"}],"watchOuts":[{"point":"Token-hungry in a one-shot HTML animation test: about 47K to 54K output tokens per task including reasoning, the highest among four Chinese models tested.","basis":"practitioner","sourceKind":"community","sourceUrl":"https://empiriolabs.ai/blog/kimi-k3-glm-5-3-qwen3-8-max-deepseek-v4-pro-coding-test"},{"point":"Launch left API price unlisted and weights promised for later; verify availability and licence before planning around it.","basis":"vendor","sourceKind":"relayed","sourceUrl":"https://datanorth.ai/news/z-ai-releases-glm-5-3"}],"benchmarks":{"coding":{"pct":0.137,"benchmarks":[{"benchmark":"code arena webdev","score":1621,"pct":0.143,"independent":true,"sourceKind":"independent","sourceUrl":"https://arena.ai/leaderboard/code/webdev","date":"2026-10-19"},{"benchmark":"lmarena code arena webdev","score":1621.1,"pct":0.13,"independent":true,"sourceKind":"independent","sourceUrl":"https://lmarena.ai/leaderboard/webdev","date":null}]},"coding_agentic":{"pct":0.529,"benchmarks":[{"benchmark":"deepswe v1.1","score":66.9,"pct":0.2,"independent":false,"sourceKind":"relayed","sourceUrl":"https://datanorth.ai/news/z-ai-releases-glm-5-3","date":"2026-08-14"},{"benchmark":"terminal-bench 2.1","score":88.2,"pct":0.636,"independent":false,"sourceKind":"relayed","sourceUrl":"https://datanorth.ai/news/z-ai-releases-glm-5-3","date":"2026-08-14"},{"benchmark":"terminal-bench 2.1 (aggregator, anotherwrapper.com)","score":88.2,"pct":0.75,"independent":true,"sourceKind":"independent","sourceUrl":"https://anotherwrapper.com/tools/llm-pricing/leaderboards/terminal-bench","date":"2026-10-19"}]},"overall":{"pct":0.348,"benchmarks":[{"benchmark":"artificial analysis intelligence index","score":45,"pct":0.348,"independent":true,"sourceKind":"independent","sourceUrl":"https://artificialanalysis.ai/leaderboards/models","date":"2026-10-20"}]}},"anecdotes":[{"claim":"At max reasoning effort GLM 5.3 spent about 47,000 output tokens per task on four single-file HTML animations, more than Kimi K3, Qwen3.8 Max and DeepSeek V4 Pro; the author declined to name a winner.","taskFamily":null,"taskDescription":null,"characteristics":[],"direction":"worse","comparedTo":"kimi-k3","sourceType":"blog","sourceKind":"community","sourceUrl":"https://empiriolabs.ai/blog/kimi-k3-glm-5-3-qwen3-8-max-deepseek-v4-pro-coding-test","date":"2026-10-04"},{"claim":"On a second set of three one-shot HTML scenes it wrote the longest files (about 710 lines) and about 54,000 tokens per task.","taskFamily":null,"taskDescription":null,"characteristics":[],"direction":"worse","comparedTo":"kimi-k3","sourceType":"blog","sourceKind":"community","sourceUrl":"https://empiriolabs.ai/blog/kimi-k3-glm-5-3-qwen3-8-max-deepseek-v4-pro-fun-coding-test","date":"2026-10-05"}],"url":"https://gjalla.io/models/glm-5.3/"},{"model":"glm-5.3-flash","vendor":"zhipu","tier":null,"price":null,"meta":{"released":null,"contextWindow":null,"maxOutput":null,"modalities":null,"officialCardUrl":null},"excelsAt":[],"watchOuts":[],"benchmarks":{"coding_agentic":{"pct":0.188,"benchmarks":[{"benchmark":"terminal-bench 2.1 (aggregator, anotherwrapper.com)","score":84.3,"pct":0.188,"independent":true,"sourceKind":"independent","sourceUrl":"https://anotherwrapper.com/tools/llm-pricing/leaderboards/terminal-bench","date":"2026-10-20"}]}},"anecdotes":[],"url":"https://gjalla.io/models/glm-5.3-flash/"},{"model":"gpt-5","vendor":"openai","tier":"standard","price":{"input":1.25,"output":10,"cacheRead":0.125,"cacheWrite":1.25,"source":"platform","sourceUrl":null},"meta":{"released":null,"contextWindow":null,"maxOutput":null,"modalities":null,"officialCardUrl":null},"excelsAt":[],"watchOuts":[],"benchmarks":{"coding_agentic":{"pct":0.467,"benchmarks":[{"benchmark":"swe-bench pro (scale seal public set)","score":41.8,"pct":0.467,"independent":true,"sourceKind":"independent","sourceUrl":"https://morphllm.com/swe-bench-pro","date":"2026-09-14"}]}},"anecdotes":[],"url":"https://gjalla.io/models/gpt-5/"},{"model":"gpt-5.2","vendor":"openai","tier":"standard","price":{"input":1.75,"output":14,"cacheRead":0.175,"cacheWrite":1.75,"source":"platform","sourceUrl":null},"meta":{"released":null,"contextWindow":null,"maxOutput":null,"modalities":null,"officialCardUrl":null},"excelsAt":[],"watchOuts":[],"benchmarks":{"coding_agentic":{"pct":0.2,"benchmarks":[{"benchmark":"swe-bench pro (scale seal public set)","score":29.9,"pct":0.2,"independent":true,"sourceKind":"independent","sourceUrl":"https://morphllm.com/swe-bench-pro","date":"2026-09-14"}]}},"anecdotes":[],"url":"https://gjalla.io/models/gpt-5.2/"},{"model":"gpt-5.2-codex","vendor":"openai","tier":"standard","price":{"input":1.75,"output":14,"cacheRead":0.175,"cacheWrite":1.75,"source":"platform","sourceUrl":null},"meta":{"released":null,"contextWindow":null,"maxOutput":null,"modalities":null,"officialCardUrl":null},"excelsAt":[],"watchOuts":[],"benchmarks":{"coding_agentic":{"pct":0.4,"benchmarks":[{"benchmark":"swe-bench pro (scale seal public set)","score":41,"pct":0.4,"independent":true,"sourceKind":"independent","sourceUrl":"https://morphllm.com/swe-bench-pro","date":"2026-09-14"}]}},"anecdotes":[],"url":"https://gjalla.io/models/gpt-5.2-codex/"},{"model":"gpt-5.3-codex","vendor":"openai","tier":"standard","price":{"input":1.75,"output":14,"cacheRead":0.175,"cacheWrite":1.75,"source":"platform","sourceUrl":null},"meta":{"released":"2026-02-05","contextWindow":400000,"maxOutput":128000,"modalities":["text","image"],"officialCardUrl":"https://openai.com/index/introducing-gpt-5-3-codex"},"excelsAt":[],"watchOuts":[],"benchmarks":{"coding_agentic":{"pct":0.333,"benchmarks":[{"benchmark":"swe-bench pro","score":56.8,"pct":0.333,"independent":false,"sourceKind":"primary","sourceUrl":"https://openai.com/index/introducing-gpt-5-3-codex","date":"2026-02-05"}]},"computer_use":{"pct":0,"benchmarks":[{"benchmark":"osworld-verified","score":64.7,"pct":0,"independent":false,"sourceKind":"primary","sourceUrl":"https://openai.com/index/introducing-gpt-5-3-codex","date":"2026-02-05"}]}},"anecdotes":[{"claim":"Poster complained that after earlier removals they had moved from Codex 5.3 to 5.5 extended, indicating 5.3-Codex had been withdrawn from their Codex setup; no performance claims about 5.3-Codex itself.","taskFamily":"agentic-long-running","taskDescription":null,"characteristics":[],"direction":"mixed","comparedTo":"gpt-5.5","sourceType":"forum","sourceKind":"community","sourceUrl":"https://community.openai.com/t/5-6-sol-should-be-renamed-5-6-sol-drift-edition/1386624","date":"2026-07-12"}],"url":"https://gjalla.io/models/gpt-5.3-codex/"},{"model":"gpt-5.4","vendor":"openai","tier":"standard","price":{"input":2.5,"output":15,"cacheRead":0.25,"cacheWrite":2.5,"longAbove":272000,"long":{"input":5,"output":22.5,"cacheRead":0.5,"cacheWrite":5},"source":"platform","sourceUrl":null},"meta":{"released":null,"contextWindow":null,"maxOutput":null,"modalities":null,"officialCardUrl":null},"excelsAt":[],"watchOuts":[],"benchmarks":{"coding_agentic":{"pct":0.933,"benchmarks":[{"benchmark":"swe-bench pro (scale seal public set)","score":59.1,"pct":0.933,"independent":true,"sourceKind":"independent","sourceUrl":"https://morphllm.com/swe-bench-pro","date":"2026-09-14"}]}},"anecdotes":[],"url":"https://gjalla.io/models/gpt-5.4/"},{"model":"gpt-5.5","vendor":"openai","tier":"deep","price":{"input":5,"output":30,"cacheRead":0.5,"cacheWrite":5,"longAbove":272000,"long":{"input":10,"output":45,"cacheRead":1,"cacheWrite":10},"source":"platform","sourceUrl":null},"meta":{"released":"2026-04-24","contextWindow":1050000,"maxOutput":128000,"modalities":["text","image"],"officialCardUrl":"https://developers.openai.com/api/docs/models/gpt-5.5"},"excelsAt":[{"point":"Described by heavy Codex users as fast, consistent, close to instructions and low-rework for routine development, preferred over 5.6 Sol for those tasks.","basis":"practitioner","sourceKind":"community","sourceUrl":"https://community.openai.com/t/please-to-keep-gpt-5-5-available-as-a-legacy-option-in-codex/1397813"}],"watchOuts":[{"point":"Retired from ChatGPT-signed-in Codex on 2026-10-14; still available via API.","basis":"vendor","sourceKind":"relayed","sourceUrl":"https://www.vincentschmalbach.com/gpt-5-5-retirement-codex/"}],"benchmarks":{"coding_agentic":{"pct":0.417,"benchmarks":[{"benchmark":"swe-bench pro","score":58.6,"pct":0.417,"independent":false,"sourceKind":"relayed","sourceUrl":"https://rits.shanghai.nyu.edu/ai/openai-releases-gpt-5-5-agentic-coding-ceiling-tops-14-benchmarks/","date":"2026-04-24"}]},"computer_use":{"pct":0.75,"benchmarks":[{"benchmark":"osworld-verified","score":78.7,"pct":0.75,"independent":false,"sourceKind":"relayed","sourceUrl":"https://rits.shanghai.nyu.edu/ai/openai-releases-gpt-5-5-agentic-coding-ceiling-tops-14-benchmarks/","date":"2026-04-24"}]}},"anecdotes":[{"claim":"Several forum users treated 5.5 as the stable coding model after 5.6 Sol felt unstable, wanting it kept available; one recommends 5.5 High for bounded implementation alongside 5.6 Sol as planner/reviewer.","taskFamily":"feature","taskDescription":null,"characteristics":[],"direction":"better","comparedTo":"gpt-5.6-sol","sourceType":"forum","sourceKind":"community","sourceUrl":"https://community.openai.com/t/5-6-sol-should-be-renamed-5-6-sol-drift-edition/1386624","date":"2026-07-15"},{"claim":"Users comparing against 5.6 say Luna medium is about on par with 5.5 medium in intelligence at lower cost, and Terra Ultra gave roughly twice 5.5's quality and speed (anecdotal, relayed).","taskFamily":"feature","taskDescription":null,"characteristics":[],"direction":"worse","comparedTo":"gpt-5.6-luna","sourceType":"reddit","sourceKind":"community","sourceUrl":"https://go.tabbit.ai/model/gpt-5-6-luna/reviews/reddit-codex-luna-default-model-coding-experience","date":null},{"claim":"A heavy Codex user kept GPT-5.5 as his all-day main model and went back to it after a week with Astra; in his logs Sol used about 2.25x GPT-5.5's tokens per session and ran far longer on the same work. Single user's impressions, no task benchmarks.","taskFamily":"agentic-long-running","taskDescription":null,"characteristics":[],"direction":"better","comparedTo":"gpt-5.6-sol","sourceType":"blog","sourceKind":"community","sourceUrl":"https://www.vincentschmalbach.com/gpt-5-5-retirement-codex/","date":"2026-09-21"},{"claim":"Several Codex Pro users asked to keep GPT-5.5 as a legacy option, saying Sol over-reasons, widens scope, spawns needless sub-agents and burns quota on ordinary tasks, while 5.5 needs little rework. One poster cites a controlled GitHub run where 5.5 used web search and finished 3 of 3 trials, unverified. Posters admit harness effects may be the cause.","taskFamily":"chore","taskDescription":null,"characteristics":[],"direction":"better","comparedTo":"gpt-5.6-sol","sourceType":"forum","sourceKind":"community","sourceUrl":"https://community.openai.com/t/please-to-keep-gpt-5-5-available-as-a-legacy-option-in-codex/1397813","date":"2026-09-15"}],"url":"https://gjalla.io/models/gpt-5.5/"},{"model":"gpt-5.6-cyber","vendor":"openai","tier":"frontier","price":{"input":12.5,"output":75,"cacheRead":1.25,"cacheWrite":15.625,"longAbove":272000,"long":{"input":25,"output":112.5,"cacheRead":2.5,"cacheWrite":31.25},"source":"platform","sourceUrl":null},"meta":{"released":"2026-08-10","contextWindow":400000,"maxOutput":128000,"modalities":["text","image"],"officialCardUrl":"https://developers.openai.com/api/docs/models/gpt-5.6-cyber"},"excelsAt":[],"watchOuts":[],"benchmarks":{},"anecdotes":[{"claim":"SpecterOps CTO, paraphrased in a third-party article, says the model finished in under a day parts of expert-level vulnerability research that earlier models had not completed after weeks of intermittent effort.","taskFamily":"research","taskDescription":null,"characteristics":[],"direction":"better","comparedTo":null,"sourceType":"vendor","sourceKind":"relayed","sourceUrl":"https://we0.ai/articles/gpt-5-6-cyber-and-daybreak","date":"2026-08-13"},{"claim":"OpenAI says the model helped find two previously unknown Chrome V8 vulnerabilities that can be chained to escape the V8 heap sandbox, verified by humans and disclosed to Google (vendor claim).","taskFamily":"research","taskDescription":null,"characteristics":[],"direction":"better","comparedTo":null,"sourceType":"vendor","sourceKind":"relayed","sourceUrl":"https://we0.ai/articles/gpt-5-6-cyber-and-daybreak","date":"2026-08-13"},{"claim":"On ExploitBench with the standard 300-round budget, GPT-5.6 Sol under Daybreak Blue did better and used tokens more efficiently than Cyber; Cyber narrowed the gap at 600 rounds because it spends larger reasoning budgets.","taskFamily":"research","taskDescription":null,"characteristics":[],"direction":"mixed","comparedTo":"gpt-5.6-sol","sourceType":"news","sourceKind":"community","sourceUrl":"https://we0.ai/articles/gpt-5-6-cyber-and-daybreak","date":"2026-08-13"},{"claim":"Positioned as the purpose-trained model that answers advanced exploit-development requests general models refuse, available only to vetted researchers through Daybreak Red; Sol under Daybreak Blue is the broader defensive-work option.","taskFamily":"operations-support","taskDescription":null,"characteristics":[],"direction":"better","comparedTo":"gpt-5.6-sol","sourceType":"news","sourceKind":"community","sourceUrl":"https://runtimewire.com/article/openai-gpt-5-6-cyber-daybreak-red","date":"2026-08-10"}],"url":"https://gjalla.io/models/gpt-5.6-cyber/"},{"model":"gpt-5.6-luna","vendor":"openai","tier":"fast","price":{"input":0.2,"output":1.2,"cacheRead":0.02,"cacheWrite":0.25,"longAbove":272000,"long":{"input":0.4,"output":1.8,"cacheRead":0.04,"cacheWrite":0.5},"source":"platform","sourceUrl":null},"meta":{"released":"2026-07-09","contextWindow":1050000,"maxOutput":128000,"modalities":["text","image"],"officialCardUrl":"https://developers.openai.com/api/docs/models/gpt-5.6-luna"},"excelsAt":[],"watchOuts":[],"benchmarks":{"coding_agentic":{"pct":0.446,"benchmarks":[{"benchmark":"artificial analysis coding agent index","score":75,"pct":0.6,"independent":true,"sourceKind":"independent","sourceUrl":"https://artificialanalysis.ai/articles/gpt-5-6-has-landed","date":"2026-07-09"},{"benchmark":"terminal-bench 2.1 (aggregator, anotherwrapper.com)","score":84.7,"pct":0.292,"independent":true,"sourceKind":"independent","sourceUrl":"https://anotherwrapper.com/tools/llm-pricing/leaderboards/terminal-bench","date":"2026-10-20"}]}},"anecdotes":[{"claim":"A r/codex poster switched their default from Sol high to Luna max for planning and brainstorming, with subagents on Luna xhigh and Sol high for plan review; says Luna medium is roughly on par with GPT-5.5 medium and as cheap as small models. Relayed secondhand; original not fetched.","taskFamily":"feature","taskDescription":null,"characteristics":[],"direction":"better","comparedTo":"gpt-5.5","sourceType":"reddit","sourceKind":"community","sourceUrl":"https://go.tabbit.ai/model/gpt-5-6-luna/reviews/reddit-codex-luna-default-model-coding-experience","date":null},{"claim":"In the same thread one user rated Luna Max the best of the 5.6 lineup for UI design, and another says Luna solved problems that GPT-5.5 xhigh could not (anecdotal).","taskFamily":"feature","taskDescription":null,"characteristics":[],"direction":"better","comparedTo":"gpt-5.5","sourceType":"reddit","sourceKind":"community","sourceUrl":"https://go.tabbit.ai/model/gpt-5-6-luna/reviews/reddit-codex-luna-default-model-coding-experience","date":null},{"claim":"Dissent in that thread: Luna Max loads very slowly versus Sol, and lower-tier models may falsely report Git operations as completed.","taskFamily":"chore","taskDescription":null,"characteristics":[],"direction":"worse","comparedTo":"gpt-5.6-sol","sourceType":"reddit","sourceKind":"community","sourceUrl":"https://go.tabbit.ai/model/gpt-5-6-luna/reviews/reddit-codex-luna-default-model-coding-experience","date":null},{"claim":"CodeRabbit positions Luna as a low-reasoning first-pass lane for summaries and lightweight prechecks in review pipelines (a positioning recommendation, no Luna scores given).","taskFamily":"code-review","taskDescription":null,"characteristics":[],"direction":"mixed","comparedTo":null,"sourceType":"blog","sourceKind":"community","sourceUrl":"https://www.coderabbit.ai/blog/gpt-5-6-sol-and-terra-benchmark","date":"2026-07-09"}],"url":"https://gjalla.io/models/gpt-5.6-luna/"},{"model":"gpt-5.6-sol","vendor":"openai","tier":"deep","price":{"input":4,"output":20,"cacheRead":0.4,"cacheWrite":5,"longAbove":272000,"long":{"input":8,"output":30,"cacheRead":0.8,"cacheWrite":10},"source":"platform","sourceUrl":null},"meta":{"released":"2026-07-09","contextWindow":1050000,"maxOutput":128000,"modalities":["text","image"],"officialCardUrl":"https://developers.openai.com/api/docs/models/gpt-5.6-sol"},"excelsAt":[{"point":"Flagship for complex reasoning, coding and agentic work, described as particularly strong at command-line and multi-step coding.","basis":"independent","sourceKind":"independent","sourceUrl":"https://openrouter.ai/openai/gpt-5.6-sol"},{"point":"Strong browsing and computer use: BrowseComp 90.4 percent and OSWorld 2.0 62.6 percent as reported by OpenAI.","basis":"vendor","sourceKind":"relayed","sourceUrl":"https://www.chatbase.co/blog/gpt-5-6-sol"},{"point":"Highest presentation Elo in Artificial Analysis's AA-Briefcase evaluation; programmatic tool calling cuts round trips.","basis":"independent","sourceKind":"independent","sourceUrl":"https://www.chatbase.co/blog/gpt-5-6-sol"},{"point":"Much stronger execution than GPT-5.5 on cross-stack debugging and notably more polished Office documents.","basis":"independent","sourceKind":"independent","sourceUrl":"https://laprompt.com/blog/ai-news/gpt-5-6-real-world-review-codex"},{"point":"High taste scores on prototypes, with better visual hierarchy and interactivity than alternatives tested.","basis":"independent","sourceKind":"independent","sourceUrl":"https://www.lennysnewsletter.com/p/how-i-ai-gpt-56-review-how-a-solo"}],"watchOuts":[{"point":"Sonar found higher bug and vulnerability density in its generated Java code, so testing and review remain necessary.","basis":"independent","sourceKind":"independent","sourceUrl":"https://www.chatbase.co/blog/gpt-5-6-sol"},{"point":"Long-context recall fell to 73.8 percent at 512K-1M tokens, and requests over 272K tokens bill higher.","basis":"independent","sourceKind":"independent","sourceUrl":"https://www.chatbase.co/blog/gpt-5-6-sol"},{"point":"Long threads degrade, subagents multiply token use, and high effort can exhaust five-hour windows.","basis":"practitioner","sourceKind":"community","sourceUrl":"https://laprompt.com/blog/ai-news/gpt-5-6-real-world-review-codex"},{"point":"Priciest in its family at $5 and $30 per million tokens; poor fit for simple classification or extraction.","basis":"independent","sourceKind":"independent","sourceUrl":"https://www.chatbase.co/blog/gpt-5-6-sol"}],"benchmarks":{"coding_agentic":{"pct":0.697,"benchmarks":[{"benchmark":"artificial analysis coding agent index","score":80,"pct":1,"independent":true,"sourceKind":"independent","sourceUrl":"https://artificialanalysis.ai/articles/gpt-5-6-has-landed","date":"2026-07-09"},{"benchmark":"deepswe v1.1","score":72.7,"pct":0.5,"independent":false,"sourceKind":"primary","sourceUrl":"https://openai.com/index/gpt-6-astra/","date":"2026-09-03"},{"benchmark":"terminal-bench 2.1","score":88.8,"pct":0.818,"independent":false,"sourceKind":"relayed","sourceUrl":"https://rits.shanghai.nyu.edu/ai/openai-launches-gpt-5-6-family-sol-terra-and-luna-reach-general-availability/","date":"2026-07-09"},{"benchmark":"terminal-bench 2.1 (aggregator, anotherwrapper.com)","score":88.8,"pct":0.854,"independent":true,"sourceKind":"independent","sourceUrl":"https://anotherwrapper.com/tools/llm-pricing/leaderboards/terminal-bench","date":"2026-10-19"},{"benchmark":"terminal-bench 4.0","score":37.3,"pct":0.312,"independent":false,"sourceKind":"primary","sourceUrl":"https://openai.com/index/gpt-6-astra/","date":"2026-09-03"}]},"reasoning":{"pct":0.833,"benchmarks":[{"benchmark":"gpqa diamond","score":94.6,"pct":0.833,"independent":false,"sourceKind":"primary","sourceUrl":"https://openai.com/index/gpt-6-astra/","date":"2026-09-03"}]}},"anecdotes":[{"claim":"On extra high in Codex, follows specs that express intent (overly detailed specs hurt), handles UX feedback better than 5.5, and sometimes makes useful out-of-scope refactors; but misses visual problems such as overlaps unless shown screenshots.","taskFamily":"feature","taskDescription":null,"characteristics":[],"direction":"mixed","comparedTo":"gpt-5.5","sourceType":"forum","sourceKind":"community","sourceUrl":"https://community.openai.com/t/gpt-5-6-sol-vs-terra-what-are-you-seeing-in-real-development-during-these-first-days/1386726","date":"2026-07-13"},{"claim":"Says Sol is not usable for serious terminal coding workflows: it loses constraints, repeats mistakes and drifts from instructions. Several replies agree, and one says it ignores architecture docs and invents services.","taskFamily":"agentic-long-running","taskDescription":null,"characteristics":[],"direction":"worse","comparedTo":"gpt-5.5","sourceType":"forum","sourceKind":"community","sourceUrl":"https://community.openai.com/t/5-6-sol-should-be-renamed-5-6-sol-drift-edition/1386624","date":"2026-07-12"},{"claim":"Finds Sol at max effort useful as a planner and reviewer but not as the hands-on coder; suggests pairing it with GPT-5.5 High for bounded implementation.","taskFamily":"code-review","taskDescription":null,"characteristics":[],"direction":"mixed","comparedTo":"gpt-5.5","sourceType":"forum","sourceKind":"community","sourceUrl":"https://community.openai.com/t/5-6-sol-should-be-renamed-5-6-sol-drift-edition/1386624","date":"2026-07-15"},{"claim":"Roundup of seven creator tests: Sol turns a defined job into a finished artifact quickly and cheaply in Codex (one game build about $4.50 vs about $14.22 for Claude Fable) but Fable was more imaginative; Ultra mode multi-agent runs can cost hundreds of dollars. Not a controlled comparison.","taskFamily":"feature","taskDescription":null,"characteristics":[],"direction":"mixed","comparedTo":"claude-fable-5","sourceType":"blog","sourceKind":"community","sourceUrl":"https://www.ai.joaoqueiros.com/blog/gpt-5-6-sol-review-roundup-codex-strengths","date":"2026-07-12"},{"claim":"CodeRabbit recommends Sol for long-horizon coding and for review passes where missing a bug is costly, but notes noisy comments (precision 31.6%, 61 nitpicks), fuzzy judgment and an occasional stuck loop (one simple change took eight turns).","taskFamily":"code-review","taskDescription":null,"characteristics":[],"direction":"mixed","comparedTo":"gpt-5.6-terra","sourceType":"blog","sourceKind":"community","sourceUrl":"https://www.coderabbit.ai/blog/gpt-5-6-sol-and-terra-benchmark","date":"2026-07-09"},{"claim":"Same-thread user running Sol at extra high for 6-8 hour days reports no weekly limit hits and says false 'fixed' claims are rare and tied to missing tools to see UX bugs, while another user saw 5.6 claim an issue resolved when it was still present.","taskFamily":"bug-fix","taskDescription":null,"characteristics":[],"direction":"mixed","comparedTo":null,"sourceType":"forum","sourceKind":"community","sourceUrl":"https://community.openai.com/t/gpt-5-6-sol-vs-terra-what-are-you-seeing-in-real-development-during-these-first-days/1386726","date":"2026-07-13"},{"claim":"Made Sol a daily driver after it won a taste-scored benchmark of PRDs, prototypes and debugging, and said it reconsidered approaches more readily than Fable.","taskFamily":null,"taskDescription":null,"characteristics":[],"direction":"better","comparedTo":"claude-fable-5","sourceType":"newsletter","sourceKind":"community","sourceUrl":"https://www.lennysnewsletter.com/p/how-i-ai-gpt-56-review-how-a-solo","date":"2026-07-13"},{"claim":"Used Codex with the Chrome tool to answer around 500 LinkedIn messages, fill forms and test web apps, though it has a green palette bias and weak voice output.","taskFamily":null,"taskDescription":null,"characteristics":[],"direction":"mixed","comparedTo":"claude-sonnet-5","sourceType":"newsletter","sourceKind":"community","sourceUrl":"https://www.lennysnewsletter.com/p/how-i-ai-gpt-56-review-how-a-solo","date":"2026-07-13"},{"claim":"Execution and Office-file polish improved over GPT-5.5, but long threads degraded and a compaction loop burned an estimated 10 to 15 percent of an allowance.","taskFamily":null,"taskDescription":null,"characteristics":[],"direction":"mixed","comparedTo":"gpt-5.5","sourceType":"blog","sourceKind":"community","sourceUrl":"https://laprompt.com/blog/ai-news/gpt-5-6-real-world-review-codex","date":"2026-07-31"},{"claim":"Sol beat Fable 5.1 on structured tasks, compression and short posts, while Fable was better on judgment-heavy slide decks and persona answers.","taskFamily":null,"taskDescription":null,"characteristics":[],"direction":"mixed","comparedTo":"claude-fable-5.1","sourceType":"blog","sourceKind":"community","sourceUrl":"https://every.to/vibe-check/fable-5-1-vibe-check","date":"2026-09-01"},{"claim":"Sol's prose scored a readability grade of 8.74 versus 6.95 for Opus 5.5, meaning more complex, harder-to-read writing.","taskFamily":null,"taskDescription":null,"characteristics":[],"direction":"worse","comparedTo":"claude-opus-5.5","sourceType":"blog","sourceKind":"community","sourceUrl":"https://every.to/vibe-check/vibe-check-opus-5-5-is-pulling-our-codex-converts-back-to-claude","date":"2026-09-22"},{"claim":"A code-review benchmark gave GPT-5.6 Sol a relative bug rate of 0.72, better than GPT-6 Sol's 1.38 (secondhand roundup).","taskFamily":null,"taskDescription":null,"characteristics":[],"direction":"better","comparedTo":"gpt-6-sol","sourceType":"blog","sourceKind":"community","sourceUrl":"https://hackernoon.com/156-real-world-examples-of-how-to-use-gpt-6-luna-and-sol","date":"2026-09-29"}],"url":"https://gjalla.io/models/gpt-5.6-sol/"},{"model":"gpt-5.6-terra","vendor":"openai","tier":"standard","price":{"input":2,"output":12,"cacheRead":0.2,"cacheWrite":2.5,"longAbove":272000,"long":{"input":4,"output":18,"cacheRead":0.4,"cacheWrite":5},"source":"platform","sourceUrl":null},"meta":{"released":"2026-07-09","contextWindow":1050000,"maxOutput":128000,"modalities":["text","image"],"officialCardUrl":"https://developers.openai.com/api/docs/models/gpt-5.6-terra"},"excelsAt":[],"watchOuts":[],"benchmarks":{"coding_agentic":{"pct":0.671,"benchmarks":[{"benchmark":"artificial analysis coding agent index","score":77,"pct":0.8,"independent":true,"sourceKind":"independent","sourceUrl":"https://artificialanalysis.ai/articles/gpt-5-6-has-landed","date":"2026-07-09"},{"benchmark":"terminal-bench 2.1 (aggregator, anotherwrapper.com)","score":87.4,"pct":0.542,"independent":true,"sourceKind":"independent","sourceUrl":"https://anotherwrapper.com/tools/llm-pricing/leaderboards/terminal-bench","date":"2026-10-19"}]}},"anecdotes":[{"claim":"A Plus user on Terra at medium says the whole weekly Codex allowance was gone in about six hours of active development, instructions were followed early then gradually forgotten, and the model made changes outside the requested scope.","taskFamily":"agentic-long-running","taskDescription":null,"characteristics":[],"direction":"worse","comparedTo":"gpt-5.6-sol","sourceType":"forum","sourceKind":"community","sourceUrl":"https://community.openai.com/t/gpt-5-6-sol-vs-terra-what-are-you-seeing-in-real-development-during-these-first-days/1386726","date":"2026-07-13"},{"claim":"Says Terra Ultra works well in Codex when tasks are explained in detail, claiming roughly twice the quality and speed of GPT-5.5.","taskFamily":"feature","taskDescription":null,"characteristics":[],"direction":"better","comparedTo":"gpt-5.5","sourceType":"forum","sourceKind":"community","sourceUrl":"https://community.openai.com/t/5-6-sol-should-be-renamed-5-6-sol-drift-edition/1386624","date":"2026-07-14"},{"claim":"In CodeRabbit's tests Terra solved 40.7% of coding tasks versus 63.7% for Sol while using about 2.6x the output tokens per task, and was weaker at review; CodeRabbit sees it as fit for bounded repeatable work and first-pass triage, not as a cost saver by default.","taskFamily":"feature","taskDescription":null,"characteristics":[],"direction":"worse","comparedTo":"gpt-5.6-sol","sourceType":"blog","sourceKind":"community","sourceUrl":"https://www.coderabbit.ai/blog/gpt-5-6-sol-and-terra-benchmark","date":"2026-07-09"},{"claim":"A Codex Reddit thread (relayed secondhand by an aggregator) says Terra burned nearly as much usage as Sol without being better, and that Sol at a lower setting is the better value; Terra described as poorly positioned.","taskFamily":"feature","taskDescription":null,"characteristics":[],"direction":"worse","comparedTo":"gpt-5.6-sol","sourceType":"reddit","sourceKind":"community","sourceUrl":"https://go.tabbit.ai/model/gpt-5-6-luna/reviews/reddit-codex-luna-default-model-coding-experience","date":null},{"claim":"In a creator review roundup, Terra was preferred over Sol for writing direct, concise PRDs.","taskFamily":"writing-content","taskDescription":null,"characteristics":[],"direction":"better","comparedTo":"gpt-5.6-sol","sourceType":"blog","sourceKind":"community","sourceUrl":"https://www.ai.joaoqueiros.com/blog/gpt-5-6-sol-review-roundup-codex-strengths","date":"2026-07-12"},{"claim":"Reports GPT-5.4 at medium beat 5.6 Terra at story writing, with Terra having trouble tracking objects and character knowledge.","taskFamily":"writing-content","taskDescription":null,"characteristics":[],"direction":"worse","comparedTo":"gpt-5.4","sourceType":"forum","sourceKind":"community","sourceUrl":"https://community.openai.com/t/5-6-sol-should-be-renamed-5-6-sol-drift-edition/1386624","date":"2026-07-13"}],"url":"https://gjalla.io/models/gpt-5.6-terra/"},{"model":"gpt-6-astra","vendor":"openai","tier":"frontier","price":{"input":10,"output":50,"cacheRead":1,"cacheWrite":12.5,"longAbove":272000,"long":{"input":20,"output":75,"cacheRead":2,"cacheWrite":25},"source":"platform","sourceUrl":null},"meta":{"released":"2026-09-03","contextWindow":1050000,"maxOutput":128000,"modalities":["text","image"],"officialCardUrl":"https://openai.com/index/gpt-6-astra/"},"excelsAt":[{"point":"Computer use for forms, CRM updates, web research and data analysis; OSWorld 2.0 72.6 percent in about 40 minutes.","basis":"vendor","sourceKind":"primary","sourceUrl":"https://openai.com/da-DK/index/gpt-6-astra/"},{"point":"Produces slides, spreadsheets and documents that follow existing templates and a user's style.","basis":"vendor","sourceKind":"primary","sourceUrl":"https://openai.com/da-DK/index/gpt-6-astra/"},{"point":"OpenAI's strongest coding model, asks clarifying questions only when answers matter, and handles new instructions mid-task.","basis":"vendor","sourceKind":"primary","sourceUrl":"https://openai.com/da-DK/index/gpt-6-astra/"},{"point":"Top results on hard science and math: GPQA Diamond 96.0 and FrontierMath Tier 4 97.6 percent.","basis":"vendor","sourceKind":"primary","sourceUrl":"https://openai.com/da-DK/index/gpt-6-astra/"},{"point":"Strong at 3D and CAD work; BenchCAD 95.9 percent and a Blender house built in about 40 minutes.","basis":"independent","sourceKind":"independent","sourceUrl":"https://hierarchy.ch/en/news/gpt-6-astra-vs-claude-fable-5-1"},{"point":"Token-efficient with parallel swarm mode up to 40 subagents; one reviewer saw about 4x more work per dollar.","basis":"independent","sourceKind":"independent","sourceUrl":"https://finance.biggo.com/news/eb22b1f124ad5b97"},{"point":"First model to score 43/43 on Plotly's analytics test, finding all five hidden-data issues.","basis":"practitioner","sourceKind":"community","sourceUrl":"https://plotly.com/blog/introducing-dataanalyticsbench/"},{"point":"Highest functional and secure-code pass rates among OpenAI models on Endor Labs' Codex benchmark (82.1% functional, 34.6% secure), with zero confirmed cheating (vendor-run benchmark).","basis":"independent","sourceKind":"independent","sourceUrl":"https://www.endorlabs.com/learn/gpt-6-sol-on-codex-average-scores-quarter-the-cost"}],"watchOuts":[{"point":"Quality swings between peaks and failures; it ignored explicit instructions, made unwanted changes and merged an unfinished PR.","basis":"practitioner","sourceKind":"community","sourceUrl":"https://finance.biggo.com/news/eb22b1f124ad5b97"},{"point":"Trails Fable 5.1 on coding finish and UI polish; may stop early or leave features looking done but broken.","basis":"practitioner","sourceKind":"community","sourceUrl":"https://hierarchy.ch/en/news/gpt-6-astra-vs-claude-fable-5-1"},{"point":"Extra safety checks can halt legitimate work, including defensive security; advanced exploit tasks are refused.","basis":"vendor","sourceKind":"primary","sourceUrl":"https://openai.com/da-DK/index/gpt-6-astra/"},{"point":"Ranked tenth on Vals AI's Excel modeling benchmark, weaker than Claude models on LBO and DCF tasks.","basis":"independent","sourceKind":"independent","sourceUrl":"https://www.vals.ai/benchmarks/emb"},{"point":"Highest cost on that test (about $1,161 per person per year estimate) with Sol matching it for about a tenth.","basis":"practitioner","sourceKind":"community","sourceUrl":"https://plotly.com/blog/introducing-dataanalyticsbench/"},{"point":"Cost: about $2.34 per task on Endor's Codex benchmark vs $0.52 for GPT-6 Sol at only modestly lower security scores.","basis":"independent","sourceKind":"independent","sourceUrl":"https://www.endorlabs.com/learn/gpt-6-sol-on-codex-average-scores-quarter-the-cost"}],"benchmarks":{"coding":{"pct":0.957,"benchmarks":[{"benchmark":"lmarena code arena webdev","score":1777.2,"pct":0.957,"independent":true,"sourceKind":"independent","sourceUrl":"https://lmarena.ai/leaderboard/webdev","date":null}]},"coding_agentic":{"pct":0.553,"benchmarks":[{"benchmark":"artificial analysis coding agent index","score":62,"pct":0.2,"independent":true,"sourceKind":"independent","sourceUrl":"https://artificialanalysis.ai/articles/benchmarking-gpt-6-astra","date":"2026-09-09"},{"benchmark":"deepswe v1.1","score":74.1,"pct":0.7,"independent":false,"sourceKind":"primary","sourceUrl":"https://openai.com/index/gpt-6-astra/","date":"2026-09-03"},{"benchmark":"terminal-bench 2.1 (aggregator, anotherwrapper.com)","score":87.3,"pct":0.5,"independent":true,"sourceKind":"independent","sourceUrl":"https://anotherwrapper.com/tools/llm-pricing/leaderboards/terminal-bench","date":"2026-10-19"},{"benchmark":"terminal-bench 4.0","score":59,"pct":0.812,"independent":true,"sourceKind":"independent","sourceUrl":"https://artificialanalysis.ai/articles/benchmarking-gpt-6-astra","date":"2026-09-09"}]},"knowledge":{"pct":0.692,"benchmarks":[{"benchmark":"humanity's last exam","score":57.2,"pct":0.692,"independent":false,"sourceKind":"primary","sourceUrl":"https://openai.com/index/gpt-6-astra/","date":"2026-09-03"}]},"overall":{"pct":0.696,"benchmarks":[{"benchmark":"artificial analysis intelligence index","score":53,"pct":0.696,"independent":true,"sourceKind":"independent","sourceUrl":"https://artificialanalysis.ai/articles/benchmarking-gpt-6-astra","date":"2026-09-09"}]},"reasoning":{"pct":1,"benchmarks":[{"benchmark":"gpqa diamond","score":96,"pct":1,"independent":false,"sourceKind":"primary","sourceUrl":"https://openai.com/index/gpt-6-astra/","date":"2026-09-03"}]},"tool_use":{"pct":1,"benchmarks":[{"benchmark":"automationbench-aa","score":69,"pct":1,"independent":true,"sourceKind":"independent","sourceUrl":"https://artificialanalysis.ai/articles/benchmarking-gpt-6-astra","date":"2026-09-09"}]}},"anecdotes":[{"claim":"Gave Astra a roughly 3-hour goal-mode refactor (split dashboard into its own repo plus new design system). It handled implementation, debugging and verification with little supervision but the first result dropped the author's favourite feature; reaching a verified result took about 12h40 elapsed and exhausted his weekly Pro limit three times. Author thinks UI work may not be its strength and that it is only worth it for complex problems or big budgets.","taskFamily":"agentic-long-running","taskDescription":null,"characteristics":[],"direction":"mixed","comparedTo":null,"sourceType":"blog","sourceKind":"community","sourceUrl":"https://blog.vibecodingdad.com/p/i-gave-gpt-6-astra-a-3-hour-coding","date":"2026-09-08"},{"claim":"On one ordinary coding task in an existing codebase Astra used about 30% of a weekly limit on a $100 plan, reasoned carefully but was not dramatically better than GPT-5.6 Sol and still introduced its own bugs; author keeps Sol as default for 80-90% of coding and escalates to Astra only for large-codebase redesigns or hard bugs.","taskFamily":"feature","taskDescription":null,"characteristics":[],"direction":"worse","comparedTo":"gpt-5.6-sol","sourceType":"blog","sourceKind":"community","sourceUrl":"https://bilimgram.medium.com/gpt-6-astra-burned-30-of-my-weekly-limit-on-one-simple-coding-task-e229bec92d08","date":"2026-09-05"},{"claim":"Recommends Astra at medium effort by default: citing a dev.to coding test, medium was cheaper and faster than Sol at high, while Astra at high was slower and its review pass missed a startup bug that the medium run caught. Suggests medium for diff review and extraction, high for long autonomous tool-call runs. Secondhand result.","taskFamily":"code-review","taskDescription":null,"characteristics":[],"direction":"mixed","comparedTo":"gpt-5.6-sol","sourceType":"blog","sourceKind":"community","sourceUrl":"https://ilikekillnerds.com/2026/09/06/gpt-5-6-sol-to-gpt-6-astra-reasoning-effort/","date":"2026-09-06"},{"claim":"Argues Astra suits an escalation role: repository-wide changes, terminal-heavy agents, SRE/debugging, long investigations and multi-tool workflows; not worth the price for classification, short rewriting or routine summaries. Post says it is adapted from a gateway vendor's research, so treat as opinion with commercial interest.","taskFamily":"agentic-long-running","taskDescription":null,"characteristics":[],"direction":"better","comparedTo":"gpt-5.6-sol","sourceType":"blog","sourceKind":"community","sourceUrl":"https://dev.to/mayacollins1/gpt-6-astra-is-expensive-per-token-im-not-sure-thats-the-right-metric-1621","date":"2026-09-08"},{"claim":"Found Astra overkill as an orchestrator agent: it wasted most of its effort getting sidetracked. Also said it burns credits heavily (a second user, Supranova, made the same credit complaint on 2026-09-29).","taskFamily":"agentic-long-running","taskDescription":null,"characteristics":[],"direction":"worse","comparedTo":null,"sourceType":"forum","sourceKind":"community","sourceUrl":"https://community.openai.com/t/announcing-gpt-6-sol-and-gpt-6-luna-in-the-api-codex-and-chatgpt/1399925","date":"2026-09-23"},{"claim":"Forum user reports refusals on cyber requests rose from roughly half to about 94% with Astra, a regression for security research work.","taskFamily":"research","taskDescription":null,"characteristics":[],"direction":"worse","comparedTo":null,"sourceType":"forum","sourceKind":"community","sourceUrl":"https://community.openai.com/t/introducing-gpt-6-astra-the-most-intelligent-and-aligned-model-in-the-world/1394703","date":"2026-09-03"},{"claim":"Reviewer preferred Fable 5.1 for reliable production code but liked Astra's token efficiency, computer use and 3D rendering; Astra ignored instructions and swung in quality.","taskFamily":null,"taskDescription":null,"characteristics":[],"direction":"mixed","comparedTo":"claude-fable-5.1","sourceType":"blog","sourceKind":"community","sourceUrl":"https://finance.biggo.com/news/eb22b1f124ad5b97","date":"2026-09-07"},{"claim":"Astra rebuilt a house from a floor plan in Blender and tested its own Unreal export, but stopped early on coding and left features unfinished.","taskFamily":null,"taskDescription":null,"characteristics":[],"direction":"mixed","comparedTo":"claude-fable-5.1","sourceType":"blog","sourceKind":"community","sourceUrl":"https://hierarchy.ch/en/news/gpt-6-astra-vs-claude-fable-5-1","date":"2026-09-08"},{"claim":"Astra's app clones were poor and chess play was blunder after blunder, yet computer-use work like book recommendations and medical-record review was strong.","taskFamily":null,"taskDescription":null,"characteristics":[],"direction":"mixed","comparedTo":"claude-fable-5.1","sourceType":"blog","sourceKind":"community","sourceUrl":"https://medium.com/jonathans-musings/a-short-review-of-openais-gpt-6-astra-091fa5beccf9","date":"2026-09-05"},{"claim":"Astra won most head-to-heads with Fable 5.1 on coding and reasoning because it understood intent, but still needed oversight.","taskFamily":null,"taskDescription":null,"characteristics":[],"direction":"better","comparedTo":"claude-fable-5.1","sourceType":"blog","sourceKind":"community","sourceUrl":"https://www.mindstudio.ai/blog/gpt-6-astra-first-impressions","date":"2026-09-13"},{"claim":"Astra was a stronger editor and handled timed tasks better than Opus 5.5, though Opus was ahead on a latency-budget coding task.","taskFamily":null,"taskDescription":null,"characteristics":[],"direction":"better","comparedTo":"claude-opus-5.5","sourceType":"blog","sourceKind":"community","sourceUrl":"https://every.to/vibe-check/vibe-check-opus-5-5-is-pulling-our-codex-converts-back-to-claude","date":"2026-09-22"},{"claim":"An Astra reviewer settled in zero to two review loops versus five to seven with other OpenAI models, and was less nitpicky than Fable.","taskFamily":null,"taskDescription":null,"characteristics":[],"direction":"better","comparedTo":"claude-fable-5.1","sourceType":"hacker-news","sourceKind":"community","sourceUrl":"https://news.ycombinator.com/item?id=49881850","date":null},{"claim":"For one creator's business deliverables (deck, financials, dashboard), Astra produced functional but plain output, with a 17-slide deck and mostly basic sum formulas in Sheets. The post reports no Opus result for this task. Subjective, single creator, vendor blog.","taskFamily":"data-analysis","taskDescription":null,"characteristics":[],"direction":"mixed","comparedTo":null,"sourceType":"blog","sourceKind":"community","sourceUrl":"https://www.mindstudio.ai/blog/opus-5-5-vs-gpt-6-astra","date":"2026-09-24"},{"claim":"Astra was the first model to ace Plotly's 43-question analytics test (43/43) but was the costliest. Single run, private questions, one harness.","taskFamily":"data-analysis","taskDescription":null,"characteristics":[],"direction":"better","comparedTo":"claude-opus-5.5","sourceType":"blog","sourceKind":"community","sourceUrl":"https://plotly.com/blog/introducing-dataanalyticsbench/","date":"2026-10-14"}],"url":"https://gjalla.io/models/gpt-6-astra/"},{"model":"gpt-6-luna","vendor":"openai","tier":"fast","price":{"input":0.1,"output":0.5,"cacheRead":0.01,"cacheWrite":0.125,"longAbove":272000,"long":{"input":0.2,"output":0.75,"cacheRead":0.02,"cacheWrite":0.25},"source":"platform","sourceUrl":null},"meta":{"released":"2026-09-22","contextWindow":1000000,"maxOutput":128000,"modalities":["text","image"],"officialCardUrl":"https://openai.com/index/introducing-gpt-6-sol-and-luna/"},"excelsAt":[{"point":"Aimed at repetitive high-volume work such as invoice extraction, request classification and structured summaries.","basis":"vendor","sourceKind":"relayed","sourceUrl":"https://www.smartworld.it/guide/gpt-6-astra-sol-luna-guida-ufficiale-openai-tre-modelli.html"},{"point":"Lowest price in the family at $0.10 input and $0.50 output per million tokens.","basis":"vendor","sourceKind":"primary","sourceUrl":"https://openai.com/index/introducing-gpt-6-sol-and-luna/"},{"point":"AutomationBench up 5.4 points on its predecessor at 58 percent lower cost per task.","basis":"vendor","sourceKind":"primary","sourceUrl":"https://openai.com/index/introducing-gpt-6-sol-and-luna/"},{"point":"Cost per index task about 62 percent lower than GPT-5.6 Luna with a near-identical Intelligence Index score.","basis":"independent","sourceKind":"independent","sourceUrl":"https://www.orcarouter.ai/blog/gpt-6-luna-vs-gpt-5-6-luna"},{"point":"Works well for structured deterministic tasks like intake categorization, field sanitization and JSON or query generation.","basis":"independent","sourceKind":"independent","sourceUrl":"https://www.layer3labs.io/guides/gpt-6-luna-review"},{"point":"Max-effort hallucination rate on AA-Omniscience fell from 93% to 77% and the index rose from -10 to 1, at about 60% lower cost per index task.","basis":"independent","sourceKind":"independent","sourceUrl":"https://artificialanalysis.ai/articles/gpt-6-sol-and-luna-push-the-cost-efficiency-frontier"}],"watchOuts":[{"point":"Knowledge-work deliverables regressed versus GPT-5.6 Luna (AA-Briefcase down about 46 Elo), and output tokens rose about 23 percent.","basis":"independent","sourceKind":"independent","sourceUrl":"https://www.orcarouter.ai/blog/gpt-6-luna-vs-gpt-5-6-luna"},{"point":"Drifts on multi-step, multi-file software and long computer-use chains that need state tracking.","basis":"independent","sourceKind":"independent","sourceUrl":"https://www.layer3labs.io/guides/gpt-6-luna-review"},{"point":"In OpenAI's test it failed to disclose a broken search tool 28.7 percent of the time.","basis":"vendor","sourceKind":"primary","sourceUrl":"https://openai.com/index/introducing-gpt-6-1-sol/"},{"point":"One report found Kubernetes manifests passed linting but crash-looped on a real cluster in all three tries.","basis":"practitioner","sourceKind":"community","sourceUrl":"https://hackernoon.com/156-real-world-examples-of-how-to-use-gpt-6-luna-and-sol"},{"point":"GDPval-AA dropped about 75 Elo vs GPT-5.6 Luna and SWE-Atlas-QnA fell from 49% to 44%.","basis":"independent","sourceKind":"independent","sourceUrl":"https://artificialanalysis.ai/articles/gpt-6-sol-and-luna-push-the-cost-efficiency-frontier"}],"benchmarks":{"coding_agentic":{"pct":0.081,"benchmarks":[{"benchmark":"deepswe v1.1","score":66.6,"pct":0.1,"independent":false,"sourceKind":"primary","sourceUrl":"https://openai.com/index/introducing-gpt-6-sol-and-luna/","date":"2026-09-22"},{"benchmark":"terminal-bench 4.0","score":13,"pct":0.062,"independent":true,"sourceKind":"independent","sourceUrl":"https://artificialanalysis.ai/models/comparisons/gpt-6-luna-vs-gpt-6-sol","date":null}]},"knowledge":{"pct":0.154,"benchmarks":[{"benchmark":"humanity's last exam","score":39,"pct":0.154,"independent":true,"sourceKind":"independent","sourceUrl":"https://artificialanalysis.ai/models/comparisons/gpt-6-luna-vs-gpt-6-sol","date":null}]},"overall":{"pct":0,"benchmarks":[{"benchmark":"artificial analysis intelligence index v4.3.2","score":38,"pct":0,"independent":true,"sourceKind":"independent","sourceUrl":"https://artificialanalysis.ai/models/comparisons/gpt-6-luna-vs-gpt-6-sol","date":null},{"benchmark":"vals index","score":51.22,"pct":0,"independent":true,"sourceKind":"independent","sourceUrl":"https://www.vals.ai/models/openai_gpt-6-luna","date":"2026-09-22"}]},"tool_use":{"pct":0,"benchmarks":[{"benchmark":"automationbench-aa","score":53,"pct":0,"independent":true,"sourceKind":"independent","sourceUrl":"https://artificialanalysis.ai/models/comparisons/gpt-6-luna-vs-gpt-6-sol","date":null}]}},"anecdotes":[{"claim":"In a poster's three-task Codex test, Luna at high passed all three tasks for about $0.002 each, including the dependency scheduler that GPT-5.6 Luna failed; poster concluded it looks like a better model as well as a 50% cheaper one.","taskFamily":"feature","taskDescription":null,"characteristics":[],"direction":"better","comparedTo":"gpt-5.6-luna","sourceType":"forum","sourceKind":"community","sourceUrl":"https://community.openai.com/t/announcing-gpt-6-sol-and-gpt-6-luna-in-the-api-codex-and-chatgpt/1399925","date":"2026-09-22"},{"claim":"A developer runs a cheaper reverse-orchestration workflow with Luna and 6.1 Sol handling reviews and says it works very well.","taskFamily":"code-review","taskDescription":null,"characteristics":[],"direction":"better","comparedTo":"gpt-6-astra","sourceType":"forum","sourceKind":"community","sourceUrl":"https://community.openai.com/t/announcing-gpt-6-sol-and-gpt-6-luna-in-the-api-codex-and-chatgpt/1399925","date":"2026-10-08"},{"claim":"Switching an invoice pipeline to Luna lifted field accuracy from 82.5 to roughly 93-95 percent versus gpt-4o and cut cost to about $0.0044 per invoice.","taskFamily":null,"taskDescription":null,"characteristics":[],"direction":"better","comparedTo":"gpt-4o","sourceType":"github","sourceKind":"community","sourceUrl":"https://github.com/Juamp1Sch/Facturear/pull/37","date":"2026-09-25"},{"claim":"Luna matched Sol and its predecessor on a Dijkstra visualizer but produced a confusing negative-infinity table on a broken negative-weight graph.","taskFamily":null,"taskDescription":null,"characteristics":[],"direction":"mixed","comparedTo":"gpt-6-sol","sourceType":"blog","sourceKind":"community","sourceUrl":"https://www.datacamp.com/blog/gpt-6-sol-and-luna","date":"2026-09-23"},{"claim":"Luna wrote Kubernetes manifests that passed linting but all crash-looped on a real k3s cluster.","taskFamily":null,"taskDescription":null,"characteristics":[],"direction":"worse","comparedTo":null,"sourceType":"blog","sourceKind":"community","sourceUrl":"https://hackernoon.com/156-real-world-examples-of-how-to-use-gpt-6-luna-and-sol","date":"2026-09-29"},{"claim":"A whisky-bottle classifier matched the old Luna's 79 of 105 correct while cutting cost 31 percent, though runs were slower (secondhand roundup).","taskFamily":null,"taskDescription":null,"characteristics":[],"direction":"mixed","comparedTo":"gpt-5.6-luna","sourceType":"blog","sourceKind":"community","sourceUrl":"https://hackernoon.com/156-real-world-examples-of-how-to-use-gpt-6-luna-and-sol","date":"2026-09-29"},{"claim":"A developer estimated a cache-heavy workload would cost about $184 a month on Luna versus roughly $40 on a cheaper rival (secondhand roundup).","taskFamily":null,"taskDescription":null,"characteristics":[],"direction":"worse","comparedTo":null,"sourceType":"blog","sourceKind":"community","sourceUrl":"https://hackernoon.com/156-real-world-examples-of-how-to-use-gpt-6-luna-and-sol","date":"2026-09-29"}],"url":"https://gjalla.io/models/gpt-6-luna/"},{"model":"gpt-6-sol","vendor":"openai","tier":"standard","price":{"input":2,"output":10,"cacheRead":0.2,"cacheWrite":2.5,"longAbove":272000,"long":{"input":4,"output":15,"cacheRead":0.4,"cacheWrite":5},"source":"platform","sourceUrl":null},"meta":{"released":"2026-09-22","contextWindow":1050000,"maxOutput":128000,"modalities":["text","image"],"officialCardUrl":"https://openai.com/index/introducing-gpt-6-sol-and-luna/"},"excelsAt":[{"point":"Hallucination rate on Artificial Analysis AA-Omniscience dropped from 92% (GPT-5.6 Sol) to 60%, and Terminal-Bench 4.0 rose to 43-44% from 37-40%.","basis":"independent","sourceKind":"independent","sourceUrl":"https://artificialanalysis.ai/articles/gpt-6-sol-and-luna-push-the-cost-efficiency-frontier"}],"watchOuts":[{"point":"Knowledge-work deliverables regressed: GDPval-AA fell about 100 Elo vs GPT-5.6 Sol, which Artificial Analysis attributes to weaker presentation and omitted rubric elements.","basis":"independent","sourceKind":"independent","sourceUrl":"https://artificialanalysis.ai/articles/gpt-6-sol-and-luna-push-the-cost-efficiency-frontier"},{"point":"Security-focused tests lag Astra: Endor Labs SecPass 25.1% vs 34.6%, and Sonar found 285 vulnerabilities per million Java lines vs 178 for Astra (Endor is a security vendor testing its own benchmark).","basis":"independent","sourceKind":"independent","sourceUrl":"https://www.endorlabs.com/learn/gpt-6-sol-on-codex-average-scores-quarter-the-cost"},{"point":"System card reports statistically significant regressions in some disallowed-content categories (e.g. self-harm) vs GPT-5.6 (secondary report of the OpenAI system card).","basis":"vendor","sourceKind":"relayed","sourceUrl":"https://andrew.ooo/answers/what-changed-latest-openai-model-release-gpt-6-october-2026/"}],"benchmarks":{"coding":{"pct":0.696,"benchmarks":[{"benchmark":"lmarena code arena webdev","score":1683,"pct":0.696,"independent":true,"sourceKind":"independent","sourceUrl":"https://arena.ai/leaderboard/code/webdev","date":"2026-10-19"}]},"coding_agentic":{"pct":0.205,"benchmarks":[{"benchmark":"artificial analysis coding agent index","score":57,"pct":0,"independent":true,"sourceKind":"independent","sourceUrl":"https://codingfleet.com/blog/gpt-6-sol-vs-gpt-5-6-sol/","date":"2026-09-22"},{"benchmark":"deepswe v1.1","score":68.8,"pct":0.3,"independent":false,"sourceKind":"primary","sourceUrl":"https://openai.com/index/introducing-gpt-6-sol-and-luna/","date":"2026-09-22"},{"benchmark":"terminal-bench 2.1 (aggregator, anotherwrapper.com)","score":83.2,"pct":0.021,"independent":true,"sourceKind":"independent","sourceUrl":"https://anotherwrapper.com/tools/llm-pricing/leaderboards/terminal-bench","date":"2026-10-20"},{"benchmark":"terminal-bench 4.0","score":44,"pct":0.5,"independent":true,"sourceKind":"independent","sourceUrl":"https://artificialanalysis.ai/models/comparisons/gpt-6-luna-vs-gpt-6-sol","date":null}]},"knowledge":{"pct":0.538,"benchmarks":[{"benchmark":"humanity's last exam","score":48,"pct":0.538,"independent":true,"sourceKind":"independent","sourceUrl":"https://artificialanalysis.ai/models/comparisons/gpt-6-luna-vs-gpt-6-sol","date":null}]},"overall":{"pct":0.25,"benchmarks":[{"benchmark":"artificial analysis intelligence index v4.3.2","score":48,"pct":0.333,"independent":true,"sourceKind":"independent","sourceUrl":"https://artificialanalysis.ai/models/comparisons/gpt-6-luna-vs-gpt-6-sol","date":null},{"benchmark":"vals index","score":57.54,"pct":0.167,"independent":true,"sourceKind":"independent","sourceUrl":"https://www.vals.ai/models/openai_gpt-6-sol","date":"2026-09-22"}]},"tool_use":{"pct":0.333,"benchmarks":[{"benchmark":"automationbench-aa","score":62,"pct":0.333,"independent":true,"sourceKind":"independent","sourceUrl":"https://artificialanalysis.ai/models/comparisons/gpt-6-luna-vs-gpt-6-sol","date":null}]}},"anecdotes":[{"claim":"Several commenters found GPT-6 Sol worse than 5.6 Sol: one spent a day fixing its code before switching back to 5.6, another saw degraded results when refactoring, and a third calls it a regression. Comments under a launch write-up, so anecdotal.","taskFamily":"refactor","taskDescription":null,"characteristics":[],"direction":"worse","comparedTo":"gpt-5.6-sol","sourceType":"blog","sourceKind":"community","sourceUrl":"https://handyai.substack.com/p/model-drop-gpt-61-sol","date":"2026-09-29"},{"claim":"On a poster's three-task Codex test, Sol at high passed all three at roughly $0.02-0.06 per task, while GPT-5.6 Luna had failed the dependency-scheduler task.","taskFamily":"feature","taskDescription":null,"characteristics":[],"direction":"better","comparedTo":"gpt-5.6-luna","sourceType":"forum","sourceKind":"community","sourceUrl":"https://community.openai.com/t/announcing-gpt-6-sol-and-gpt-6-luna-in-the-api-codex-and-chatgpt/1399925","date":"2026-09-22"},{"claim":"Reports that on AutomationBench-style workflow problems GPT-6 Sol at max reasoning made no headway and produced worse output at higher cost than lower effort.","taskFamily":"operations-support","taskDescription":null,"characteristics":[],"direction":"worse","comparedTo":null,"sourceType":"forum","sourceKind":"community","sourceUrl":"https://community.openai.com/t/announcing-gpt-6-sol-and-gpt-6-luna-in-the-api-codex-and-chatgpt/1399925","date":"2026-09-22"},{"claim":"Calls Sol at high decent for agent orchestration but says it needs a larger context window.","taskFamily":"agentic-long-running","taskDescription":null,"characteristics":[],"direction":"mixed","comparedTo":null,"sourceType":"forum","sourceKind":"community","sourceUrl":"https://community.openai.com/t/announcing-gpt-6-sol-and-gpt-6-luna-in-the-api-codex-and-chatgpt/1399925","date":"2026-09-23"},{"claim":"Analysis built on Artificial Analysis data: coding is the one clear gain over 5.6 Sol (Coding Agent Index 57 vs 55, similar cost-halving) while knowledge-work deliverables regressed (GDPval-AA about 100 Elo lower, weaker presentation); calls the launch a price cut with roughly level capability.","taskFamily":"writing-content","taskDescription":null,"characteristics":[],"direction":"worse","comparedTo":"gpt-5.6-sol","sourceType":"blog","sourceKind":"community","sourceUrl":"https://codingfleet.com/blog/gpt-6-sol-vs-gpt-5-6-sol/","date":"2026-09-22"},{"claim":"Codex run on 179 security-sensitive tasks: Sol was the cheapest run Endor Labs had measured ($0.52 per task vs $2.34 for Astra) and improved functional and security pass rates over GPT-5.6 Sol, but trailed Astra on both (72.1% vs 82.1% functional).","taskFamily":null,"taskDescription":null,"characteristics":[],"direction":"mixed","comparedTo":"gpt-6-astra","sourceType":"blog","sourceKind":"community","sourceUrl":"https://www.endorlabs.com/learn/gpt-6-sol-on-codex-average-scores-quarter-the-cost","date":"2026-10-01"},{"claim":"Sonar's Java analysis found Sol passed 83.09% of tests (1.1 points above GPT-5.6 Sol) but its vulnerability density was 60% higher than Astra's and it wrote about 27% less code with sparse comments.","taskFamily":null,"taskDescription":null,"characteristics":[],"direction":"mixed","comparedTo":"gpt-6-astra","sourceType":"blog","sourceKind":"community","sourceUrl":"https://www.sonarsource.com/blog/open-ai-gpt-6-sol-evaluation/","date":"2026-09-24"}],"url":"https://gjalla.io/models/gpt-6-sol/"},{"model":"gpt-6.1-sol","vendor":"openai","tier":"standard","price":{"input":2,"output":10,"cacheRead":0.1,"cacheWrite":2.5,"longAbove":272000,"long":{"input":4,"output":15,"cacheRead":0.2,"cacheWrite":5},"source":"platform","sourceUrl":null},"meta":{"released":"2026-09-29","contextWindow":1050000,"maxOutput":128000,"modalities":["text","image"],"officialCardUrl":"https://openai.com/index/introducing-gpt-6-1-sol/"},"excelsAt":[{"point":"Near-Astra performance on agentic coding, computer use and professional work at one-fifth of Astra's price ($2 and $10 per million).","basis":"vendor","sourceKind":"primary","sourceUrl":"https://openai.com/index/introducing-gpt-6-1-sol/"},{"point":"Strong at multi-step business workflows and scientific work, with cheap cached input at $0.10 per million tokens.","basis":"vendor","sourceKind":"primary","sourceUrl":"https://openai.com/index/introducing-gpt-6-1-sol/"},{"point":"Highest Terminal-Bench 4.0 score in its comparison table (58.18 percent) and top code-migration reviewer rating (4.5 of 5).","basis":"independent","sourceKind":"independent","sourceUrl":"https://www.gradually.ai/en/ai-models/gpt-6.1-sol/"},{"point":"Follows specs literally and writes the smallest code, suiting high-volume agentic coding with precise requirements.","basis":"independent","sourceKind":"independent","sourceUrl":"https://thomas-wiegold.com/blog/gpt-6-1-sol-review/"},{"point":"Fewer alignment failures than GPT-6 Sol, such as respecting restrictions and disclosing broken tools.","basis":"vendor","sourceKind":"primary","sourceUrl":"https://openai.com/index/introducing-gpt-6-1-sol/"},{"point":"Perfect 43/43 on Plotly's analytics test at about 10 percent of Astra's cost, finding all five hidden-data issues.","basis":"practitioner","sourceKind":"community","sourceUrl":"https://plotly.com/blog/introducing-dataanalyticsbench/"},{"point":"On Artificial Analysis, accuracy and knowledge reliability improved sharply over GPT-6 Sol (AA-Omniscience 41.5 vs 27.1; hallucination rate 54% vs 60%) at about $0.72 per index task vs $3.26 for Astra (relayed via a reseller blog).","basis":"independent","sourceKind":"independent","sourceUrl":"https://www.orcarouter.ai/blog/gpt-6-1-sol-independent-benchmark"},{"point":"Sonar's Java analysis shows vulnerability density down about 32% vs GPT-6 Sol (193 vs 285 per million lines) with pass rate 85.29%, just under Astra's 85.85%.","basis":"independent","sourceKind":"independent","sourceUrl":"https://www.sonarsource.com/blog/openai-gpt-6-1-sol-evaluation/"}],"watchOuts":[{"point":"Slow, about 51 tokens per second at max effort; literal interpretation means prompts need more detail.","basis":"independent","sourceKind":"independent","sourceUrl":"https://thomas-wiegold.com/blog/gpt-6-1-sol-review/"},{"point":"Weak on cybersecurity: 39.29 percent on CyberBench with 0 percent on proof-of-concept tasks.","basis":"independent","sourceKind":"independent","sourceUrl":"https://www.gradually.ai/en/ai-models/gpt-6.1-sol/"},{"point":"Astra remains better for the hardest science and computer-use work; Sol trails by about 2 points on OSWorld 2.0.","basis":"vendor","sourceKind":"primary","sourceUrl":"https://openai.com/index/introducing-gpt-6-1-sol/"},{"point":"Heavy tasks can be costly and slow, for example about $6.51 and 96 minutes per code migration at max effort.","basis":"independent","sourceKind":"independent","sourceUrl":"https://www.gradually.ai/en/ai-models/gpt-6.1-sol/"},{"point":"Cost per run on small bug fixes was lower than Argon and Sonnet 5.5 in one user's eval, but see Argon note on pi cost reporting.","basis":"practitioner","sourceKind":"community","sourceUrl":"https://news.ycombinator.com/item?id=50073879"},{"point":"Writes much more code than GPT-6 Sol (about 39% more lines in Sonar's Java run); total bugs and code smells rose in raw count even though per-line density fell, so review load grows.","basis":"independent","sourceKind":"independent","sourceUrl":"https://www.sonarsource.com/blog/openai-gpt-6-1-sol-evaluation/"},{"point":"Trails Astra on Artificial Analysis Terminal-Bench 4.0 (56.1 vs 59.1), SciCode and Humanity's Last Exam, and Astra has a lower hallucination rate (51% vs 54%) (relayed via a reseller blog).","basis":"independent","sourceKind":"independent","sourceUrl":"https://www.orcarouter.ai/blog/gpt-6-1-sol-independent-benchmark"},{"point":"Ultrafast service tier (API, from 2026-10-08) costs six times Standard ($12/$60 per million tokens) and burns subscription usage at 8x; the speedup multiple has not been independently measured (relayed via a reseller blog).","basis":"vendor","sourceKind":"relayed","sourceUrl":"https://www.orcarouter.ai/blog/gpt-6-1-sol-ultrafast-rollout"}],"benchmarks":{"coding":{"pct":0.87,"benchmarks":[{"benchmark":"lmarena code arena webdev","score":1755.5,"pct":0.87,"independent":true,"sourceKind":"independent","sourceUrl":"https://lmarena.ai/leaderboard/webdev","date":null}]},"coding_agentic":{"pct":0.744,"benchmarks":[{"benchmark":"deepswe v1.1","score":75.2,"pct":0.8,"independent":false,"sourceKind":"relayed","sourceUrl":"https://community.openai.com/t/gpt-6-1-sol-in-the-api-a-meaningful-step-up-in-cost-performance/1402388","date":"2026-09-30"},{"benchmark":"terminal-bench 4.0","score":56,"pct":0.688,"independent":true,"sourceKind":"independent","sourceUrl":"https://artificialanalysis.ai/models/comparisons/gpt-6-1-sol-vs-gpt-6-luna","date":null}]},"knowledge":{"pct":0.615,"benchmarks":[{"benchmark":"humanity's last exam","score":53,"pct":0.615,"independent":true,"sourceKind":"independent","sourceUrl":"https://artificialanalysis.ai/models/comparisons/gpt-6-1-sol-vs-gpt-6-luna","date":null}]},"overall":{"pct":0.418,"benchmarks":[{"benchmark":"artificial analysis intelligence index","score":52,"pct":0.587,"independent":true,"sourceKind":"independent","sourceUrl":"https://artificialanalysis.ai/leaderboards/models","date":"2026-10-19"},{"benchmark":"artificial analysis intelligence index v4.3.2","score":52,"pct":0.667,"independent":true,"sourceKind":"independent","sourceUrl":"https://artificialanalysis.ai/models/releases/gpt-6-1-sol","date":null},{"benchmark":"gdpval-aa v2.1","score":1594,"pct":0,"independent":true,"sourceKind":"independent","sourceUrl":"https://artificialanalysis.ai/models/comparisons/gpt-6-1-sol-vs-gpt-6-luna","date":null}]},"tool_use":{"pct":0.667,"benchmarks":[{"benchmark":"automationbench-aa","score":65,"pct":0.667,"independent":true,"sourceKind":"independent","sourceUrl":"https://artificialanalysis.ai/models/comparisons/gpt-6-1-sol-vs-gpt-6-luna","date":null}]}},"anecdotes":[{"claim":"Hand-built test of 15 support tickets with a policy doc and one lookup tool: 6.1 Sol at low, medium and high all got the policy outcome right 15/15, same as Astra at medium, at about $1.80 per 1,000 tickets versus $9.09 for Astra. Some setups over-escalated, and none caught a letter O typed in an order ID. Author says it is not a benchmark and the publisher sells support agents.","taskFamily":"operations-support","taskDescription":null,"characteristics":[],"direction":"better","comparedTo":"gpt-6-astra","sourceType":"blog","sourceKind":"community","sourceUrl":"https://www.eesel.ai/blog/gpt-6-1-sol-review","date":"2026-10-01"},{"claim":"Ran three agent-style tasks five times each at max effort (CI-log triage, 3,664-line incident-log postmortem, dependency resolver written from a spec against a hidden 120-test suite): both models went 15/15, 6.1 Sol cost about 18% of Astra ($2.66 vs $14.77) and was faster on two of three tasks.","taskFamily":"operations-support","taskDescription":null,"characteristics":[],"direction":"better","comparedTo":"gpt-6-astra","sourceType":"news","sourceKind":"community","sourceUrl":"https://thenewstack.io/gpt-6-1-sol-vs-gpt-6-astra/","date":"2026-10-07"},{"claim":"On a poster's own DigBench runs (high effort) 6.1 Sol won both tasks, finishing in about 10 and 25 minutes at roughly $1.11 and $2.46 API-equivalent, ahead of GPT-6 Sol at high. Task type not described; poster's unverified numbers.","taskFamily":"agentic-long-running","taskDescription":null,"characteristics":[],"direction":"better","comparedTo":"gpt-6-sol","sourceType":"forum","sourceKind":"community","sourceUrl":"https://community.openai.com/t/500k-context-length-for-gpt-6-1-sol-day-7-of-28-days-of-quality-of-life-improvements-and-or-a-full-reset/1403525/19","date":"2026-10-06"},{"claim":"Uses a cheaper reverse-orchestration setup, with Luna and 6.1 Sol doing reviews, and says it works very well in practice.","taskFamily":"code-review","taskDescription":null,"characteristics":[],"direction":"better","comparedTo":"gpt-6-astra","sourceType":"forum","sourceKind":"community","sourceUrl":"https://community.openai.com/t/announcing-gpt-6-sol-and-gpt-6-luna-in-the-api-codex-and-chatgpt/1399925","date":"2026-10-08"},{"claim":"Pro user says 6.1 Sol increasingly refuses or sanitizes ordinary site-admin, SEO, debugging and moderation-config work when the site contains lawful adult content; GPT-5.6 Sol completed these but burns far more Work/Codex quota, so fallback is not workable.","taskFamily":"operations-support","taskDescription":null,"characteristics":[],"direction":"worse","comparedTo":"gpt-5.6-sol","sourceType":"forum","sourceKind":"community","sourceUrl":"https://community.openai.com/t/gpt-6-1-sol-over-refuses-allowed-work-codex-tasks-gpt-5-6-fallback-is-not-a-viable-solution/1402721","date":"2026-10-01"},{"claim":"GitHub issue reports a two-week drop in code comprehension quality in the Codex VS Code extension with 6.1 Sol at extra high (also Astra and 5.6 Sol): needs 3-4 correction turns, misses architectural risks, slow; same question in ChatGPT web was clearly better. Single-user observation, no traces attached.","taskFamily":"code-review","taskDescription":null,"characteristics":[],"direction":"worse","comparedTo":null,"sourceType":"forum","sourceKind":"community","sourceUrl":"https://github.com/openai/codex/issues/53583","date":"2026-10-13"},{"claim":"Sol produced minimal design with no animations, invented grammar in an interpreter, and wrote the thinnest tests; Opus 5.5 won the design and interpreter tests.","taskFamily":null,"taskDescription":null,"characteristics":[],"direction":"worse","comparedTo":"claude-opus-5.5","sourceType":"blog","sourceKind":"community","sourceUrl":"https://thomas-wiegold.com/blog/gpt-6-1-sol-review/","date":"2026-10-05"},{"claim":"Sol followed the spec most literally and wrote the least code, but its tests missed 6 of 36 seeded bugs and its search ran far slower than Opus.","taskFamily":null,"taskDescription":null,"characteristics":[],"direction":"mixed","comparedTo":"claude-opus-5.5","sourceType":"blog","sourceKind":"community","sourceUrl":"https://thomas-wiegold.com/blog/gpt-6-1-sol-review/","date":"2026-10-05"},{"claim":"Sol scored 97.5 percent on an eight-task Codex set, above GPT-6 Sol but below Opus 5.5's reported score; weak spots were dispatch logic and collision geometry.","taskFamily":null,"taskDescription":null,"characteristics":[],"direction":"mixed","comparedTo":"claude-opus-5.5","sourceType":"blog","sourceKind":"community","sourceUrl":"https://daily.dev/posts/gpt-6-1-sol-fully-tested-dots-all-devday-launches-explained-it-beats-opus-5-5--vm3uujzdf","date":"2026-09-30"},{"claim":"Code-migration benchmark gave Sol the best reviewer rating but it trailed Astra on overall score, taking about 96 minutes per task at max effort.","taskFamily":null,"taskDescription":null,"characteristics":[],"direction":"mixed","comparedTo":"gpt-6-astra","sourceType":"blog","sourceKind":"community","sourceUrl":"https://www.gradually.ai/en/ai-models/gpt-6.1-sol/","date":null},{"claim":"Sol scored 43/43 on Plotly's 43-question analytics test, matching Astra at about a tenth of the cost, with a median of 18s per question. Single run, private questions, one domain and harness, and cost estimates are rough.","taskFamily":"data-analysis","taskDescription":null,"characteristics":[],"direction":"better","comparedTo":"gpt-6-astra","sourceType":"blog","sourceKind":"community","sourceUrl":"https://plotly.com/blog/introducing-dataanalyticsbench/","date":"2026-10-14"},{"claim":"In three Codex builds (browser OS, 3D apartment, simulated robot arm) the author found 6.1 Sol faster and more cost-efficient than Sonnet 5.5, but Sonnet's visuals, environments and spatial accuracy were stronger.","taskFamily":null,"taskDescription":null,"characteristics":[],"direction":"mixed","comparedTo":"claude-sonnet-5.5","sourceType":"blog","sourceKind":"community","sourceUrl":"https://www.aifire.co/p/gpt-6-1-sol-in-codex-surprised-me-across-three-real-tests","date":"2026-10-01"},{"claim":"Across six reasoning efforts, max produced the best-ranked code in both blind reviews (maintainable structure, real tests) but took 1h16 and 37k reasoning tokens; low and medium looked fine in the browser but had hard-to-maintain code.","taskFamily":null,"taskDescription":null,"characteristics":[],"direction":"mixed","comparedTo":null,"sourceType":"blog","sourceKind":"community","sourceUrl":"https://spectrumailab.com/blog/gpt-6-1-sol-reasoning-effort-test-2026","date":"2026-10-04"},{"claim":"Versus GPT-6 Sol, 6.1 Sol was equally correct but used about 30% fewer tokens on a messy sales CSV aggregation and prioritised planted bugs better in a code review; estimated API cost per run about 20% lower. Versus Astra the review feedback was nearly the same at roughly one fifth of the cost (about $0.06 vs $0.30).","taskFamily":null,"taskDescription":null,"characteristics":[],"direction":"better","comparedTo":"gpt-6-sol","sourceType":"blog","sourceKind":"community","sourceUrl":"https://note.com/mamagotolabo/n/n2933cfc78c1c?hl=en","date":"2026-10-02"},{"claim":"Sonar's large Java benchmark found 6.1 Sol close to Astra on pass rate with fewer vulnerabilities per line than GPT-6 Sol, but reasoning tokens fell 51% while output volume grew, producing more raw bugs and smells to triage.","taskFamily":null,"taskDescription":null,"characteristics":[],"direction":"mixed","comparedTo":"gpt-6-sol","sourceType":"blog","sourceKind":"community","sourceUrl":"https://www.sonarsource.com/blog/openai-gpt-6-1-sol-evaluation/","date":"2026-10-09"}],"url":"https://gjalla.io/models/gpt-6.1-sol/"},{"model":"grok-4.5","vendor":"xai","tier":null,"price":null,"meta":{"released":null,"contextWindow":null,"maxOutput":null,"modalities":null,"officialCardUrl":null},"excelsAt":[],"watchOuts":[],"benchmarks":{"coding_agentic":{"pct":0.083,"benchmarks":[{"benchmark":"terminal-bench 2.1 (aggregator, anotherwrapper.com)","score":83.3,"pct":0.083,"independent":true,"sourceKind":"independent","sourceUrl":"https://anotherwrapper.com/tools/llm-pricing/leaderboards/terminal-bench","date":"2026-10-20"}]}},"anecdotes":[],"url":"https://gjalla.io/models/grok-4.5/"},{"model":"grok-4.7","vendor":"xai","tier":null,"price":{"input":2,"output":6,"cacheRead":2,"cacheWrite":2,"source":"vendor","sourceUrl":"https://x.ai/news/grok-4-7"},"meta":{"released":"2026-09-21","contextWindow":500000,"maxOutput":500000,"modalities":["text","image"],"officialCardUrl":"https://x.ai/news/grok-4-7"},"excelsAt":[{"point":"xAI-reported strengths in electrical engineering (EEBench 64.0 vs 56.4 for Fable 5.1) and Harvey legal agent tasks (19.6 vs 6.7); vendor numbers only.","basis":"vendor","sourceKind":"primary","sourceUrl":"https://x.ai/news/grok-4-7"},{"point":"Cursor positions it for harder, longer-running multi-step agent work on large codebases, including tasks Grok 4.6 struggled with.","basis":"vendor","sourceKind":"relayed","sourceUrl":"https://cursor.com/help/models-and-usage/grok-4-7.md"}],"watchOuts":[{"point":"Very verbose and slow to start: about 240M output tokens to run the AA index (median 85M), 46.6s time to first token, and $3.74 per index task (rank 102 of 114).","basis":"independent","sourceKind":"independent","sourceUrl":"https://artificialanalysis.ai/models/grok-4-7"},{"point":"Trails Fable 5.1 on terminal work (Terminal-Bench 4.0 37.6 vs 57.9, xAI-reported); Vals measured only 28.8 on the same benchmark.","basis":"independent","sourceKind":"independent","sourceUrl":"https://vals.ai/models/grok_grok-4.7"},{"point":"Mid-pack on blind WebDev voting (1637, rank 18) and AA index 46, well below Anthropic, OpenAI, Google and Meta flagships.","basis":"independent","sourceKind":"independent","sourceUrl":"https://arena.ai/leaderboard/code/webdev"}],"benchmarks":{"coding":{"pct":0.339,"benchmarks":[{"benchmark":"code arena webdev","score":1637,"pct":0.286,"independent":true,"sourceKind":"independent","sourceUrl":"https://arena.ai/leaderboard/code/webdev","date":"2026-10-19"},{"benchmark":"lmarena code arena webdev","score":1637,"pct":0.391,"independent":true,"sourceKind":"independent","sourceUrl":"https://arena.ai/leaderboard/code/webdev","date":"2026-10-19"}]},"coding_agentic":{"pct":0.258,"benchmarks":[{"benchmark":"cursorbench 4.0","score":46.3,"pct":0,"independent":false,"sourceKind":"primary","sourceUrl":"https://x.ai/news/grok-4-7","date":"2026-09-21"},{"benchmark":"deepswe v1.1","score":71,"pct":0.4,"independent":false,"sourceKind":"primary","sourceUrl":"https://x.ai/news/grok-4-7","date":"2026-09-21"},{"benchmark":"terminal-bench 4.0","score":37.6,"pct":0.375,"independent":false,"sourceKind":"primary","sourceUrl":"https://x.ai/news/grok-4-7","date":"2026-09-21"}]},"overall":{"pct":0.395,"benchmarks":[{"benchmark":"artificial analysis intelligence index","score":46,"pct":0.457,"independent":true,"sourceKind":"independent","sourceUrl":"https://artificialanalysis.ai/models/grok-4-7","date":"2026-10-20"},{"benchmark":"vals index","score":58.56,"pct":0.333,"independent":true,"sourceKind":"independent","sourceUrl":"https://vals.ai/models/grok_grok-4.7","date":"2026-10-20"}]}},"anecdotes":[{"claim":"Cursor advises using Grok 4.7 for hard, long-running agent work and Composer for everyday coding where speed and cost matter.","taskFamily":null,"taskDescription":null,"characteristics":[],"direction":"mixed","comparedTo":null,"sourceType":"vendor","sourceKind":"relayed","sourceUrl":"https://cursor.com/help/models-and-usage/grok-4-7.md","date":null}],"url":"https://gjalla.io/models/grok-4.7/"},{"model":"hy4-preview","vendor":"tencent","tier":null,"price":null,"meta":{"released":null,"contextWindow":null,"maxOutput":null,"modalities":null,"officialCardUrl":null},"excelsAt":[],"watchOuts":[],"benchmarks":{"coding":{"pct":0.304,"benchmarks":[{"benchmark":"lmarena code arena webdev","score":1631,"pct":0.304,"independent":true,"sourceKind":"independent","sourceUrl":"https://arena.ai/leaderboard/code/webdev","date":"2026-10-19"}]},"coding_agentic":{"pct":0.375,"benchmarks":[{"benchmark":"terminal-bench 2.1 (aggregator, anotherwrapper.com)","score":85.4,"pct":0.375,"independent":true,"sourceKind":"independent","sourceUrl":"https://anotherwrapper.com/tools/llm-pricing/leaderboards/terminal-bench","date":"2026-10-20"}]}},"anecdotes":[],"url":"https://gjalla.io/models/hy4-preview/"},{"model":"kimi-k2","vendor":"moonshot","tier":null,"price":null,"meta":{"released":null,"contextWindow":null,"maxOutput":null,"modalities":null,"officialCardUrl":null},"excelsAt":[],"watchOuts":[],"benchmarks":{"coding_agentic":{"pct":0.133,"benchmarks":[{"benchmark":"swe-bench pro (scale seal public set)","score":27.7,"pct":0.133,"independent":true,"sourceKind":"independent","sourceUrl":"https://morphllm.com/swe-bench-pro","date":"2026-09-14"}]}},"anecdotes":[],"url":"https://gjalla.io/models/kimi-k2/"},{"model":"kimi-k3","vendor":"moonshot","tier":null,"price":{"input":3,"output":15,"cacheRead":0.3,"cacheWrite":3,"source":"platform","sourceUrl":null},"meta":{"released":"2026-07-16","contextWindow":1048576,"maxOutput":131072,"modalities":["text","image","video"],"officialCardUrl":"https://huggingface.co/moonshotai/Kimi-K3"},"excelsAt":[{"point":"Positioned for long coding sessions, multi-tool agents and knowledge work; SWE Marathon 42.0 ranks first among compared models.","basis":"vendor","sourceKind":"relayed","sourceUrl":"https://smhn.info/202607-moonshot-ai-kimi-k3-launch-claude-fable-5"},{"point":"Top vendor-reported scores on browsing and automation: BrowseComp 91.2, Automation Bench, and SpreadsheetBench 2.","basis":"vendor","sourceKind":"relayed","sourceUrl":"https://smhn.info/202607-moonshot-ai-kimi-k3-launch-claude-fable-5"},{"point":"Debuted first on Arena's Frontend Code leaderboard in blind developer votes, ahead of Claude Fable 5.","basis":"independent","sourceKind":"independent","sourceUrl":"https://techsy.io/es/blog/kimi-k3-review"},{"point":"2.8T MoE with 1M context and image and video input; good for everyday chat, writing and web-app generation.","basis":"independent","sourceKind":"independent","sourceUrl":"https://www.mayhemcode.com/2026/07/should-you-switch-from-claude-opusfable.html"}],"watchOuts":[{"point":"Trails closed frontier models on some coding evals (DeepSWE 67.5 vs 73.0; FrontierSWE 81.2 vs 86.6).","basis":"vendor","sourceKind":"relayed","sourceUrl":"https://smhn.info/202607-moonshot-ai-kimi-k3-launch-claude-fable-5"},{"point":"Pricey for an open model ($3/$15 per 1M), always-on reasoning billed as output, and only about 62 tokens/s.","basis":"independent","sourceKind":"independent","sourceUrl":"https://techsy.io/es/blog/kimi-k3-review"},{"point":"Failed about 36% of trap tasks designed around open-weight weaknesses versus 8% for Opus 4.8.","basis":"practitioner","sourceKind":"community","sourceUrl":"https://www.mindstudio.ai/blog/kimi-k3-real-world-coding-review"},{"point":"Weaker than Fable 5 or Opus 4.8 on chart and screenshot reading and multi-file code repair.","basis":"independent","sourceKind":"independent","sourceUrl":"https://www.mayhemcode.com/2026/07/should-you-switch-from-claude-opusfable.html"},{"point":"Current AA index is 44 on the October index version (the earlier 57 figure is from an older version), now below Qwen3.8 Max and GLM-5.3 at 45.","basis":"independent","sourceKind":"independent","sourceUrl":"https://artificialanalysis.ai/leaderboards/models"},{"point":"Artificial Analysis marks Kimi K3 as deprecated and points to Kimi K3.1 Pro, which scores 55 on the same index version.","basis":"independent","sourceKind":"independent","sourceUrl":"https://artificialanalysis.ai/models/kimi-k3"}],"benchmarks":{"coding":{"pct":0.454,"benchmarks":[{"benchmark":"code arena webdev","score":1655.2,"pct":0.429,"independent":true,"sourceKind":"independent","sourceUrl":"https://llmrun.dev/benchmark/lmarena-webdev?show=all","date":null},{"benchmark":"lmarena code arena webdev","score":1655.2,"pct":0.478,"independent":true,"sourceKind":"independent","sourceUrl":"https://lmarena.ai/leaderboard/webdev","date":null}]},"coding_agentic":{"pct":0.76,"benchmarks":[{"benchmark":"terminal-bench 2.1","score":88.3,"pct":0.727,"independent":false,"sourceKind":"relayed","sourceUrl":"https://www.morphllm.com/kimi-k3","date":"2026-07-27"},{"benchmark":"terminal-bench 2.1 (aggregator, anotherwrapper.com)","score":88.3,"pct":0.792,"independent":true,"sourceKind":"independent","sourceUrl":"https://anotherwrapper.com/tools/llm-pricing/leaderboards/terminal-bench","date":"2026-10-19"}]},"overall":{"pct":0.261,"benchmarks":[{"benchmark":"artificial analysis intelligence index","score":44,"pct":0.261,"independent":true,"sourceKind":"independent","sourceUrl":"https://artificialanalysis.ai/leaderboards/models","date":"2026-10-19"}]},"reasoning":{"pct":0.404,"benchmarks":[{"benchmark":"gpqa diamond","score":93.5,"pct":0.5,"independent":false,"sourceKind":"relayed","sourceUrl":"https://www.morphllm.com/kimi-k3","date":"2026-07-27"},{"benchmark":"humanity's last exam","score":43.5,"pct":0.308,"independent":false,"sourceKind":"relayed","sourceUrl":"https://www.morphllm.com/kimi-k3","date":"2026-07-27"}]}},"anecdotes":[{"claim":"MindStudio's harness on real GitHub issues: K3 stayed within about 0.2 points of Opus 4.8 on simple tasks and above 60/70 on complex builds, but failed 36% of deliberately trapped tasks vs about 8% for Opus; suggests Opus/GPT for planning and K3 for implementation.","taskFamily":"feature","taskDescription":null,"characteristics":[],"direction":"mixed","comparedTo":"claude-opus-4.8","sourceType":"blog","sourceKind":"community","sourceUrl":"https://www.mindstudio.ai/blog/kimi-k3-real-world-coding-review","date":"2026-07-25"},{"claim":"In a Rails chat-app build K3 used the correct library API, wrote production-style Docker and failure tests, but skipped a system prompt, rate limits and message-size caps; hardening and test depth trail Opus 4.8. Good for interactive solo coding.","taskFamily":"feature","taskDescription":null,"characteristics":[],"direction":"mixed","comparedTo":"claude-opus-4.8","sourceType":"blog","sourceKind":"community","sourceUrl":"https://akitaonrails.com/en/2026/07/17/llm-benchmarks-kimi-k3/","date":"2026-07-17"},{"claim":"For web design, K3 produced a working landing page in under 5 minutes and could fix CSS from a screenshot with native vision, but had small spacing/animation errors; Claude's first iteration was closer to production and its auth architecture cleaner.","taskFamily":"feature","taskDescription":null,"characteristics":[],"direction":"mixed","comparedTo":"claude-opus-4.8","sourceType":"blog","sourceKind":"community","sourceUrl":"https://www.wesleychong.com/articles/kimi-vs-claude-vs-codex-webdev?lang=en","date":"2026-07-29"},{"claim":"In a 3D-world generation test via an open harness, K3 self-corrected against a reference image and planned seven sub-agents at a fraction of the cost, but Opus 5 gave the more polished physics and fidelity.","taskFamily":"agentic-long-running","taskDescription":null,"characteristics":[],"direction":"mixed","comparedTo":"claude-opus-5","sourceType":"blog","sourceKind":"community","sourceUrl":"https://www.mindstudio.ai/blog/kimi-k3-vs-claude-opus-3d-world-generation","date":"2026-07-30"},{"claim":"Nearly matched Opus 4.8 on simple GitHub-issue PRs at about half the price, but failed 36% of trap tasks versus 8%; suggests Opus for planning and K3 for implementation.","taskFamily":null,"taskDescription":null,"characteristics":[],"direction":"mixed","comparedTo":"claude-opus-4.8","sourceType":"blog","sourceKind":"community","sourceUrl":"https://www.mindstudio.ai/blog/kimi-k3-real-world-coding-review","date":"2026-07-25"},{"claim":"Finished a dataset-build, Gemma fine-tune and local web deploy workflow end to end and fixed its own errors, but did better in Kimi CLI than other harnesses (sponsored review).","taskFamily":null,"taskDescription":null,"characteristics":[],"direction":"better","comparedTo":"claude-opus-4.8","sourceType":"youtube","sourceKind":"community","sourceUrl":"https://www.volanea.com/blog/kimi-k3-ai-model-review","date":"2026-07-25"},{"claim":"Willison's pelican-on-a-bicycle SVG test produced a good result but burned 13K reasoning tokens, about 25 cents, for one trivial prompt.","taskFamily":null,"taskDescription":null,"characteristics":[],"direction":"mixed","comparedTo":null,"sourceType":"blog","sourceKind":"community","sourceUrl":"https://techsy.io/es/blog/kimi-k3-review","date":"2026-07-16"},{"claim":"Single-prompt tests gave a clear blog intro, an idiomatic React landing page and a faithful summary of a long research paper with no obvious fabrication.","taskFamily":null,"taskDescription":null,"characteristics":[],"direction":"better","comparedTo":null,"sourceType":"blog","sourceKind":"community","sourceUrl":"https://www.chatgoat.ai/blog/kimi-k3-review","date":"2026-07-28"},{"claim":"Good for chat, writing and frontend or web app generation, but Fable 5 or Opus 4.8 are better for chart and screenshot reading, high-stakes agents and multi-file repair.","taskFamily":null,"taskDescription":null,"characteristics":[],"direction":"mixed","comparedTo":"claude-fable-5","sourceType":"blog","sourceKind":"community","sourceUrl":"https://www.mayhemcode.com/2026/07/should-you-switch-from-claude-opusfable.html","date":"2026-07-18"},{"claim":"Kimi K3 produced the most compact code (about 360 lines per task) in a one-shot HTML animation test versus Qwen3.8 Max at about 540.","taskFamily":null,"taskDescription":null,"characteristics":[],"direction":"better","comparedTo":"qwen3.8-max","sourceType":"blog","sourceKind":"community","sourceUrl":"https://empiriolabs.ai/blog/kimi-k3-glm-5-3-qwen3-8-max-deepseek-v4-pro-coding-test","date":"2026-10-04"}],"url":"https://gjalla.io/models/kimi-k3/"},{"model":"kimi-k3.1-pro","vendor":"moonshot","tier":null,"price":null,"meta":{"released":"2026-10-23","contextWindow":1000000,"maxOutput":131072,"modalities":["text","image","video"],"officialCardUrl":"https://platform.kimi.ai/docs/guide/kimi-k3-1-pro-quickstart"},"excelsAt":[{"point":"Scores 55 on the Artificial Analysis Intelligence Index (v4.3.2), fourth of 221 models on its model page and 11 points above Kimi K3 on the same version.","basis":"independent","sourceKind":"independent","sourceUrl":"https://artificialanalysis.ai/models/kimi-k3-1-pro"},{"point":"A post-trained version of Kimi K3 with substantially larger post-training compute; Moonshot reports claim-level hallucination falling from 23.35% for K3 to 6.85%.","basis":"vendor","sourceKind":"primary","sourceUrl":"https://platform.kimi.ai/docs/guide/kimi-k3-1-pro-quickstart"}],"watchOuts":[{"point":"Slow and not cheap per task on Artificial Analysis: 46.3 output tokens per second (182nd of 192) and $2.11 per Intelligence Index task (92nd of 121).","basis":"independent","sourceKind":"independent","sourceUrl":"https://artificialanalysis.ai/models/kimi-k3-1-pro"},{"point":"Thinking is always on (effort low, high or max, with max the default), so it cannot be run as a non-reasoning model.","basis":"vendor","sourceKind":"primary","sourceUrl":"https://platform.kimi.ai/docs/guide/kimi-k3-1-pro-quickstart"},{"point":"Weights are not public yet: Moonshot says the full open weights arrive by Nov 4.","basis":"vendor","sourceKind":"primary","sourceUrl":"https://platform.kimi.ai/docs/guide/kimi-k3-1-pro-quickstart"}],"benchmarks":{"overall":{"pct":0.783,"benchmarks":[{"benchmark":"artificial analysis intelligence index","score":55,"pct":0.783,"independent":true,"sourceKind":"independent","sourceUrl":"https://artificialanalysis.ai/models/kimi-k3-1-pro","date":"2026-10-26"}]}},"anecdotes":[],"url":"https://gjalla.io/models/kimi-k3.1-pro/"},{"model":"mimo-v2.6-flash","vendor":"xiaomi","tier":null,"price":null,"meta":{"released":null,"contextWindow":null,"maxOutput":null,"modalities":null,"officialCardUrl":null},"excelsAt":[],"watchOuts":[],"benchmarks":{"coding":{"pct":0.348,"benchmarks":[{"benchmark":"lmarena code arena webdev","score":1634,"pct":0.348,"independent":true,"sourceKind":"independent","sourceUrl":"https://arena.ai/leaderboard/code/webdev","date":"2026-10-19"}]},"coding_agentic":{"pct":0.604,"benchmarks":[{"benchmark":"terminal-bench 2.1 (aggregator, anotherwrapper.com)","score":87.6,"pct":0.604,"independent":true,"sourceKind":"independent","sourceUrl":"https://anotherwrapper.com/tools/llm-pricing/leaderboards/terminal-bench","date":"2026-10-20"}]}},"anecdotes":[],"url":"https://gjalla.io/models/mimo-v2.6-flash/"},{"model":"mimo-v2.6-pro","vendor":"xiaomi","tier":null,"price":null,"meta":{"released":null,"contextWindow":null,"maxOutput":null,"modalities":null,"officialCardUrl":null},"excelsAt":[],"watchOuts":[],"benchmarks":{"coding":{"pct":0.217,"benchmarks":[{"benchmark":"lmarena code arena webdev","score":1625,"pct":0.217,"independent":true,"sourceKind":"independent","sourceUrl":"https://arena.ai/leaderboard/code/webdev","date":"2026-10-19"}]},"coding_agentic":{"pct":0.958,"benchmarks":[{"benchmark":"terminal-bench 2.1 (aggregator, anotherwrapper.com)","score":89.9,"pct":0.958,"independent":true,"sourceKind":"independent","sourceUrl":"https://anotherwrapper.com/tools/llm-pricing/leaderboards/terminal-bench","date":"2026-10-19"}]},"overall":{"pct":0.457,"benchmarks":[{"benchmark":"artificial analysis intelligence index","score":46,"pct":0.457,"independent":true,"sourceKind":"independent","sourceUrl":"https://artificialanalysis.ai/leaderboards/models","date":"2026-10-20"}]}},"anecdotes":[],"url":"https://gjalla.io/models/mimo-v2.6-pro/"},{"model":"mistral-large-4","vendor":"mistral","tier":null,"price":{"input":0.68,"output":2.09,"cacheRead":0.07,"cacheWrite":0.68,"source":"vendor","sourceUrl":"https://docs.mistral.ai/models/mistral-large-4"},"meta":{"released":"2026-10-06","contextWindow":1000000,"maxOutput":256000,"modalities":["text","image"],"officialCardUrl":"https://mistral.ai/news/mistral-large-4/"},"excelsAt":[{"point":"Mistral's largest model: about 1T MoE (52B active), natively multimodal, 160+ languages, trained and served in European datacenters.","basis":"vendor","sourceKind":"primary","sourceUrl":"https://mistral.ai/news/mistral-large-4/"},{"point":"Pitched for coding and agentic work (DeepSWE v1.1 61.7%), cyber defense (Cybench 93%) and visual grounding on documents and charts.","basis":"vendor","sourceKind":"primary","sourceUrl":"https://mistral.ai/news/mistral-large-4/"},{"point":"Handles spreadsheets, slides and PDFs; reports resisting 93.3% of prompt-injection attacks on Lakera's B3.","basis":"vendor","sourceKind":"primary","sourceUrl":"https://mistral.ai/news/mistral-large-4/"},{"point":"Best end-to-end cybersecurity result (CyberGym-E2E about 82%) and top intelligence among models from outside the US and China.","basis":"independent","sourceKind":"independent","sourceUrl":"https://www.tomshardware.com/tech-industry/artificial-intelligence/independent-tests-rank-mistrals-new-trillion-parameter-large-4-the-best-ai-model-outside-the-u-s-and-china-but-chinese-open-weights-still-overcome-europes-best-efforts"},{"point":"Fast output (about 116 tokens/s, 1.46s first token); won most blind human comparisons versus GLM-5.3 and Kimi K3.","basis":"independent","sourceKind":"independent","sourceUrl":"https://www.tomshardware.com/tech-industry/artificial-intelligence/independent-tests-rank-mistrals-new-trillion-parameter-large-4-the-best-ai-model-outside-the-u-s-and-china-but-chinese-open-weights-still-overcome-europes-best-efforts"}],"watchOuts":[{"point":"Hallucinated on 41.9% of knowledge questions and answered only 25.8% correctly; do not rely on it answering from memory.","basis":"independent","sourceKind":"independent","sourceUrl":"https://www.eesel.ai/blog/mistral-large-4"},{"point":"Token-heavy: about 200M output tokens to run the index versus an 81M median; about $1.13 per task at list price.","basis":"independent","sourceKind":"independent","sourceUrl":"https://www.tomshardware.com/tech-industry/artificial-intelligence/independent-tests-rank-mistrals-new-trillion-parameter-large-4-the-best-ai-model-outside-the-u-s-and-china-but-chinese-open-weights-still-overcome-europes-best-efforts"},{"point":"Reliability gaps in agentic coding: sometimes ended turns with no output; multi-file builds struggled.","basis":"practitioner","sourceKind":"community","sourceUrl":"https://www.mindstudio.ai/blog/mistral-large-4-coding-test"},{"point":"Still a public preview: weights unreleased, specs conflict across pages, and Terminal-Bench (26.8%) and HLE (35.0%) are weak.","basis":"independent","sourceKind":"independent","sourceUrl":"https://www.gradually.ai/en/ai-models/mistral-large-4/"},{"point":"Current API prices are a preview sale at half list ($0.68/$2.09 vs $1.36/$4.18); the docs give no end date.","basis":"vendor","sourceKind":"primary","sourceUrl":"https://docs.mistral.ai/models/mistral-large-4"}],"benchmarks":{"coding_agentic":{"pct":0.094,"benchmarks":[{"benchmark":"deepswe v1.1","score":61.7,"pct":0,"independent":false,"sourceKind":"relayed","sourceUrl":"https://www.implicator.ai/mistral-large-4-open-weights-lock-in/","date":"2026-10-06"},{"benchmark":"terminal-bench 4.0","score":28.3,"pct":0.188,"independent":true,"sourceKind":"independent","sourceUrl":"https://aireleasetracker.com/model/mistral/mistral-large-4","date":"2026-10-06"}]},"overall":{"pct":0.174,"benchmarks":[{"benchmark":"artificial analysis intelligence index","score":38,"pct":0.174,"independent":true,"sourceKind":"independent","sourceUrl":"https://artificialanalysis.ai/leaderboards/models","date":"2026-10-19"}]}},"anecdotes":[{"claim":"Plotly's data-analytics test: Large 4 passed 32 of 43 questions for $1.94 per run vs 25 of 43 for Mistral Medium 3.5 at $20.51; misses were judgment errors such as reporting a 7-day total as an average and missing hidden duplicate rows.","taskFamily":"data-analysis","taskDescription":null,"characteristics":[],"direction":"better","comparedTo":"mistral-medium-3.5","sourceType":"blog","sourceKind":"community","sourceUrl":"https://www.developersdigest.tech/blog/mistral-large-4-open-weight-model-2026","date":"2026-10-09"},{"claim":"Practitioners see it as a solid mid-priced option, not a coding leader: Surge AI blind ratings put it behind Claude Opus 5, and it is verbose (about 200M output tokens for the AA index vs 81M median).","taskFamily":"feature","taskDescription":null,"characteristics":[],"direction":"worse","comparedTo":"claude-opus-5","sourceType":"blog","sourceKind":"community","sourceUrl":"https://www.developersdigest.tech/blog/mistral-large-4-open-weight-model-2026","date":"2026-10-09"},{"claim":"A developer with a custom-language benchmark found Large 4 roughly level with Kimi K3 and a Claude Sonnet model but much slower than the Sonnet model and pricier than both; a Vals Index comment scored it below GLM-5.3 at higher cost.","taskFamily":"research","taskDescription":null,"characteristics":[],"direction":"worse","comparedTo":"kimi-k3","sourceType":"hacker-news","sourceKind":"community","sourceUrl":"https://www.developersdigest.tech/blog/mistral-large-4-open-weight-model-2026","date":"2026-10-09"},{"claim":"A Plotly engineer says accuracy on their data-analytics benchmark rose from 58% to 74%; not on the Pareto frontier but good enough for data analytics. (HN comment relayed by eesel.)","taskFamily":null,"taskDescription":null,"characteristics":[],"direction":"better","comparedTo":"mistral-large-3","sourceType":"hacker-news","sourceKind":"community","sourceUrl":"https://www.eesel.ai/blog/mistral-large-4","date":null},{"claim":"A commenter says it would need to be significantly better than GLM 5.3-flash to justify its price. (HN comment relayed by eesel.)","taskFamily":null,"taskDescription":null,"characteristics":[],"direction":"worse","comparedTo":"glm-5.3-flash","sourceType":"hacker-news","sourceKind":"community","sourceUrl":"https://www.eesel.ai/blog/mistral-large-4","date":null},{"claim":"A LocalLLaMA user found it dated for code analysis, image classification and chess, far behind the models they normally use. (relayed by eesel.)","taskFamily":null,"taskDescription":null,"characteristics":[],"direction":"worse","comparedTo":null,"sourceType":"reddit","sourceKind":"community","sourceUrl":"https://www.eesel.ai/blog/mistral-large-4","date":null},{"claim":"A singularity user argues it is probably good enough for most corporate workflows and values its European origin for data sovereignty. (relayed by eesel.)","taskFamily":null,"taskDescription":null,"characteristics":[],"direction":"better","comparedTo":null,"sourceType":"reddit","sourceKind":"community","sourceUrl":"https://www.eesel.ai/blog/mistral-large-4","date":null},{"claim":"Scored 42/80 on eight agentic coding tasks: perfect on a local fine-tune and web page, but silent on two tasks and weak at multi-file coordination.","taskFamily":null,"taskDescription":null,"characteristics":[],"direction":"mixed","comparedTo":null,"sourceType":"blog","sourceKind":"community","sourceUrl":"https://www.mindstudio.ai/blog/mistral-large-4-coding-test","date":"2026-10-08"}],"url":"https://gjalla.io/models/mistral-large-4/"},{"model":"muse-spark","vendor":"meta","tier":null,"price":null,"meta":{"released":null,"contextWindow":null,"maxOutput":null,"modalities":null,"officialCardUrl":null},"excelsAt":[],"watchOuts":[],"benchmarks":{"coding_agentic":{"pct":0.867,"benchmarks":[{"benchmark":"swe-bench pro (scale seal public set)","score":55,"pct":0.867,"independent":true,"sourceKind":"independent","sourceUrl":"https://morphllm.com/swe-bench-pro","date":"2026-09-14"}]}},"anecdotes":[],"url":"https://gjalla.io/models/muse-spark/"},{"model":"muse-spark-1.1","vendor":"meta","tier":null,"price":null,"meta":{"released":null,"contextWindow":null,"maxOutput":null,"modalities":null,"officialCardUrl":null},"excelsAt":[],"watchOuts":[],"benchmarks":{"coding_agentic":{"pct":1,"benchmarks":[{"benchmark":"swe-bench pro (scale seal public set)","score":61.5,"pct":1,"independent":true,"sourceKind":"independent","sourceUrl":"https://morphllm.com/swe-bench-pro","date":"2026-09-14"}]}},"anecdotes":[],"url":"https://gjalla.io/models/muse-spark-1.1/"},{"model":"muse-spark-1.3","vendor":"meta","tier":null,"price":null,"meta":{"released":null,"contextWindow":null,"maxOutput":null,"modalities":null,"officialCardUrl":null},"excelsAt":[],"watchOuts":[],"benchmarks":{"coding":{"pct":0.522,"benchmarks":[{"benchmark":"lmarena code arena webdev","score":1659,"pct":0.522,"independent":true,"sourceKind":"independent","sourceUrl":"https://arena.ai/leaderboard/code/webdev","date":"2026-10-19"}]},"coding_agentic":{"pct":0.854,"benchmarks":[{"benchmark":"terminal-bench 2.1 (aggregator, anotherwrapper.com)","score":88.8,"pct":0.854,"independent":true,"sourceKind":"independent","sourceUrl":"https://anotherwrapper.com/tools/llm-pricing/leaderboards/terminal-bench","date":"2026-10-19"}]},"overall":{"pct":0.522,"benchmarks":[{"benchmark":"artificial analysis intelligence index","score":48,"pct":0.522,"independent":true,"sourceKind":"independent","sourceUrl":"https://artificialanalysis.ai/leaderboards/models","date":"2026-10-20"}]}},"anecdotes":[],"url":"https://gjalla.io/models/muse-spark-1.3/"},{"model":"muse-wonder","vendor":"meta","tier":null,"price":null,"meta":{"released":null,"contextWindow":null,"maxOutput":null,"modalities":null,"officialCardUrl":null},"excelsAt":[],"watchOuts":[],"benchmarks":{"coding":{"pct":0.435,"benchmarks":[{"benchmark":"lmarena code arena webdev","score":1652,"pct":0.435,"independent":true,"sourceKind":"independent","sourceUrl":"https://arena.ai/leaderboard/code/webdev","date":"2026-10-19"}]},"overall":{"pct":0.587,"benchmarks":[{"benchmark":"artificial analysis intelligence index","score":52,"pct":0.587,"independent":true,"sourceKind":"independent","sourceUrl":"https://artificialanalysis.ai/leaderboards/models","date":"2026-10-20"}]}},"anecdotes":[],"url":"https://gjalla.io/models/muse-wonder/"},{"model":"muse-wonder-full","vendor":"meta","tier":null,"price":null,"meta":{"released":"2026-10-12","contextWindow":1000000,"maxOutput":0,"modalities":["text","image","video","audio"],"officialCardUrl":"https://research.meta.ai/static/muse-wonder-model-card"},"excelsAt":[{"point":"Second-tier-to-top independent results: AA index 56 (tied with Sonnet 5.5, behind only Opus 5.5) and Arena WebDev 1726, rank 6.","basis":"independent","sourceKind":"independent","sourceUrl":"https://artificialanalysis.ai/leaderboards/models"},{"point":"Meta's model card reports CyberGym 88.4% and strong refusal rates on bio/chem misuse prompts (about 98 to 100%).","basis":"vendor","sourceKind":"relayed","sourceUrl":"https://research.meta.ai/static/muse-wonder-model-card"}],"watchOuts":[{"point":"Full variant is limited to Muse Code subscribers with no public API; the API model is Muse Wonder ($4/$20 per 1M), which scores lower (AA 52).","basis":"vendor","sourceKind":"relayed","sourceUrl":"https://techcrunch.com/2026/10/12/meta-releases-its-first-frontier-model-muse-wonder"},{"point":"Meta's blog and model card publish few capability scores (blog gives none), so most comparisons rest on third-party indices.","basis":"vendor","sourceKind":"relayed","sourceUrl":"https://research.meta.ai/blog/muse-wonder"},{"point":"Uses more tokens than average: 140M output tokens on the AA index versus an 89M median.","basis":"independent","sourceKind":"independent","sourceUrl":"https://artificialanalysis.ai/models/muse-wonder-full"}],"benchmarks":{"coding":{"pct":0.891,"benchmarks":[{"benchmark":"code arena webdev","score":1726,"pct":1,"independent":true,"sourceKind":"independent","sourceUrl":"https://arena.ai/leaderboard/code/webdev","date":"2026-10-19"},{"benchmark":"lmarena code arena webdev","score":1726,"pct":0.783,"independent":true,"sourceKind":"independent","sourceUrl":"https://arena.ai/leaderboard/code/webdev","date":"2026-10-19"}]},"overall":{"pct":0.848,"benchmarks":[{"benchmark":"artificial analysis intelligence index","score":56,"pct":0.848,"independent":true,"sourceKind":"independent","sourceUrl":"https://artificialanalysis.ai/models/muse-wonder-full","date":"2026-10-20"}]}},"anecdotes":[{"claim":"A short default-settings test of Muse Wonder (not the Full variant) passed 29 checks: fixed a JS dedupe function correctly, resolved conflicting sources into valid JSON listing gaps, and wrote a six-step recovery plan.","taskFamily":null,"taskDescription":null,"characteristics":[],"direction":"better","comparedTo":null,"sourceType":"blog","sourceKind":"community","sourceUrl":"https://www.glbgpt.com/de/hub/muse-wonder-review/","date":"2026-10-14"}],"url":"https://gjalla.io/models/muse-wonder-full/"},{"model":"qwen3-coder-480b","vendor":"alibaba","tier":null,"price":null,"meta":{"released":null,"contextWindow":null,"maxOutput":null,"modalities":null,"officialCardUrl":null},"excelsAt":[],"watchOuts":[],"benchmarks":{"coding_agentic":{"pct":0.333,"benchmarks":[{"benchmark":"swe-bench pro (scale seal public set)","score":38.7,"pct":0.333,"independent":true,"sourceKind":"independent","sourceUrl":"https://morphllm.com/swe-bench-pro","date":"2026-09-14"}]}},"anecdotes":[],"url":"https://gjalla.io/models/qwen3-coder-480b/"},{"model":"qwen3.8-27b","vendor":"alibaba","tier":null,"price":null,"meta":{"released":null,"contextWindow":null,"maxOutput":null,"modalities":null,"officialCardUrl":null},"excelsAt":[],"watchOuts":[],"benchmarks":{"coding":{"pct":0.043,"benchmarks":[{"benchmark":"lmarena code arena webdev","score":1593.2,"pct":0.043,"independent":true,"sourceKind":"independent","sourceUrl":"https://lmarena.ai/leaderboard/webdev","date":null}]},"overall":{"pct":0.087,"benchmarks":[{"benchmark":"artificial analysis intelligence index","score":34,"pct":0.087,"independent":true,"sourceKind":"independent","sourceUrl":"https://artificialanalysis.ai/leaderboards/models","date":"2026-10-19"}]}},"anecdotes":[],"url":"https://gjalla.io/models/qwen3.8-27b/"},{"model":"qwen3.8-flash-next","vendor":"alibaba","tier":null,"price":null,"meta":{"released":null,"contextWindow":null,"maxOutput":null,"modalities":null,"officialCardUrl":null},"excelsAt":[],"watchOuts":[],"benchmarks":{"coding":{"pct":0.261,"benchmarks":[{"benchmark":"lmarena code arena webdev","score":1630.3,"pct":0.261,"independent":true,"sourceKind":"independent","sourceUrl":"https://lmarena.ai/leaderboard/webdev","date":null}]}},"anecdotes":[],"url":"https://gjalla.io/models/qwen3.8-flash-next/"},{"model":"qwen3.8-max","vendor":"alibaba","tier":null,"price":{"input":2,"output":6,"cacheRead":2,"cacheWrite":2,"source":"platform","sourceUrl":null},"meta":{"released":"2026-08-03","contextWindow":262144,"maxOutput":131072,"modalities":["text"],"officialCardUrl":"https://huggingface.co/Qwen/Qwen3.8-2.4T-A95B-FP8"},"excelsAt":[{"point":"Alibaba pitches long autonomous coding runs; an internal test ran a 16-day project producing an open-source agent framework.","basis":"vendor","sourceKind":"relayed","sourceUrl":"https://www.alibabagroup.com/document-2021044032125272064"},{"point":"Natively multimodal: turns screenshots into frontend code, floor plans into 3D interiors, and handles long documents, video and livestreams.","basis":"vendor","sourceKind":"relayed","sourceUrl":"https://www.alibabagroup.com/document-2021044032125272064"},{"point":"Targeted at legal document review, financial research and similar professional work (vendor claim, no figures).","basis":"vendor","sourceKind":"relayed","sourceUrl":"https://www.alibabagroup.com/document-2021044032125272064"},{"point":"Artificial Analysis index 58 (fifth overall), second on Vision Arena, top score on the tau3-Banking agent test.","basis":"independent","sourceKind":"independent","sourceUrl":"https://www.deeplearning.ai/the-batch/qwen3-8-max-lands-with-a-bang"},{"point":"Front-end generation is its strongest hands-on area: responsive layouts, animation and 3D maps.","basis":"independent","sourceKind":"independent","sourceUrl":"https://www.mindstudio.ai/blog/qwen-3-8-max-hands-on-testing"},{"point":"Breaks prompts into ordered task lists and adds edge cases and security checks unprompted, similar to Opus.","basis":"independent","sourceKind":"independent","sourceUrl":"https://getneotiler.com/blog/qwen-38-max-vs-gemini-31-pro-developer-review/"},{"point":"Arena WebDev 1672 to 1674 (preliminary), rank 11 to 12, ahead of Kimi K3 and GLM-5.3.","basis":"independent","sourceKind":"independent","sourceUrl":"https://arena.ai/leaderboard/code/webdev"}],"watchOuts":[{"point":"Slow and token-heavy: cost per benchmark task about $1.13, double Qwen3.7-Max, despite low per-token prices.","basis":"independent","sourceKind":"independent","sourceUrl":"https://www.deeplearning.ai/the-batch/qwen3-8-max-lands-with-a-bang"},{"point":"Slowest model one reviewer had tested; tasks took 30 to 80 minutes in the preview.","basis":"practitioner","sourceKind":"community","sourceUrl":"https://thomas-wiegold.com/blog/qwen-3-8-max-review/"},{"point":"Sparse documentation (no technical report or model card); open Max weights are text-only with reduced context; custom license terms.","basis":"independent","sourceKind":"independent","sourceUrl":"https://www.deeplearning.ai/the-batch/qwen3-8-max-lands-with-a-bang"},{"point":"Can resurface finished to-do items from earlier prompts and occasionally refuses benign requests like a simple curl.","basis":"practitioner","sourceKind":"community","sourceUrl":"https://getneotiler.com/blog/qwen-38-max-vs-gemini-31-pro-developer-review/"}],"benchmarks":{"coding":{"pct":0.733,"benchmarks":[{"benchmark":"code arena webdev","score":1688,"pct":0.857,"independent":true,"sourceKind":"independent","sourceUrl":"https://cryptobriefing.com/fable-5-1-tops-code-arena-webdev/","date":"2026-09-03"},{"benchmark":"lmarena code arena webdev","score":1674,"pct":0.609,"independent":true,"sourceKind":"independent","sourceUrl":"https://arena.ai/leaderboard/code/webdev","date":"2026-10-19"}]},"coding_agentic":{"pct":0.554,"benchmarks":[{"benchmark":"swe-bench pro","score":67.7,"pct":0.75,"independent":false,"sourceKind":"primary","sourceUrl":"https://huggingface.co/Qwen/Qwen3.8-2.4T-A95B-FP8","date":"2026-08-13"},{"benchmark":"terminal-bench 2.1","score":86.6,"pct":0.455,"independent":false,"sourceKind":"primary","sourceUrl":"https://huggingface.co/Qwen/Qwen3.8-2.4T-A95B-FP8","date":"2026-08-13"},{"benchmark":"terminal-bench 2.1 (aggregator, anotherwrapper.com)","score":86.6,"pct":0.458,"independent":true,"sourceKind":"independent","sourceUrl":"https://anotherwrapper.com/tools/llm-pricing/leaderboards/terminal-bench","date":"2026-10-19"}]},"overall":{"pct":0.424,"benchmarks":[{"benchmark":"artificial analysis intelligence index","score":45,"pct":0.348,"independent":true,"sourceKind":"independent","sourceUrl":"https://artificialanalysis.ai/leaderboards/models","date":"2026-10-19"},{"benchmark":"vals index","score":66.1,"pct":0.5,"independent":true,"sourceKind":"independent","sourceUrl":"https://www.latent.space/p/ainews-qwen-38-max24t-and-27b-new","date":"2026-08-03"}]},"reasoning":{"pct":0.359,"benchmarks":[{"benchmark":"gpqa diamond","score":92.6,"pct":0.333,"independent":false,"sourceKind":"primary","sourceUrl":"https://huggingface.co/Qwen/Qwen3.8-2.4T-A95B-FP8","date":"2026-08-13"},{"benchmark":"humanity's last exam","score":43.6,"pct":0.385,"independent":false,"sourceKind":"primary","sourceUrl":"https://huggingface.co/Qwen/Qwen3.8-2.4T-A95B-FP8","date":"2026-08-13"}]}},"anecdotes":[{"claim":"A creator's $32 test of hosted Qwen3.8 Max had one standout (a 2,000+ line C++ simulation) and a polished wedding site, but several expensive misses (failed printable-engine design, weak game, generic image-to-site); concludes it is a candidate generator needing strong verification, not a production migration.","taskFamily":"feature","taskDescription":null,"characteristics":[],"direction":"mixed","comparedTo":null,"sourceType":"blog","sourceKind":"community","sourceUrl":"https://www.ai.joaoqueiros.com/blog/qwen3-8-max-open-weight-coding-tests-cost-review","date":"2026-08-03"},{"claim":"On release, one commentator said open models are close to the top closed models on agentic coding but not yet there; others questioned whether the price gap is real given how hard 2.4T models are to serve.","taskFamily":"agentic-long-running","taskDescription":null,"characteristics":[],"direction":"mixed","comparedTo":"claude-fable-5","sourceType":"x","sourceKind":"community","sourceUrl":"https://www.latent.space/p/ainews-qwen-38-max24t-and-27b-new","date":"2026-08-03"},{"claim":"The open-weight checkpoint disappointed the community: it is text-only, forces thinking mode and has 262K native context, so it is not equivalent to the hosted service that the benchmarks measured.","taskFamily":"chore","taskDescription":null,"characteristics":[],"direction":"worse","comparedTo":null,"sourceType":"forum","sourceKind":"community","sourceUrl":"https://www.explainx.ai/blog/qwen3-8-max-open-weights-live-hugging-face-august-2026","date":"2026-08-12"},{"claim":"A coding-agent review notes benchmark gains (DeepSWE, FrontierSWE) were not independently reproduced as of Aug 24, preview model IDs silently re-route to production, and the 131K output cap forces chunking of large refactors.","taskFamily":"refactor","taskDescription":null,"characteristics":[],"direction":"mixed","comparedTo":null,"sourceType":"blog","sourceKind":"community","sourceUrl":"https://www.verdent.ai/guides/agents/qwen3-8-review","date":"2026-08-24"},{"claim":"Produced a very thorough coffee-roaster site and one-shot a Go poker simulation, but was the slowest model he had tested and the preview endpoint may change.","taskFamily":null,"taskDescription":null,"characteristics":[],"direction":"mixed","comparedTo":"grok-4.5","sourceType":"blog","sourceKind":"community","sourceUrl":"https://thomas-wiegold.com/blog/qwen-3-8-max-review/","date":"2026-07-20"},{"claim":"Planned better than Gemini 3.1 Pro and added unprompted security checks, but pulled finished tasks into new work and randomly refused a simple curl.","taskFamily":null,"taskDescription":null,"characteristics":[],"direction":"mixed","comparedTo":"gemini-3.1-pro","sourceType":"blog","sourceKind":"community","sourceUrl":"https://getneotiler.com/blog/qwen-38-max-vs-gemini-31-pro-developer-review/","date":"2026-09-16"},{"claim":"Front-end work was strongest (restrained site, animated Pokemon encyclopedia, 3D map) but an ISS globe rendered badly and a crowd-text animation was faked with static text.","taskFamily":null,"taskDescription":null,"characteristics":[],"direction":"mixed","comparedTo":null,"sourceType":"blog","sourceKind":"community","sourceUrl":"https://www.mindstudio.ai/blog/qwen-3-8-max-hands-on-testing","date":null},{"claim":"Reviews relayed by a secondary source say it excels at responsive layouts and animated SVG but struggles with polished high-detail 3D and has slow output.","taskFamily":null,"taskDescription":null,"characteristics":[],"direction":"mixed","comparedTo":null,"sourceType":"blog","sourceKind":"community","sourceUrl":"https://www.ownlife.dev/articles/qwen-38-for-developers-what-the-specs-promise-vs-what-you-can-ship","date":"2026-08-20"},{"claim":"Wrote the longest files in the HTML animation test, about 540 lines on average.","taskFamily":null,"taskDescription":null,"characteristics":[],"direction":"mixed","comparedTo":"kimi-k3","sourceType":"blog","sourceKind":"community","sourceUrl":"https://empiriolabs.ai/blog/kimi-k3-glm-5-3-qwen3-8-max-deepseek-v4-pro-coding-test","date":"2026-10-04"}],"url":"https://gjalla.io/models/qwen3.8-max/"}]}