When Sol and Luna Released i cranked it up to ultra and used it for a code review, i was using it as an advisor on my foreman workflow and kept noticing it took 30+ mins to complete and sometimes timeout, while fable on Max effort and Gemini 3.1 Pro worked with no issues
I expected to burn more tokens but the time it took for something simple was just ridiculous, Noticed the same pattern with my Hermes bot
Sol was supposed to be the one!!!
it annoyed me enough to start doing some targeted experiments to find what’s going on since we have way too many LLM written crap when you search for something like this that just regurgitate the same slop.
The setup
Six coding models the same three from-scratch tasks, single-shot, and measured what each one spends: generated tokens, wall-clock, and how much of the output is reasoning versus actual code.
Three tasks, each “reimplement a piece of the Python standard library without importing it”:
- a regex matcher (re.fullmatch)
- a Python-literal parser (ast.literal_eval)
- a POSIX shell splitter (shlex.split)
All three are specified, self-contained, and unambiguous about what counts as correct.
Grading
Borrowed the idea from Kaizen: Metamorphic Fuzzing and Differential Testing for LLM-Translated HPC Applications
instead of a fixed test suite, I generate thousands of random inputs per task and compare each model’s function against the real stdlib function on every one. Agreement over that corpus is the quality score
Every run is single-shot, no agentic self-verification: one response, graded as-is.
Test Subjects
Five hosted models plus GLM-5.2, an open-weight coder run at full thinking.
Results – TLDR quick view (for skimmers like me)
Verdict – Sol xhigh
Sol xhigh was the most token- and time-efficient configuration in this three-task benchmark by a wide margin. Luna xhigh generated about 4× as many tokens, while Opus 4.8 max generated about 11.4× as many. Sol also recorded the lowest combined wall-clock time across the suite. Its advantage came from both compact code and a comparatively modest reasoning trace.
At the opposite extreme, Opus 4.8 max was the least token-efficient configuration tested. It generated approximately 281K tokens across the three-task suite, about 98% of which were reasoning tokens.
Reasoning effort is where the tokens go
Higher reasoning effort substantially increased token use and latency, but the quality gains were uneven. Luna max generated about 44% more tokens and took about 57% longer than Luna xhigh. It achieved higher overall agreement, but the improvement was concentrated primarily in the regex task rather than appearing consistently across all three tasks.
Per-task detail
Tokens = coding + reasoning. Quality = agreement with the stdlib oracle over the fuzz corpus (1.000 = no mismatches observed in the tested corpus). Codex models and GLM-5.2: 3 trials (mean). Anthropic models: 1 trial. Effort is matched for Sol vs Luna (both xhigh via Codex); Fable ran high, Opus max, GLM-5.2 at full/default thinking.
| Model / effort | quality | coding tok | reasoning tok | total tok | time |
|---|
| regex | | | | | |
| Sol xhigh | 1.000 | 7,987 | 5,230 | 13,217 | 304s |
| Luna xhigh | 0.77 | 25,752 | 23,076 | 48,828 | 487s |
| Luna max | 0.999 | 30,612 | 27,528 | 58,140 | 752s |
| Fable 5 high | 1.000 | 3,569 | 43,288 | 46,857 | 601s |
| Opus 4.8 max | n/a | 1,898 | 129,363 | 131,261 | 1681s |
| GLM-5.2 full | 0.81 | 3,392 | 34,566 | 37,958 | 627s |
| literal | | | | | |
| Sol xhigh | 0.93 | 6,455 | 3,680 | 10,135 | 183s |
| Luna xhigh | 1.000 | 17,575 | 14,692 | 32,267 | 337s |
| Luna max | 1.000 | 33,913 | 30,759 | 64,672 | 634s |
| Fable 5 high | 1.000 | 3,731 | 42,572 | 46,303 | 602s |
| Opus 4.8 max | 1.000 | 2,939 | 129,750 | 132,689 | 1619s |
| GLM-5.2 full | 0.54 | 1,693 | 33,751 | 35,444 | 818s |
| shlex | | | | | |
| Sol xhigh | 1.000 | 802 | 447 | 1249 | 35s |
| Luna xhigh | 1.000 | 8,500 | 8,136 | 16,636 | 174s |
| Luna max | 0.99 | 9,261 | 8,941 | 18,202 | 185s |
| Fable 5 high | 1.000 | 447 | 3,269 | 3,716 | 50s |
| Opus 4.8 max | 1.000 | 598 | 16,503 | 17,101 | 196s |
| GLM-5.2 full | 0.63 | 183 | 29,901 | 30,084 | 760s |
- Sol is efficient on both axes. It writes moderate code (~15K total) and reasons moderately (~9K, 38%). Nobody else is balanced like this.
- Fable & Opus: compact code, runaway reasoning. Their solutions are the smallest (5–8K code total) — but at high/max effort they generate 89K (Fable) and 276K (Opus) reasoning tokens. Opus-max is 98% reasoning.
- Luna is bloated on both. It writes the most code (52–74K — 400+ line solutions) and reasons ~half its tokens. Worst of both patterns.
- GLM-5.2 over-reasons and under-delivers. At full thinking it spends 90–99% of its tokens reasoning (103K/trial, ~4× Sol) yet posts the lowest quality in the set (0.54–0.81). Same task, same prompt, its reasoning swung 11K→77K tokens and wall time 62s→1937s — the least predictable model measured. It’s the clearest single example of effort not converting to output.
- More tokens ≠ correct. Single-shot, each model drops a task: Sol → literal (0.93), Luna-xhigh → regex (0.77), GLM → all three. The token premium buys no reliability.
Then i got paranoid about my own test methodology and went digging around again to re-confirm
Differential fuzzing
published method Grading against a trusted reference over thousands of fuzzed inputs is “Eq@DFuzz” (arXiv 2602.15761); see also Kaizen (2607.04058), ISSTA’24 oracle-guided selection. Catches what fixed unit tests miss.
Trusted oracle
standard Reference is Python’s own re/ast/shlex; models must match but may not import it (verified — no banned imports).
Token metric
matches field Output+reasoning tokens under fixed settings, jointly with accuracy, is exactly OckBench (2511.05722): “when models have similar accuracy, how much do they diverge in output-token efficiency?”
Self-correcting
verified The harness caught its own bugs twice — 4 mislabeled oracle cases, and an extraction glitch on Opus’s output — both found and fixed, not buried.
Limits
- Anthropic reasoning is estimated. The API bundles thinking into output tokens, so Fable/Opus code is measured via tiktoken and reasoning is the remainder. Since their code is small, the reasoning estimate is robust directionally, but the exact split is approximate — and uses the OpenAI tokenizer on Anthropic code.
- Provider-native token counts. OpenAI (Sol/Luna) and Anthropic (Fable/Opus) tokenizers differ. Sol-vs-Luna is a clean, same-tokenizer, same-harness comparison; comparisons to Fable/Opus are cross-tokenizer and cross-harness — directional, not exact.
- Effort is not uniform. Sol & Luna are effort-matched (Codex xhigh; Luna also at max); Fable ran high, Opus max. Read within-provider first.
- Sample size. Codex and GLM n=3 (with large variance); Anthropic n=1 (each run costs $3–4 and ~10–28 min). Treat Anthropic cells as single-point.
- Single-shot ≠ cost-per-completed-task. Raw generation only; production cost would add retries for the reliability misses.
