As The Geek Learns

About As The Geek Learns

Tools and training for IT professionals. Join James Cruce—a systems engineer with 25+ years managing enterprise VMware infrastructure—for practical PowerCLI tutorials, automation tips, and lessons from the trenches. Much to learn, there always is. astgl.com (https://astgl.com?utm_medium=podcast)

Publisher

The Geek

Category

education · technology

Language

en

Explicit

No

First episode

6 Dec 2025

Latest episode

26 Oct 2026

Latest episodes

21 episodes in the feed.

1. 

   26 Oct 2026

   The Uninstall Is Part of the Evaluation

   I spent an afternoon evaluating a tool I ended up removing. The removal is the part worth writing about. The tool was codebase-memory-mcp, an MCP server that parses your codebase into a knowledge graph so an AI agent can ask structural questions. "What calls this function?" "What breaks if I change this signature?" That's a real gap in my setup. I have semantic search over my notes. I don't have anything that answers who calls this with an exact answer instead of a good guess. So I looked at it seriously. Here's how that went, and the thing I got wrong. Don't evaluate the README The README is 864 lines. It leads with thirteen badges, a link to an arXiv preprint, and claims like "120x fewer tokens" and "the Linux kernel in 3 minutes." None of that is evidence. It's marketing copy, and marketing copy is the same length whether the tool is great or terrible. So I cloned it and looked at the actual repository. That takes about five minutes and tells you things no README will: About six months old. 2,610 commits, 166 contributors. And 2,147 of those commits are from one person writing under four different name-and-email combinations. That's 82% of the project from a single pair of hands on roughly 154,000 lines of first-party C, not counting the vendored parsers. Worth knowing before you adopt it. Also worth knowing: 41,000 stars and 543 open issues. Popular, actively used, and carrying a real backlog. But the engineering discipline is real. Sanitizer builds, soak tests, CodeQL, SLSA provenance, a DCO. Someone who ships a Dockerfile for a MemorySanitizer test image is not winging it. Both of those facts matter. Neither is in the README. And here's where I have to correct myself, because it's the same mistake in a different coat. My first pass at those numbers was wrong. I'd cloned with git clone --depth 50 out of habit, because it's faster, and I only wanted a look around. A shallow clone gives you a truncated history and git log reports on it without complaint. No warning, no asterisk. I read "first commit: June" off a repository that actually started in February and got the commit count and contributor spread wrong too. I only caught it because I went to verify something unrelated, the star count, and the creation date. The API return didn't match the date my clone claimed. Same lesson as everything else in this piece: the tool answered confidently, and the answer was wrong, and nothing about the output looked wrong. git rev-parse --is-shallow-repository would have told me in one second. I didn't ask. The actual risk wasn't the code Here's what I found when I grepped the install path: the installer writes to settings.json. It installs PreToolUse, SessionStart, and SubagentStart hooks. It drops skill directories. It splices your CLAUDE.md between marker comments. And it does that across 43 coding-agent clients it auto-detects by its own --help count. That's not a security problem. It's exactly what the tool advertises. But my ~/.claude has a hand-tuned hook chain I built over months, an allowlist .gitignore, and multiple sessions running at once. An auto-detecting installer is the wrong shape for that. As The Geek Learns is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber. So I installed with --skip-config. Binary only. Then I did the part most people skip: I checked whether the flag actually worked. I took SHA-256 hashes of every config file I cared about before installing, and again after. Then I grepped for the tool's marker string across every agent directory. The hashes matched. The grep came back empty. But an empty grep proves nothing on its own. Maybe I typo'd the path, maybe the file wasn't readable. So I ran the same grep for a string I knew was in the file. It returned 66 hits. Now the empty result means something. That's the habit worth stealing from this whole story: a flag's name is not evidence that the flag worked. Neither is a clean scan, unless you've proven your scanner can find something. One thing did change: it appended a PATH line to my .zshrc. Harmless, clearly labeled. But --skip-config means agent config only, and nobody tells you that. Grade it against something you already know For the pilot I indexed a real codebase: 847 Python files, indexed in 12 seconds into about 33,000 nodes and 155,000 edges. Fast. Genuinely fast. But speed isn't the question. Any tool can be fast and useless. The question is whether the answers are right, and you can't judge that on a codebase you don't know. So I picked a question I'd already answered by hand months ago. Back then I'd worked out, by reading source files one at a time, that this particular tool quietly installs a second MCP server, but only through two specific commands, and not through the path I was using. It took a while. It mattered, because the answer decided how I installed it. I asked the graph the same question. One query for the callers of that function. Two non-test results, exactly the two commands I'd found by hand, and nothing else. That's a good test, and I'd suggest it for anything in this category. Don't ask a new retrieval tool something you can't verify. You'll grade it on whether the answer sounds right, which is how you end up trusting a confident wrong answer. Ask it something you already dug out the hard way. You'll know instantly whether it's real. The costs nobody advertises The index for that one repository was 229 MB. There are lighter modes. The middle setting cut it to 105 MB with no loss to the core package or the tests. It drops docs, JSON fixtures, and example frontends. The lightest went to 84 MB but also dropped git co-change data, which turned out to be the one thing in the graph I couldn't get anywhere else: which files historically change together. Touch this auth file, expect to touch both handlers. No AST can tell you that. Only history can. So: about 105 MB per repository, and plan for it before you point this at a dozen repos. That number appears nowhere in the documentation. I only have it because I ran all three modes and measured. Thanks for reading As The Geek Learns! This post is public, so feel free to share it. Then I tried to uninstall it uninstall failed. Not because of anything to do with the tool's own files. It failed because ~/.cursor/mcp.json on my machine is a symlink into my dotfiles repo, a completely ordinary setup, and the uninstaller refuses to write through symlinks. Refusing to follow symlinks is a good instinct. Following them blindly is how you get exploited. But it treated that refusal as fatal to the whole operation: error: agent_config agent=Cursor op=mcp_uninstall path=~/.cursor/mcp.json (target: symlink, 43 bytes) error: one or more agent cleanup operations failed; executable and index removal were not started Read that second line again. Removing the executable and the indexes never started, because of a check on a Cursor config file. A file that, because I'd installed with --skip-config, had nothing in it to remove in the first place. 386 MB left on disk after a command whose entire job is removal. A 282 MB binary and 104 MB of indexes, all of which I removed by hand. And it gets better. The same run printed lines like removed MCP config entry and removed PreToolUse + SessionStart + SubagentStart hooks for client after client. None of that happened. I checked the modification times. One of the files it claimed to have cleaned was last touched in May, months before I installed anything. Two of the directories it named don't exist on my machine at all. The output describes work it didn't do. If you read the summary instead of the filesystem, you'd believe a dozen configs had just been rewritten. The part where I embarrassed myself I wrote it up as a bug report. Good repro steps, minimal setup, the works. Step one created a symlinked Cursor config so the reader could trigger it: ln -sf ~/.dotfiles/cursor/mcp.json ~/.cursor/mcp.json Read that closely. ln -sf. Force. If you already had a real ~/.cursor/mcp.json, that command destroys it. Silently. No prompt, no backup. I was about to publish a bug report to a repo with 41,000 stars, whose reproduction steps would wreck the config of anyone who pasted them. I caught it, but not by rereading the draft. I caught it because I ran a separate check specifically looking for writes to real paths before posting. The re-read wouldn't have found it; I'd read that command four times already and it looked completely normal. The fix was to run the whole thing in a throwaway HOME. Here's why I think this one is worth sitting with: reproduction steps are the only code you write that's designed to be pasted into a stranger's shell. The blast radius is inverted from normal code. When you write a script, you're the one who runs it on your machine, where you know what exists. A repro runs on machines configured differently from yours, by people who are trusting you. A repro that's destructive on someone else's setup is a bug you're shipping to your readers. What I'd actually keep Four things, none of which are about this particular tool: Clone it before you read about it. Five minutes in the repository beats an hour with the README. Commit history, contributor spread, and CI config will tell you what you're really adopting. Verify the flag, don't trust it. Hash before, hash after. And if a scan comes back clean, prove the scan can find something first. Grade new tools on questions you've already answered. It's the only way to catch a confident wrong answer. Uninstall it before you commit to it. This is the one I'd been skipping for years. Installing is the vendor's happy path: tested, demoed, and polished. Uninstalling is where the bodies are. If a tool can't cleanly remove itself, that tells you how carefully it handles your machine as opposed to its own features. I filed the issue. I'd look at the tool again in a few versions, because the graph queries genuinely delivered and that co-change data is something I don't have anywhere else. But I'd pin the version. And I'd expect to tear it down by hand. Building an AI-assisted development workflow and writing down what actually works. Subscribe for the parts the documentation leaves out. Found this useful? I share practical lessons from my systems engineering journey at As The Geek Learns (https://astgl.substack.com). Get full access to As The Geek Learns at astgl.com/subscribe (https://astgl.com/subscribe?utm_medium=podcast&utm_campaign=CTA_4)

2. 

   22 Sept 2026

   Local Laya vs Hosted Jev: Why My Agents Make Typed Decisions on My Mac

   You ask a chat model a yes-or-no question and get back three paragraphs. Then your code has to parse the paragraphs, guess what it meant, and hope the JSON is valid this time. I got tired of that for the small, bounded choices my agents make all day. Here's the model I picked, the one I didn't, and the replay numbers behind the decision. This is my account of one deployment. Whether Laya beats Jev in general is a different question, and I didn't test it. The setup A coding agent makes dozens of small decisions that aren't really language tasks. Which specialist should take this job? Does this alert need a human right now? Which local model can handle this request without blowing the memory budget? Prose is the wrong output type for a bounded question. TypeSafe's Jev (https://docs.typesafe.ai/introduction) is built for exactly this. You hand it a state and a set of predefined questions, and it returns typed answers. A choice picks among named options. A score rates on an ordered scale. A noul returns a probability for a yes-or-no statement. No paragraph to interpret. I liked it immediately. But I had a requirement Jev couldn't meet. I wanted routine decisions running on my Mac Studio, with local data staying local. I also wanted to control the exact model, the permitted options, and what happens when the model is unsure or down. Jev is an early-access hosted service. Laya (https://huggingface.co/convaiinnovations/laya-typed-decisions) is an open-weight model with the same kind of typed-decision interface, published with downloadable weights and an Apache 2.0 runtime. For this system, that settled it. What's actually going on Both models solve the same narrow problem: turn a compact state and a few questions into structured decisions. Neither writes code or reasons through a long investigation. It's a specialist. Your main model keeps its job. Jev's managed API is a good deal on paper. TypeSafe lists $0.042 per million input tokens with no output-token charge (https://typesafe.ai/blog/introducing-system-one-models-and-jev) and says Jev handles choices with up to 255 options. Hosted also means nobody babysits model files, accelerator compatibility, startup time, or process supervision. That's real work I chose to own. As The Geek Learns is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber. Laya's trade-offs run the other way. I downloaded a pinned checkpoint once and serve it over a loopback-only API. Routine decision requests never leave the machine. There's no per-token bill, though the workstation, electricity, storage, and my time still cost money. I didn't calculate a break-even. Privacy and control were the decisive reasons. Cost was a side effect. I have no savings number to give you. The checkpoint, convaiinnovations/laya-typed-decisions, has about 421 million parameters, a ModernBERT-large encoder, and a 1,024-token limit. Its publisher trained it on four synthetic workflows: agent observability, customer service, invoice processing, and security incidents. Model routing isn't one of those, which is a reason to test rather than assume. Here's what makes it different from asking a chat model for JSON. The Laya runtime renders the question, each option, and the state into one token sequence, with a marker before each option. A bidirectional encoder reads the whole thing, a small decision head scores the marker positions, and a softmax turns those scores into a distribution over options. Several questions against one state batch into a single call. There's no autoregressive text output, so there's nothing to parse. The published runtime (https://github.com/NandhaKishorM/laya) shows the sequence construction and the head. For a choice, my code gets an option key and a probability for every allowed key. For a score, a distribution over ordered levels plus the expected level. For a noul, the probability of true. The runtime also reports a confidence derived from the distribution's normalized entropy. That confidence is not the probability the downstream task will succeed. I picked thresholds on a separate development set, and I'd want fresh evidence before reusing them on a different model roster. Laya's authors trained with rewards based on proper scoring rules, meant to encourage honest probabilities. The model card still warns it's overconfident and needs domain-specific calibration. A typed answer can be structurally perfect and factually wrong. I treat Laya as a recommender inside a program, never as a source of permission or truth. The fix My host is a Mac Studio with an M3 Ultra and 256 GB of unified memory. That hardware is part of the result. A smaller machine needs its own memory and latency tests. The original implementation brief got two things wrong. It assumed plain transformers.AutoModel could load Laya's decision model, and that the typed checkpoint lived only in a subfolder of the base model. Neither held. The typed checkpoint is its own repository and AutoModel loads only the encoder shape. The published laya runtime reconstructs the custom decision head, so the loader uses that instead of mistaking a bare encoder for a working decision engine. Around the checkpoint sits a FastAPI service that accepts the TypeSafe wire shape, so a client written for Jev can point at my box: { "model": "laya-local", "state": "The production API is down and customers are blocked.", "questions": { "route": { "type": "choice", "instructions": "Which team should handle this?", "criteria": { "billing": "Invoices and refunds", "technical": "Bugs, outages, and API errors" } } } } My code defines the labels before Laya ever sees them. The response identifies itself as laya-local-v1. Accepting a Jev-style client alias doesn't pretend Jev supplied the answer. The service listens on 127.0.0.1:8017, because Docker already had port 8000 on this machine. The service fails closed. It loads from an explicit local directory and refuses to download during inference. In production, it requires Apple's Metal accelerator and warms the model before reporting ready. It counts the exact rendered token sequence and rejects an over-budget request instead of letting the runtime quietly shorten the state to fit. It caps Choice at 20 options by default and serializes inference so concurrent callers can't pile uncontrolled load onto the accelerator. The more important change sits outside the model. A gateway first drops any model that isn't eligible: wrong capability, wrong data class, too little context, unhealthy, past deadline, or over the memory budget. Laya gets at most five survivors plus an explicit abstain option. It never sees an ineligible model and can't grant one permission. The gateway re-validates the returned option, confidence, margin, model identity, and eligibility before reserving capacity. If Laya abstains or times out, a permitted local fallback runs. If nothing is eligible, the gateway refuses the request. Inside ClaudeClaw, Laya now handles bounded specialist dispatch and how noncritical alerts get presented. A user who names a specialist still wins. Model advice can't suppress a critical alert. I also tried Laya for post-failure recovery choices. That profile failed acceptance, so it stays disabled, and the deterministic recovery rules stay in charge. What the replay showed I wanted a measured result for my own routing decision, not a latency figure copied from someone else's machine. The service's API, engine, SDK compatibility, and routing-contract suites recorded 59 passing tests, with one optional real-checkpoint test skipped. Separate HTTP runs exercised the real checkpoint on Metal. The held-out replay used previously recorded outcomes for two models qualified for this setup, Qwen3 Coder 30B and GPT-OSS 120B. The replay simulated model health and resources. The Laya inference was real. Nobody reran the coding tasks. I chose the profile on 20 development decisions, froze it, then ran 40 held-out decisions. Each task appears with more than one latency preference, so those 40 are correlated, not 40 independent wins. On the held-out set, deterministic quality-and-latency ordering produced 33 acceptable decisions out of 40, or 82.5%. Laya, with the same eligibility rules and local fallback, produced 37 of 40, or 92.5%. Four more acceptable decisions, a 10-percentage-point gain on this one replay. Laya directly selected a model 15 times, the gateway fell back 17 times, and the gateway blocked eight cases where no candidate was available. It never selected an ineligible model. The three misses matter. One near-threshold case fell back to a model that had failed its source task. Two cases had no acceptable candidate because both underlying models had failed. Laya can't repair a task; neither candidate can do. Latency is where it got humbling. The first held-out run had a routing p95 of 143.54 ms. A later supervised replay on a busy host kept the identical 37-of-40 outcome but pushed p95 to 466.69 ms, past my 100 ms aspiration and past the gateway's 300 ms decision timeout. That run is why the timeout and fallback exist. They fired for real. Twenty near-threshold decisions repeated three times kept their selected-or-abstained status every time, but repetition doesn't add independent cases. And the busy-host replay didn't prove a live gateway enforcing 300 ms would keep all 37 acceptable choices under that load. Live checks went a bit further. The supervised Laya service and gateway reported ready. Gateway acceptance covered a direct Laya choice, local fallback, and duplicate-request rejection. ClaudeClaw's compiled dispatch checks picked the expected specialist three times. Those are bounded integration checks. They say nothing about the next agent task I haven't written yet. Why this matters I did not run Jev and Laya side by side on these 40 cases. The 10-point gain is against my deterministic router, not against Jev. Laya's publisher reports 76.6% top-choice accuracy on its 2,000-decision test split against a published Jev figure of 72.7%. The same card says Jev matches the reference probability distributions better and has better raw calibration. Those numbers are leads. I didn't run a head-to-head under one protocol. The card also warns about overconfidence, English-only training, and accuracy falling as the option list grows, while TypeSafe says Jev supports far larger choices. I'd test Jev directly before claiming Laya is more accurate, faster, or cheaper for the same production decisions. What I can say is narrower. On one frozen replay, the local system made more acceptable routing decisions than my previous ordering, and it kept the decision input, checkpoint, thresholds, and fallback policy under my control. Jev may well be the better product if you want a managed API, have large choice sets, or don't want to run local inference. I chose Laya because local processing was a requirement, and I was willing to own the testing and operations that came with it. Thanks for reading As The Geek Learns! This post is public so feel free to share it. Quick reference * Typed decisions: choice returns an option plus a probability per option, score a distribution over ordered levels, noul a probability of true. Nothing to parse. * Laya checkpoint: convaiinnovations/laya-typed-decisions, about 421M parameters, ModernBERT-large, 1,024-token limit, Apache 2.0. * Load it with the published laya runtime, not bare transformers.AutoModel, which gives you an encoder with no decision head. * Deterministic code filters eligibility first. The model only ranks survivors, plus an explicit abstain option. * Re-validate everything the model returns before acting. A decision receipt records; it doesn’t authorize. * Confidence is normalized entropy, not task success probability. Calibrate thresholds on a development set and freeze them before held-out. * Budget for timeouts. My p95 went from 143 ms to 467 ms on a busy host with identical decisions. Terms worth knowing Some of this vocabulary comes from a small corner of the AI world. Short definitions, in the order they show up. * Typed decision. An answer with a fixed shape your code can use directly, such as one option from a list or a number between 0 and 1. The opposite of a paragraph you have to parse. * State. The text or JSON you hand the model describes the situation it's deciding about. An alert, a task description, a customer message. * Choice, score, noul. Jev's three question types, which Laya copies. A choice picks one option from a list you define. A score picks a level on an ordered scale you define, like low, medium, critical. A noul is a yes-or-no statement and returns the probability that it's true. The term is TypeSafe's. * Open-weight model. A model whose trained weights you can download and run yourself, as opposed to one you can only reach through a vendor's API. * Checkpoint. A saved set of model weights. "Pinned checkpoint" means I locked to one exact version and verify it hasn't changed. * Loopback-only. The service listens on 127.0.0.1, so only programs on the same machine can reach it. Nothing on the network can. * Token. The unit a model reads. Roughly a short word or word fragment. Laya's 1,024-token limit is the total length of the state, question, and options after they're rendered together. * Encoder, bidirectional. A model that reads the entire input at once, in both directions, and produces a representation of it. It doesn't generate text. ModernBERT is a 2024-era encoder family. Chat models are decoders, which is a different design. * Autoregressive. Generating output one token at a time, each token depending on the ones before. That's how chat models write. Laya doesn't do it, which is why there's nothing to parse and why it's fast. * Decision head. A small neural network bolted onto the encoder that turns its output into scores, one per option. The base encoder alone can't make decisions. This is the piece transformers.AutoModel doesn't load. * Softmax. A math step that turns a list of raw scores into probabilities that add up to 1. * Normalized entropy. A measure of how spread out a probability distribution is, scaled from 0 to 1. Laya's confidence is 1 minus that. All options equally likely gives confidence 0. One option at 100% gives confidence 1. It measures how sure the model is, not how right it is. * Calibration, overconfidence. A calibrated model that says "80%" is right about 80% of the time. An overconfident one says 80% and is right less often. Laya's own model card says it's overconfident. * Proper scoring rule. A training reward designed so the model scores best only when it reports what it believes. It's meant to discourage bluffing. It doesn't guarantee calibration on your data. * Development set, held-out set. Two separate batches of test cases. You tune thresholds on the development set, freeze them, and then measure on the held-out set the model has never influenced. Tuning on the held-out set makes the result meaningless. * Correlated cases. Test cases that share an underlying task, so they tend to succeed or fail together. Forty correlated cases carry less evidence than forty independent ones. * p95. The latency that 95% of requests came in under. A better measure of "how slow does it get" than the average. * Metal, MPS. Apple's GPU framework and PyTorch's backend for it. Requiring MPS means the service refuses to start if the model would silently run on the CPU. * Wire shape. The exact JSON layout of a request and response. Matching Jev's wire shape means a client written for Jev works against my service without changes. * Fails closed. When something is missing or wrong, the service stops instead of guessing. No checkpoint, no start. Request too long, rejected, not trimmed. * Abstain. An explicit "none of these" option the gateway always adds. It lets Laya decline instead of forcing a pick. * Eligibility gateway. The deterministic code that filters out models Laya isn't allowed to choose before Laya sees the list, then re-checks the answer afterward. Found this useful? I share practical lessons from my systems engineering journey at As The Geek Learns (https://astgl.substack.com). Thanks for reading As The Geek Learns! This post is public, so feel free to share it. Get full access to As The Geek Learns at astgl.com/subscribe (https://astgl.com/subscribe?utm_medium=podcast&utm_campaign=CTA_4)

3. 

   17 Sept 2026

   Can a Subscription Fund a Coding-Model Study? 648 Episodes, $0 API

   You’ve got a Claude Max plan (https://claude.com/pricing), a ChatGPT Pro plan (https://openai.com/chatgpt/pricing/), and a Mac Studio with a local model. (Mac Studio specs: M3 Ultra with 256 GBs memory) You’d like to run a real benchmark across all three, but the API estimate for a proper study came back much too expensive. So I routed the whole thing through the subscriptions. It mostly worked. The part that didn’t is the useful part. The Setup This is the follow-up to my GVS5H pilot. GVS5H (https://github.com/slee-persis/GVS5H) is a research project claiming that a manager-and-worker scaffold lets smaller open models match frontier ones on hard coding problems. My adaptation runs three modes: a single call, an iterative loop that revises after public tests, and a “ledger” mode where a manager plans, fresh workers execute, and notes persist on disk between calls. The study froze 24 tasks before any generation: twelve hard LiveCodeBench (https://huggingface.co/datasets/livecodebench/code_generation_lite)problems (six AtCoder, six LeetCode) and twelve practical repair fixtures in Python and TypeScript (cancellation, atomic writes, retries, pagination). Three systems, three modes, three repetitions. That’s 648 episodes. As The Geek Learns is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber. The three systems were a local Qwen3.8-27B (https://huggingface.co/Qwen/Qwen3.8-27B) running 4-bit on MLX, GPT-6 Astra through the Codex CLI (https://openaicli.com/docs)on a ChatGPT Pro login, and Claude Fable 5.1 through Claude Code (https://code.claude.com/docs/en/overview) on a Max login. No API keys were supplied to the harness, extra spending was disabled on both accounts, and a dry subscription meant waiting for the reset, not paid credit. One caveat I’ll keep repeating: this compares three configured systems, not three sets of model weights. Claude Code controls its own temperature and context, the Codex CLI has no verified provider-side output cap, and my adapter serializes history differently than the raw APIs do. Treat every cross-system number accordingly. What’s Actually Going On The study finished with 602 of 648 episodes completed and 555 hidden-test passes. Additional API charge: zero dollars. Recorded quota waits: zero seconds on both backends, across roughly four hundred cloud episodes. Astra through Pro was flawless: all 216 planned episodes were completed, and all 216 passed, in every mode and both families. The local Qwen also completed all 216 of its episodes, passing 177. The missing 46 episodes all belong to Fable through Max, and 36 of those 46 are ledger-mode episodes. Exactly half of Fable’s planned ledger runs were never counted. Here’s why. The protocol had a hard rule: no model substitution. Any other model in a response marks the episode operationally incomplete and excluded. That’s the right rule; you can’t credit Fable for an answer Fable didn’t write. Anthropic documents that its Fable and Opus models include safety classifiers that can decline a request, and that a declined request can be retried on a fallback model. In 32 of the 46 missing episodes, the response stream contained a fallback block naming Claude Opus 5, which is precisely the handoff those docs describe. Every one of those 32 was a ledger episode. Ledger mode makes many calls per episode (planning, ideation, manager, workers, and finalization), so it gets far more chances to trip a classifier. The other 14 exclusions were a subtler failure: the CLI emitted two assistant events under one message ID, and my strict identity guard refused those too. The original run stopped after three consecutive Fable failures, with 479 episodes done. A predeclared continuation covered only the 154 never-attempted Fable episodes, with one change: a documented fallback handoff excludes that episode without halting unrelated tasks. Nothing else changed, and no original episode was retried. Note what I’m not claiming. The refusal categories weren’t captured, so I don’t know what a classifier objected to in a competitive programming problem, and I’m not saying an Opus answer would have failed. Those episodes can’t be scored. What the Numbers Say Read these results with coverage attached for full context. On the hard problems, Qwen went from 55.6% with a single call to 72.2% iterative and 77.8% with the ledger. That’s the one place the scaffold appears to help. In the paired analysis, the ledger beat the single call on five of twelve tasks and lost on none, but after Holm correction the adjusted p-value is 0.375. Suggestive, not significant. It also cost a lot: about 11.3 hours of active local inference for the 36 ledger episodes versus about 3.5 hours for the 36 single-call ones. Astra needed no help. Its ledger mode spent about 3.6 times the active seconds of its single mode on the hard problems to land on the same 36 of 36. Fable on the hard problems passed 25 of 36 single (five excluded), 30 of 36 iterative (five excluded), and 18 of 36 ledger. Count only completed episodes, and the ledger went 18 for 18. Count all planned cells, and it’s 50%. Both numbers are true. Only the second one is honest as a coverage-aware score. On the practical repairs, everyone sat near the ceiling. Qwen passed 33, 36, and 34 of 36 across the three modes. Astra passed all 36 in every mode. Fable passed all 36 single and iterative, and 17 of its 18 completed ledger episodes. One adherence note: ten Qwen ledger episodes returned a manager section in the wrong format, and nine never invoked a worker. A passing answer doesn’t prove the manager-and-worker loop ran. Total recorded output was about 6.3 million tokens. Claude Code’s API-equivalent estimate for the Fable portion was about $58, a display number rather than an invoice or a quota measurement. When I kicked off the continuation, the max usage meter read 14%; I didn’t meter that independently. Why This Matters If you’re planning to benchmark models through subscriptions, the money question answers itself. It works, and the bill is your existing plan. The question you should actually plan for is harness behavior. A subscription CLI is a product, not a raw endpoint. It may hand your request to a different model, split one message into several events, or run auxiliary calls you didn’t ask for. Your protocol has to decide in advance what each of those means. Mine decided “exclude,” which protected the numbers and cost me half of one arm. I’d make that trade again, but next time I’d instrument the refusal category. The other lesson is older. Every gain I’ve seen from the ledger scaffold arrives with a multiple on time and tokens, and the one statistically tempting gain here didn’t survive multiple-comparison correction. Measure the whole configuration on the work you care about before you commit machine days to orchestration. Quick Reference * Freeze tasks, order, prompts, and budgets before generation. * Decide the model-identity rule up front. “Any other model in the stream excludes the episode” is defensible. Write it down. * Report success over all planned cells and over completed cells, side by side, with the missing count. * Log every exclusion reason in machine-readable form. Fallback, quota, and double-event failures are different problems. * Capture the refusal category if the provider exposes it. I didn’t, and I can’t answer the obvious follow-up. * Multi-call modes multiply exposure to classifier declines. Expect exclusions to concentrate there. Thanks for reading As The Geek Learns! This post is public, so feel free to share it. Found this useful? I share practical lessons from my systems and AI engineering journey at As The Geek Learns (https://astgl.substack.com). Terminology and Definitions The tools and models in the study Claude Code-Anthropic’s command-line coding agent; it reads a codebase, edits files, and runs commands from natural-language instructions. The article ran Claude Fable 5.1 through it. (https://code.claude.com/docs/en/overview (https://code.claude.com/docs/en/overview)) Codex CLI-OpenAI’s equivalent command-line coding agent, open source and built in Rust. The article ran GPT-6 Astra through it. (https://github.com/openai/codex (https://github.com/openai/codex)) Qwen3.8-27B-a locally-run open-weight model; “27B” means roughly 27 billion parameters (the internal numeric knobs the model tunes during training. More parameters generally means more capability but more memory to run). (https://huggingface.co/Qwen/Qwen3.8-27B (https://huggingface.co/Qwen/Qwen3.8-27B)) MLX-Apple’s array/machine-learning framework built specifically for Apple Silicon (M-series chips), which is what let the study run a 27-billion-parameter model locally on a Mac. (https://github.com/ml-explore/mlx (https://github.com/ml-explore/mlx)) 4-bit (quantization)-compressing a model’s numbers from their original higher-precision format down to 4 bits each, shrinking memory use (often ~75% smaller than 16-bit) so a large model fits and runs faster on consumer hardware, at some small cost to accuracy. (https://huggingface.co/blog/4bit-transformers-bitsandbytes (https://huggingface.co/blog/4bit-transformers-bitsandbytes)) Frontier model(s)-informal industry shorthand for the most capable models currently available from major labs (what GVS5H’s smaller open models are trying to match). How models behave mid-response Temperature-a setting that controls how random or deterministic a model’s output is; low temperature sticks to the most likely wording, high temperature varies more. Relevant here because the article notes Claude Code, not the raw API, controls this setting itself. (https://platform.claude.com/docs/en/about-claude/glossary (https://platform.claude.com/docs/en/about-claude/glossary)) Context (context window)-the total amount of text (measured in tokens) a model can “see” at once, covering everything fed in plus everything it generates back. (https://platform.claude.com/docs/en/build-with-claude/context-windows (https://platform.claude.com/docs/en/build-with-claude/context-windows)) Tokens-the chunks (roughly word-pieces) a model reads and writes text in; usage, context limits, and API costs are all measured in tokens, which is why the article reports “6.3 million recorded output tokens.” (https://platform.claude.com/docs/en/about-claude/glossary (https://platform.claude.com/docs/en/about-claude/glossary)) Safety classifiers / fallback model-Anthropic’s Fable and Opus models can decline (refuse) a request via an automated safety check, and the product can silently retry that same request on a different Claude model. The article’s 32 excluded episodes trace to this documented behavior. (https://platform.claude.com/docs/en/build-with-claude/refusals-and-fallback (https://platform.claude.com/docs/en/build-with-claude/refusals-and-fallback)) Message ID / assistant event(s)-the identifiers and discrete chunks a coding agent’s API stream is broken into; the article’s other 14 exclusions came from the CLI emitting two separate response events under a single message ID, which its strict identity check treated as a red flag. The benchmark and its scaffold GVS5H-the open-source research project this study adapts: a “manager-and-worker” pattern where fresh instances of one model plan, divide, and execute work by coordinating through files on disk, aiming to let smaller open models match frontier ones on hard coding tasks. (https://github.com/slee-persis/GVS5H (https://github.com/slee-persis/GVS5H)) Manager-and-worker scaffold / multi-agent orchestration-running several model calls in coordinated roles (one planning, others executing) instead of one model answering in a single shot. What the article’s “ledger mode” is an implementation of. Ledger mode-this study’s specific three-mode label for the GVS5H-style scaffold: a manager plans, separate “worker” calls execute, and notes persist on disk between calls within one episode. Episode-this study’s unit of measurement: one complete attempt at one task, in one mode, by one system. 648 episodes = 24 tasks × 3 systems × 3 modes × 3 repetitions. LiveCodeBench-a public benchmark dataset of competitive-programming problems (pulled from sites like LeetCode and AtCoder) used to test whether a model’s code actually passes hidden tests, not just looks plausible. LiveCodeBench dataset on Hugging Face (https://huggingface.co/datasets/livecodebench/code_generation_lite (https://huggingface.co/datasets/livecodebench/code_generation_lite)) AtCoder-a Japanese competitive-programming contest platform; a source of some of the study’s “hard” problems. (https://atcoder.jp/ (https://atcoder.jp/)) LeetCode-a widely used platform for practicing coding-interview and algorithm problems; the other source of the study’s “hard” problems. (https://leetcode.com/ (https://leetcode.com/)) Hidden tests-the test cases a coding benchmark uses to actually grade a submission, kept separate from any “public” example tests the model can see. So a passing score means the code generalizes, not that it memorized the visible example. The statistics Paired analysis-comparing two conditions (e.g., ledger mode vs. single-call mode) on the same set of tasks, task by task, rather than comparing overall averages, is a more sensitive way to detect a real difference. Holm correction (Holm–Bonferroni method)-a statistical adjustment applied when you run several significance tests at once, to keep the odds of a false “it worked!” finding from stacking up across all those tests. It’s why the article’s p-value changes from 0.0625 raw to 0.375 adjusted. Wikipedia: Holm–Bonferroni method (https://en.wikipedia.org/wiki/Holm%E2%80%93Bonferroni_method) (https://en.wikipedia.org/wiki/Holm%E2%80%93Bonferroni_method)) p-value / statistical significance-a p-value estimates the odds of seeing a result this strong by chance alone if there were actually no real effect; “significant” conventionally means that odds is low enough (commonly below 0.05) to trust the effect is real rather than noise. The article’s adjusted p-value of 0.375 is well above that bar, hence “suggestive, not significant.” Wikipedia: p-value (https://en.wikipedia.org/wiki/P-value (https://en.wikipedia.org/wiki/P-value)) Get full access to As The Geek Learns at astgl.com/subscribe (https://astgl.com/subscribe?utm_medium=podcast&utm_campaign=CTA_4)