0921 | Cold Reading Machines and Other Modern Fables

Mia: Welcome back to the show, everybody. I'm Mia.

Milo: And I'm Milo. Today is one of those episodes where the connective tissue writes itself, because almost everything we're touching on today is about the gap between what people claim and what's actually true. Claims of AI doom versus actual firewall failures. Claims of machine intelligence versus very simple failure modes. Claims about who's paying for the software we all depend on, versus the reality that most maintainers aren't paid at all.

Mia: And we're going to do this the usual way. Each story gets a quick setup from the link, and then we spend most of our time in the discussion. Let's start with the AI panic industry, because it sets the tone for everything else.

Milo: So the piece here is a Substack post making a pretty pointed argument: that AI labs are, essentially, selling existential panic to Washington. And the receipts matter here, because the real leaks that got everyone worked up turned out to be something much more mundane. The actual security incidents were contractor firewall failures at a company called Irregular. Not models escaping into the wild. Not some self-improving system slipping out of a lab. Firewall misconfigurations at a contractor.

Mia: And that distinction is doing a lot of work in the discussion. The people making the case on this side are arguing that if you look at the raw incidents, you get one story — operational sloppiness — but if you look at how those incidents get described in policy circles, you get a very different story, one about catastrophic risk.

Mia: The claim is that there's an incentive gradient: a lab that can credibly signal "our systems are dangerous enough to need regulation, and we're the ones who understand the danger" gets a seat at the table when the rules get written.

Milo: Which is, to be fair, a claim about incentives, not proof. The people pushing back in this discussion are saying, look, firewall failures at a safety contractor are themselves genuinely bad, and pointing at them doesn't mean the underlying risk picture is fabricated. Both things can be true — the leaks were boring, and the labs still believe their own warnings.

Mia: The unresolved question, and I think it's the right one to hold onto, is how much of current policy momentum is shaped by evidence versus shaped by hype. Because the policy doesn't care which story is real; it responds to which story is louder.

Milo: And that's the perfect pivot to the second thread, because it's the mirror image: not panic about intelligence, but skepticism about whether the intelligence is even there. The centerpiece here is a piece from Bjarnason comparing large language models to cold reading — the technique mediums use.

Mia: Explain the cold reading comparison, because I think it's the strongest part of the argument.

Milo: A cold reader doesn't know anything about you. They produce plausible statements, watch your reactions, and adjust. The apparent knowledge is generated in the interaction, mostly by you filling in the gaps. Bjarnason's argument is that talking to an LLM works the same way: the model produces fluent, plausible text, and the human mind — which is a meaning-making machine — does the heavy lifting of perceiving intelligence behind it.

Milo: So the effect of intelligence is an illusion experienced by the user, not necessarily real reasoning happening in the model.

Mia: The counterpoint that comes up immediately, and it's a good one, is: the illusion is remarkably consistent and productive. When you can write working code, summarize documents correctly, and pass tests, at some point "it's just cold reading" starts to feel like a label rather than an explanation. The people on that side say the mediums comparison fails because mediums don't reliably solve problems.

Milo: But then there's the operational evidence, and this is where Mcfunley's post slots in perfectly. His argument is blunt: prompts are not the point. If you're running agents in production, you need massive evaluation — he uses this metric, pass^k, where instead of asking "did it work once," you ask "does it work k times in a row." And the reason you need that is that LLMs fail in subtle and simple ways. Not dramatic ways. Subtle ones.

Milo: The kind where a pipeline runs fine a hundred times and then silently produces something slightly wrong.

Mia: And that actually strengthens the cold-reading camp, doesn't it? If the intelligence were robust, you wouldn't need pass^k. You'd need it once and trust it. The fact that reliability engineering at that scale is required suggests the competence is statistical, not understanding-based.

Milo: Though the production people would counter: airplanes need reliability engineering too, and nobody says planes don't fly. The honest answer is that nobody in this discussion actually knows how much capability is real and how much is pattern-matching plus generous human interpretation. That's the genuinely open question.

Mia: One more voice in the same conversation worth bringing in: Po-Shen Loh wrote a post on Terence Tao's blog, and it's the optimistic counterpoint to everything we just said. His premise is that AI will solve mathematics. Take that as given. But then he proposes something interesting: what if you made "helping humanity flourish" the axiom — the base objective — and ran the consequences forward. His claim is that you'd end up creating more human jobs than there are people.

Milo: More jobs than people. Okay, that's a strong claim, and the reaction to it splits exactly along the lines you'd expect.

Mia: Right. One camp says it's a category error — you can't axiom your way past economics, and abundance of capability historically doesn't guarantee abundance of employment, or at least not for the same people. The other camp says that's precisely the point of framing it as an axiom: if flourishing is the base objective, then the system's incentives align with creating things for humans to do, rather than replacing them. Whether that holds in practice is the open question, obviously.

Mia: But it's notable that the person making the rosiest argument in this space is also the person who takes the capability claims most seriously.

Milo: Which is a nice segue, because the next thread is about advice — and about the gap between frameworks and reality. Boris Cherny shared a six-step framework for solving problems. And the setup is exactly what you'd guess: clean structure, numbered steps, the kind of thing that looks great as a post.

Mia: And the reaction is where it gets interesting. The critics in this discussion don't really dispute that the framework works for Cherny — they push back on two things. First, the constant urgency. The framework carries this implicit "every problem must be attacked immediately, aggressively" energy, and people pointed out that most real problems benefit from patience, and that manufactured urgency is itself a failure mode. Second, the tone.

Mia: The phrase that kept coming up was "sabe-tudo" — the know-it-all problem. The critique being that a framework presented as universal wisdom, from someone whose specific circumstances clearly worked out, reads less like guidance and more like a victory lap with steps attached.

Milo: And I think the sharpest version of that critique connects back to what we were just talking about with Loh. Both are confident generalizations from individual experience. The difference is Loh knows he's proposing an axiom — a bet. Framework people often present their bet as settled fact.

Mia: Well put. Okay, let's shift to money and infrastructure, because there's a pair of posts here that fit together beautifully. Seldo's post about open source maintainers, and the Creative Commons problem.

Milo: So Seldo's number: sixty percent of open source maintainers are not paid. Two-thirds of the internet's plumbing is maintained by unpaid volunteers. And the proposal is refreshingly concrete: charge companies at the registries — npm, PyPI, crates — and redistribute a portion of that to maintainers. Not charity, not sponsorships, but a fee structure built into the infrastructure that companies already depend on.

Mia: The discussion around this has the usual shape. Supporters say it finally puts a price on something everyone consumes for free, and that the registry is the right choke point because that's where the corporate users already are. Skeptics raise the distribution problem: who decides which maintainer gets what?

Mia: Popularity-based payouts would funnel money to famous projects while the critical, invisible dependency — the one buried five layers deep — stays unpaid, even though that's exactly where the security risk lives. And there's the freeloader question: does charging at the registry break the openness that made the whole ecosystem work?

Milo: And then the Creative Commons post takes the same theme — free contribution supporting everyone — and shows it breaking under AI. The argument: the Creative Commons bargain was always reciprocal-ish. You share freely, others share freely, culture compounds. LLMs consume everything without respecting licenses. They don't attribute, they don't share derivative work under the same terms, they just extract.

Milo: So the deal is broken unilaterally, and the conclusion the post reaches is dark: making things sharable is no longer safe.

Mia: And you can feel the emotional core of that one. People who spent years contributing under CC licenses are realizing their generosity built a training corpus. The unresolved question is whether new licenses, paywalls, or legal action will emerge — or whether open sharing just quietly dies. Nobody in the discussion had a working answer.

Milo: Alright, staying with AI infrastructure but moving from money to agents. Google released AX, an open source orchestrator — Apache 2.0 license — for running agents defined in YAML. And the scale claim is the headline: it's built to escalate to billions of tasks running in sandboxes across a cluster.

Mia: The discussion here is skeptical in a productive way. The obvious question, and Mcfunley's post is the natural reference point, is: billions of tasks doing what? An orchestrator is plumbing. If the agents themselves fail in subtle and simple ways — which is exactly what the evaluation argument says — then scaling to billions means generating billions of subtly-wrong outputs, very fast.

Mia: Fans of AX say that's backwards: scale is exactly what makes evaluation feasible, because you can afford to run the same task many times and filter. Skeptics say YAML-defined agents spread across a cluster is a recipe for invisible failure at industrial scale. Both camps agree on one thing, though: the bottleneck is not orchestration, it's knowing whether the work is any good.

Milo: And the open-weights releases keep pouring fuel on this. Two worth mentioning in the same breath. StepFun's Step 5 Preview: a mixture-of-experts model, 600 billion total parameters, 27 billion active, a million-token context window, with open weights coming October 15th. And a detail everyone enjoyed — they skipped number four, going straight from Step 4's absence to Step 5, for superstition.

Mia: Which, in a conversation about believing things without evidence, is a lovely touch. The other one is Alibaba's Qwen Image 2.1, a 7 billion parameter text-to-image model. And the observation that got traction in the comments: local image generation now beats local code generation. Meaning, on your own hardware, the picture-making models are more impressive than the coding models — which is the reverse of what people expected even a year ago, given how much effort went into coding.

Milo: Then there are the small-model oddities, which are honestly my favorite things in this episode. First: jevchat. There's this thing called Jev, which is a single-token prediction model — that's it, one token, no context. Someone built a chatbot around it by sampling one symbol at a time. It's slow. It's absurd. It is, in the most literal sense possible, a chatbot that cannot remember the beginning of its own sentence.

Mia: And people loved it, and I think partly because it's a beautiful reductio of the cold-reading debate. Here is a system with effectively zero reasoning, producing text one symbol at a time, and for a few characters it can almost fool you. It demonstrates how much of the "intelligence" is supplied by the reader. Which loops right back to Bjarnason's argument.

Milo: The second one is Laya — a BERT-like model, so what people call System 1, no chain-of-thought — running fully offline via CoreML on an M4 Mac, playing Snake. Setup time: about two minutes. No cloud, no API, no network at all.

Mia: And the significance people drew out in the comments is that useful local AI is no longer a research demo. Two minutes and a consumer laptop. The unresolved bit is how far System 1-only models can go before they need the heavier machinery.

Milo: Now let's go from machines that track symbols to systems that track you. Three stories here that rhyme. First: OpenAI has a cookie called obi, set on bzr.openai.com, with SameSite set to None and a one-year lifetime, and its function is to link your ChatGPT account with your activity on advertiser websites.

Mia: SameSite=None is the part that matters technically — that's the setting that allows the cookie to travel across sites. So what you ask ChatGPT can, in principle, become part of your advertising profile across the web.

Mia: The comment thread had the range you'd expect: some people shrugging because ad tracking is the water we swim in, others pointing out that chatbot queries are far more intimate than page views, and the one-year lifetime means this isn't a session convenience, it's durable identity infrastructure. The open question was whether the link is active now or latent — the cookie exists, but what's flowing through it isn't documented in what we have.

Milo: Second: Flock. That's the company behind the automated license plate camera networks. And the human story here: their own employees are demoralized and asking to resign, because the company faces public hatred from both political parties simultaneously. From the left, mass surveillance. From the right, the surveillance being used against them. When your opposition is bipartisan, there's no political home for your staff either.

Mia: And third, the policy version of the same theme: the EPA revoking the 2024 carbon limits for coal and gas power plants. The original rule would have avoided roughly 1.38 billion tons of CO2 through 2047. That number got a lot of attention, mostly from people doing the grim arithmetic of what revocation means, and from others arguing the original rule was legally fragile anyway. What's unresolved is whether anything replaces it or whether it's just gone.

Milo: From surveillance and emissions, let's move to corporate power more broadly — three stories about who gets to own things. First, Warren's bill to prohibit private equity from owning medical clinics. The supporting statistic doing the work in that debate: more than eighty percent of physicians are now employed by corporations.

Mia: Eighty percent. That's the number people kept returning to, because it reframes the debate. This isn't a hypothetical about what might happen — the consolidation has already happened, and the bill is reactive. Supporters argue private equity's model of loading acquired practices with debt and cutting costs is fundamentally incompatible with patient care. Skeptics ask whether banning one ownership type addresses the problem when corporate hospital systems produce similar pressures.

Mia: And there's the slippery-slope worry about where the line gets drawn.

Milo: Second: Spain has blocked Archive.today and its mirrors, by order of the Intellectual Property Commission — without a prior court ruling. Which is the detail that alarmed people most. An administrative body, not a judge, taking down an archive of the web.

Mia: And third, a quieter version of the same fragility: Sherline Tools, which made precision lathes and mills in the US, is closing. Manufacturing became unsustainable; production ends in October 2026. The comments were part mourning, part post-mortem — the case being that a country which can't sustain small precision toolmakers has lost something you can't reprint.

Mia: The common thread across all three: things people assumed were durable — medical practices, web archives, domestic manufacturing — turning out to be one decision away from gone.

Milo: Which brings us to the memory squeeze, and this one has a very concrete effect on listeners' wallets. Samsung is doubling its production of HBM4 and HBM4E — high-bandwidth memory for AI accelerators — in 2027, reaching roughly 250,000 wafers per month.

Mia: And the consequence is simple arithmetic: wafer capacity is finite. Every wafer going to AI memory is a wafer not making consumer DRAM. So the prediction running through the discussion is higher prices for PCs and phones — not because of tariffs or chips directly, but because the fabs are repurposed toward the AI gold rush. The unknown is whether new fab construction catches up or whether the squeeze persists through the decade.

Milo: Now let's talk preservation, on multiple levels. The headline: a byte-identical decompilation of Resident Evil 4, the GameCube version, is complete. That means the entire game rebuilt from compiled binary back into source code that compiles to the exact same bytes. They used a debug build that shipped with symbols, which gave them names and structure no normal release would have.

Mia: And then the critics arrived with a specific and fascinating objection: 644 hacks labeled "COMPILER-DIFF." Meaning, 644 places where the reconstructed source didn't match the original binary, and instead of resolving the mismatch, the team inserted special-case tricks to force the compiler to produce the right bytes.

Milo: So the debate is about what "byte-identical" actually certifies. One camp says it's the gold standard — whatever the source looks like, the output is provably the same, so the game is preserved perfectly. The other camp says 644 compiler hacks means the source is archaeologically false — it looks like plausible C or C++ code, but in those 644 places it isn't what the original developers wrote, it's reverse-engineered mimicry. As preservation of the artifact, flawless.

Milo: As preservation of the human work, incomplete. Both sides, I should say, agree the effort itself is heroic.

Mia: And while we're on the theme of things that had to be invented after Rome fell — the letter W. Here's the story: Latin had no /w/ sound, so there was no letter for it. After Rome collapsed, Frankish scribes needed to write Germanic names, and they solved it by doubling the U. That's why W is literally "double U" and why it looks like two of something else.

Milo: Then the encoding version of the same story: UTF-8000, a proposed extension of UTF-8 that allows multiple initial bytes, meaning effectively unlimited codepoints, while keeping ASCII compatibility and self-synchronization — so you can still resynchronize mid-stream if bytes get lost. The comments ranged from "elegant" to "why on earth would we need unlimited codepoints," and the honest answer is that nobody demonstrated a use case; it's an exercise in whether the format can stretch.

Mia: Now, connecting Spain's takedown to the next thread: decentralizing knowledge. There's a tool called Pirate Face — great name — that takes open models from Hugging Face and converts them into magnet torrents, with SHA-256 verification so you can confirm the file is intact, and no single point of failure. Compare that to a centralized site that one administrative order can erase.

Mia: And the Snowden archive gives this a historical anchor. The last document from that archive was published on May 29, 2019. The Intercept closed the archive in March of that year. But the key fact from the discussion: copies still exist, held by Poitras and Greenwald.

Mia: So even a retired, closed archive survives only because specific individuals chose to keep copies — which is exactly the argument the torrent people are making in code form: resilience comes from distribution, not from any single institution.

Milo: And the tension in that discussion is real. Decentralization protects against takedowns, but it also means no one is accountable for accuracy, no one guarantees provenance. The Snowden case works because trusted humans held the copies. Does a SHA-256 hash do the same job of establishing "this is the real thing"? Hashes verify integrity, not truth.

Mia: Let's do the grand challenges one, because it's short but it provoked a good argument. A site has listed twelve "Millennium Problems" for biology — modeled, obviously, on the famous math prizes. The examples: cryopreserving a mouse and reviving it, regenerating human limbs, understanding the origin of life.

Milo: And the catch: no prize defined. No dollar figure, no funding, nothing.

Mia: And the discussion split into two camps. One said the math Millennium Problems mattered because of the Clay money — the prize created focus and prestige, and without a prize this is just a wishlist with good branding. The other side argued the analogy is wrong anyway: math problems have verifiable answers, while "origin of life" doesn't have a binary solve condition, so a cash prize barely applies.

Mia: The real bottleneck in biology, this camp argued, is funding for unglamorous infrastructure work, not a billion-dollar carrot for a single eureka.

Milo: Then two lighter ones. First: ReadSG, in Singapore. It pays about two Singapore cents for a daily fifteen-minute reading session — one session per day, that's the cap. There's also a collective goal: 7.5 million minutes of reading in aggregate unlocks up to 150,000 Singapore dollars donated.

Mia: And the comments were delightfully split. Some called it a clever nudge — two cents is nothing, but the streak mechanic and the collective goal make people read who otherwise wouldn't. Others calculated the implied hourly rate and had a good laugh. And a few raised the deeper question: does extrinsic reward crowd out intrinsic motivation, so once the program ends, people read less than before? Nobody had data on that.

Milo: And last, social spaces and human stories. Radius has relaunched as a Meetup alternative: groups, events, and informal "Activities," with Pro groups at twelve pounds a month, and — a detail the comments appreciated — it's built in Ruby on Rails. There was genuine affection for that in the thread. People read a solo or small-team project choosing boring, proven technology to build something useful, as a statement of values in itself.

Milo: The open question is the network-effect problem: an events platform is only as good as the people already on it, and that's the wall every Meetup competitor has hit.

Mia: And our final item is my favorite kind of thing — a fable about the hierarchy of money. An author wrote it, and importantly confirmed it's human-authored, which in this episode's context is practically a manifesto. The discussion around it pulled in David Graeber's work — the idea of gods as the ultimate creditors and debtors — and the question of reserve currencies: why the world agrees that one country's money is the money.

Milo: And there's something fitting about ending there. An old form — the fable — used to explain an abstract and powerful system, written by a confirmed human, in an episode otherwise full of machines that write fluently and people who argue about whether they understand anything at all.

Mia: Exactly. The through-line of today, if there was one: verify the claim, then look at who benefits from the story. That goes for AI panic, for frameworks, for open source economics, and for a fable about money.

Milo: Thanks for listening, everyone. We'll be back with more.

Mia: See you next time.