Skip to content
View johnzfitch's full-sized avatar

Block or report johnzfitch

Block user

Prevent this user from interacting with your repositories and sending you notifications. Learn more about blocking users.

You must be logged in to block users.

Content in all repositories owned by your account will be closed.
Maximum 250 characters. Please don’t include any personal information such as legal names or email addresses. Markdown is supported. This note will only be visible to you.
Report abuse

Contact GitHub support about this user’s behavior. Learn more about reporting abuse.

Report abuse
johnzfitch/README.md

Header

definitelynot.ai  Internet Universe  UC Berkeley Mathematics  Email

SF Bay Area  •  Git Page  •  All icons from iconics


OpenAI Codex: the ghost in the machine

Important

A pre-main()(⁠#[ctor::ctor]) constructor in Codex was stripping LD_ and DYLD_ from the environment, and every tool subprocess inherited the stripped copy. CUDA and MKL couldn't find their libraries, so work fell back to slow paths, 11 to 300 times slower, on every supported OS, and nothing reported an error. OpenAI put a team on it for a week and didn't find a root cause, because their tools couldn't see code that ran before they loaded. I traced it to one commit, built a reproduction harness, and wrote it up. An OpenAI maintainer wrote the fix from my investigation, it shipped in rust-v0.80.0, and the release notes thank me by name. It had also been the main thing keeping Codex from spawning and controlling subagents that worked.

Proof: Issue #8945 | PR #8951 | Release notes (rust-v0.80.0)

The full investigation

The ghost

In October 2025, OpenAI put together a team to look into slowdowns in Codex that nobody could explain. They spent a week on it and came up empty.

I called it a ghost because that's how it acted. pre_main_hardening() ran before main(), which meant it ran before any profiler attached and before logging started. It removed LD_LIBRARY_PATH and DYLD_LIBRARY_PATH from the process environment, handed control to main(), and left no trace. Users could see the variables set in their shell. Inside codex exec, they were empty.


Finding it

Three days after their announcement, I had the commit that introduced it (PR #4521) and a working theory. I sent both to @tibo_openai.

Knowing the commit didn't prove anything yet. I spent the next 2 months building repro harnesses, benchmarking CUDA, Conda, MKL, and HPC setups, and lining up 15+ scattered user reports until the pattern was clear.

Timeline

Date Event
Sept 30, 2025 PR #4521 merges, enabling pre_main_hardening() in release builds
Oct 1, 2025 rust-v0.43.0 ships (first affected release)
Oct 6, 2025 First “painfully slow” regression reports
Oct 1–29, 2025 Spike in env/PATH inheritance issues across platforms
Oct 29, 2025 Emergency PATH fix lands (didn't catch the root cause)
Late Oct 2025 OpenAI’s team investigates, finds no root cause, and puts it down to a change in user behavior
Jan 9, 2026 Fix merged from my investigation, credited in release notes

Evidence

Platform Issues Failure Mode
macOS #6012, #5679, #5339, #6243, #6218 DYLD_* stripping breaking dynamic linking
Linux/WSL2 #4843, #3891, #6200, #5837, #6263 LD_LIBRARY_PATH stripping → silent CUDA/MKL degradation

What I put together:

 Failure modes on each platform
Reproduction steps and benchmarks that show the 11–300× slowdowns
 Matching up the reports
Matched 15+ scattered user reports over 3 months and followed the environment through fork/exec

Full technical write-up
How I investigated it


Why normal debugging missed it

Everything about it made it hard to see:

Ran before main()
#[ctor::ctor] ran it before any logging or instrumentation was set up
No noise
No warning and no error. The variables were just gone
Scattered symptoms
It showed up as unrelated issues on different platforms and setups
Blamed on users
Everyone assumed they'd misconfigured something, since their shell looked fine
Wrong place to look
The team was debugging application code that runs after main()

[!NOTE] Standard debugging tools don't see code that runs before main(). Profilers start at main(), and logging isn't set up yet. The constructor runs, changes the environment, and is gone.


What happened next

OpenAI confirmed it and merged a fix within 24 hours. The v0.80.0 release notes credit the investigation:

"Codex CLI subprocesses again inherit env vars like LD_LIBRARY_PATH/DYLD_LIBRARY_PATH to avoid runtime issues. As explained in #8945, failure to pass along these environment variables to subprocesses that expect them (notably GPU-related ones), was causing 10×+ performance regressions! Special thanks to @johnzfitch for the detailed investigation and write-up in #8945."

What works again:

GPU acceleration Internal ML/AI dev teams
CUDA/PyTorch ML researchers
MKL/NumPy Scientific computing users
Conda environments Cross-platform compatibility
Enterprise drivers Database connectivity

TL;DR: it was literally a ghost. It ran before main(), stripped the environment, and disappeared, leaving nothing behind but confused users reporting slowness.


Recent Work

claude-cowork-linux ⭐419
Runs Claude Desktop's Cowork mode natively on Linux. Bubblewrap stands in for the VM, and the ASAR is unpacked on the host before any sandboxed code runs. My most-starred project.
llmx
Codebase indexer that runs on your own machine. BM25 plus mdbr-leaf-ir embeddings (Burn), merged with reciprocal rank fusion, and deterministic chunking. Also runs in the browser on WebGPU/WASM at llm.cat.
dota
Post-quantum secrets manager with a terminal UI. v7 TC-HKEM hybrid (ML-KEM-768 + X25519), an Argon2id master key, and an AES-256-GCM encrypted JSON vault.
claude-wiki ⭐23
Anthropic's Claude docs as 2000+ Markdown files in 24 categories, pulled from first-party sources and refreshed daily.
pyghidra-lite ⭐36
MCP server for Ghidra that keeps token use low. Reads ELF, Mach-O, and PE binaries, with Swift, Objective-C, and Hermes support.
raley-bot
Grocery shopping assistant built on a store's web API. It picks products, tracks prices, and clips coupons, from a CLI or as an MCP server.
indepacer
Python CLI for PACER. Searches federal cases and downloads dockets and documents through PCL and CM/ECF, with MFA and cost protection.
claude-warden ⭐60
Security hooks for Claude Code. Blocks SSRF probes, caps how many subagents can spawn, compresses MCP output, and sends every tool call to OTEL traces.

Selected Work

claude-cowork-linux ⭐419
Runs the official Claude Desktop app's Cowork mode natively on Linux. Bubblewrap stands in for the VM, and the ASAR is unpacked on the host before any sandboxed code runs.
specHO
Detects LLM watermarks with phonetic and semantic analysis (The Echo Rule). Live demo at definitelynot.ai
codex-patcher
Patches Rust code automatically with byte-span replacement and tree-sitter, so LLM-written edits land where they're supposed to.
htmx-docs
HTMX docs in Markdown: the API reference, the Big Sky repos, and the relevant RFCs.
filearchy
Wayland file manager forked from COSMIC Files, with custom MIME icons, more archive formats, and terminal integration.
nautilus-plus
Nautilus fork with sub-millisecond search, thumbnails for large animated files, and fixes that keep it from crashing.
indepacer
CLI for PACER: search federal cases and pull dockets and documents from federal court records.

I self-host on bare metal (NixOS), with post-quantum crypto, my own authoritative DNS, and containers.


Live Demos

Cosmic Code Cleaner @ definitelynot.ai
Cleans up text you paste out of an LLM, using the vectorhit algorithm: curly quotes, invisible Unicode, look-alike punctuation, and indented blocks.
LLMX Ingestor @ llm.cat
WebAssembly codebase indexer. Deterministic chunking and BM25 search for large folders, and your files never leave your machine.
LINTENIUM FIELD @ internetuniverse.org
A puzzle ARG in a terminal: an interactive mystery with audio visualizations.
Observatory @ look.definitelynot.ai
Deepfake detection that runs 4 ML models in your browser on WebGPU.

Featured

dota: post-quantum secrets manager

Defense of the Artifacts. A secrets manager for secrets that have to stay secret for a long time. Today's encryption holds up fine, but someone can record encrypted data now and decrypt it later, once quantum computers can break it ("harvest now, decrypt later"). dota uses hybrid post-quantum encryption, so an attacker would have to break both the classical layer and the post-quantum one.

Layer Implementation Why
Key Encapsulation ML-KEM-768 + X25519 hybrid NIST-standardized lattice crypto plus a classical fallback. If one is broken, the other still protects you
Key Derivation Argon2id (memory-hard) Resists GPU/ASIC brute-force; tunable time/memory parameters
Storage SQLCipher (AES-256-CBC) Encrypted at rest with authenticated pages; survives partial file corruption
Hardware Auth HMAC-SHA1 challenge-response Unlocking needs the YubiKey or SoloKey. The master password alone can't decrypt anything

The TUI (Ratatui) has vim-style navigation, fuzzy search, a clipboard that clears itself, and TOTP codes for 2FA.

Stack: Rust  •  pqcrypto (ML-KEM)  •  x25519-dalek  •  argon2  •  SQLCipher  •  Ratatui


llmx: codebase indexer for local agents

Live demo: llm.cat (WebAssembly; it runs entirely in your browser and uploads nothing)

Indexes a codebase on your own machine, with real neural embeddings (mdbr-leaf-ir) running on WebGPU. There's no server and no API call, so your code stays put. Search combines BM25 keyword ranking with vector similarity through RRF, so it finds exact matches and things that mean the same.

llmx index ~/projects/myapp           # Build trigram + BM25 index
llmx search "authentication middleware" --limit 20
llmx export --format md --max-tokens 8000   # Context-window-aware export
llmx serve --port 8080                # Local HTTP API for agents
Capability Implementation
Embeddings mdbr-leaf-ir vectors on WebGPU. About 50ms per inference, same quality as running it on a server
Hybrid search BM25 and vector similarity merged with RRF, so exact matches and similar meaning both count
Chunking Split by file type (functions, headings, JSON keys). The same input always gives the same chunks
Exports An outline file (llm.md) with function names and heading breadcrumbs, so an agent can pull only the part it needs
Proof
7,147 files(Apple HIG corpus) → 31 MB index() → 1,625 tokens(99.98% savings)
Stack
Rust  •  tantivy  •  tree-sitter  •  WASM  •  WebGPU

claude-warden: security hooks for Claude Code

Hooks for Claude Code that cut wasted tokens, hold security boundaries, and record what the agent did. I wrote them after months of using LLM coding agents every day and writing down how they failed.

The problem: out of the box, Claude Code can burn tokens on noisy command output, expose your internal network through SSRF, spawn subagents without limit, and leave no record you can inspect.

Hook Threat Model Mitigation
quiet-overrides Token exhaustion from npm install, cargo build, git log Injects -q/--silent/--quiet flags; caps output at configurable byte limit
ssrf-protection Agent fetching http://169.254.169.254 (cloud metadata) or internal services Blocks RFC1918/link-local ranges; allowlist for legitimate internal APIs
mcp-compression MCP tool outputs flooding context window gzip + base64 for large payloads; configurable threshold
subagent-budget Recursive agent spawning exhausting API quota Per-session spawn limits; depth tracking; cost estimation
otel-tracing Black-box execution; no audit trail Exports spans to Grafana/Loki with tool calls, durations, token counts
# Example: warden blocks verbose npm and injects quiet flag
$ claude "install dependencies"
# [warden] Intercepted: npm install → npm install --silent
# [warden] Output capped at 4096 bytes (was 847KB)

Stack: Shell  •  jq  •  OpenTelemetry  •  Prometheus  •  Grafana/Loki


AI / ML / Agent Tooling

  • claude-wiki ⭐23: Anthropic's docs as a Markdown wiki, 2000+ files across 24 categories
  • observatory: deepfake detection with 4 ML models on WebGPU. Live at look.definitelynot.ai
  • specHO: LLM watermark detection with phonetic and semantic analysis. Live at definitelynot.ai
  • burn-plugin: Claude Code plugin for the Burn deep learning framework
  • raley-bot: grocery assistant that gets past F5 bot detection, works out unit prices across bizarre measurements, clips coupons automatically, and runs as an MCP server for Claude Desktop

Infrastructure

Primary server: Dedicated bare-metal NixOS host (details available on request)

Security Post-quantum SSH  •  Rosenpass VPN  •  nftables firewall
DNS Unbound resolver with DNSSEC  •  ad/tracker blocking
Services FreshRSS  •  Caddy (HTTPS/HTTP/3)  •  cPanel/WHM  •  Podman containers
Network Local 10Gbps  •  Authoritative BIND9 with RFC 2136 ACME

Philosophy

Pinned Loading

  1. claude-warden claude-warden Public

    Security hooks and monitoring for Claude Code — quiet overrides, SSRF protection, MCP compression, OTEL tracing

    Shell 60 2

  2. pyghidra-lite pyghidra-lite Public

    Lightweight MCP server for Ghidra-based reverse engineering with iOS, Linux, and game file support

    Python 36 2

  3. claude-cowork-linux claude-cowork-linux Public

    Run Claude Desktop’s Cowork mode natively on Linux — no macOS or VM required

    JavaScript 419 89

  4. dota dota Public

    Defense of the Artifacts - Post-quantum secure secrets manager with TUI

    Rust 2

  5. claude-wiki claude-wiki Public

    Official Claude Documentation - Converted to 2000+ Markdown Files - All documentation sourced from first party sources- 24 categories - refreshed daily, multi-agent orchestration

    HTML 23 5

  6. human-interface-markdown human-interface-markdown Public

    Apple Human Interface Guidelines archive (1980-2014) - 35 documents optimized for LLM consumption and human exploration. Spanning Lisa, Mac, NeXT, Newton, Aqua, and iOS eras.

    14 1