The Shelf / QA & Shipping / Deep QA Audit Loop

QA & Shipping

Deep QA Audit Loop

Your agent keeps hunting bugs until a full pass turns up nothing serious. No fixed number of rounds.

The job: go over an app until it actually works — every screen, every flow, every path through the code — and don't call it clean until the written stop rules say so.

Most agent QA is one pass. It skims the path where nothing goes wrong, says "looks good," and marks its own homework. This replaces that with a loop. A wide sweep across seven areas. Then a walk through every screen, start to finish. Then a pass that tries to prove each fix wrong. Then a re-check aimed only at what just changed — narrowing until a whole pass turns up nothing serious. Eleven written stop rules. No "want me to keep going?" pauses.

What makes it different

The bug list. Every bug in it was found the hard way in a real shipped app, and several got through standard audits before the check that catches them existed. A goal the app counted as hit when the user had never set one — one stray entry, and it threw a 365-day streak party. A "saved" screen painted over a write to disk that never happened. A security fix that, on an empty value, would have wiped the app's entire data folder. A tap-a-link handler that read fine in three audits while the phone never actually called it. Each one ships with the code search, or the live tap-through, that catches it.

How the checking is done matters as much as what gets found. A shallow pass that finds nothing looks exactly like a clean app. A checker that quietly stopped working — it hit a limit, or it errored out — looks exactly like a clean bill of health. And a fix reviewed by whoever wrote it proves nothing. The loop has an answer for all three.

What's inside

SKILL.md — the loop, the seven areas, the stop rules, the argue-against-your-own-fix pass, how to split work between models, and the closing scorecard
references/ios-simulator-audit.md — running the audit on the simulator, the fake iPhone your Mac runs: tapping through the app, the SwiftUI traps, the ways saved data goes missing, what the fake phone can't tell you about payments, and when to move to a real one
QUICKSTART.md — how to install it in Claude Code, claude.ai, and Codex, with a first prompt to paste

Who it's for

Anyone shipping with an agent — web apps, back-end services, and iPhone apps headed to Apple. The main loop doesn't care what you're building. The iPhone material is extra depth for the people who need it.

Why not a free directory download

A free directory hands you a prompt somebody wrote once. This one has been run against a working portfolio of shipped apps for months, and every bug that beat it became a permanent check. It gets updated: a new kind of bug survives an audit, the file gains the check that would have caught it. Field-tested, documented, issued as-is.

FIELD REPORT real output, not a promise

Setup: a Layer 3 (data & display) finding from a real audit of a nutrition-tracking app in our own portfolio — the incident that put the "degenerate threshold" class into the catalog. This is the finding format the loop produces.

Finding: degenerate-threshold comparison — `actual >= dailyTarget` with an unset target

Severity: HIGH Class: Layer 3 — data & display / trivially-satisfiable comparison

What was found. The app compares logged intake against a user-set daily target in 13 separate call sites (>= dailyTarget). For a brand-new user the target is 0 until calibration completes — and anything >= 0 is always true. Every one of the 13 sites therefore read "goal met" for a user who had done nothing:

Streak calculator: one stray log produced a 365-day "PERFECT" streak celebration
Daily-summary ring: rendered 100% complete on day one
Weekly report: all seven days marked hit
Milestone/badge triggers: fired off the same false positives

Why prior audits missed it. An earlier audit had found this exact bug — and guarded exactly one of the 13 call sites. The other 12 shipped across two-plus releases. The lesson written into the skill: when a threshold bug is found, grep the threshold variable name across the whole codebase in one pass; fixing the first site found does not mean the rest are guarded.

The fix. A single hasCalibratedTarget guard (target > 0) applied at the shared computation layer, so all 13 consumers gate on the same signal — plus an honest "set your target to start tracking" empty state where the false celebrations used to render.

Verification (Pass 3, adversarial). Fresh-install drive with zero calibration: streak shows 0, ring shows the empty state, no badge fires. Then a refutation probe: target set, met, then reset to 0 mid-streak — confirmed the guard doesn't retroactively wipe legitimate history.

This is one finding from one layer. A full run produces the same treatment across all seven layers, a page-by-page trace, and a closing scorecard with per-pass found/fixed/remaining counts and an explicit YES/NO on every exit condition.

SERVICE RECORD living gear — updated as the factory learns

v1.1.0 — 2026-08-25

First evolution sync — everything the loop learned in six more weeks of daily audits against a live app portfolio, folded in:

New exit condition: the submitted binary must contain the audited code. An audit blesses a commit; a submission ships a binary — and the gap between them was one re-audit's only critical finding.
New bug classes, each with the grep or drive that catches it: identical-branch ternaries that read as deliberate care, third-party fields decoded but never read, prompt/schema contract drift in LLM-backed features, discrete-set values validated only by min/max (plus the sibling string field the user actually reads), deadlines no code ever compares to today, generic head-nouns making fuzzy matches, and the == comparisons that all go stale the day a two-valued enum gains a third case.
Verification discipline: one green run of a non-deterministic path is not a measurement; a demo seed that supplies a feature's output leaves the feature untested no matter how hard you drive it; a lazy-repair hook on a route nothing calls never fires; a substring invariant must be word-bounded; grep your own diff for the very pattern you were fixing.
Data-vs-code rule: fixing the one bad record that got reported leaves the class alive — fix the code that produced it, then re-run a record nobody complained about.
iOS reference, expanded: the Release-compile gate, the stale-binary install trap, system-drawn alerts invisible to the accessibility snapshot, the screen-scale image-renderer trap, reading the sim's persisted tier off disk before trusting any pass, live-activity reclaim after force-quit, the image-only share dead end, the horizontal-pan overflow class, and a full persistence-survival check procedure.
Process: file-ownership rules for parallel fix agents, the anti-persona audit framing, whole-system integration passes when initiatives land together, and an honest note on what a clean audit does NOT prove.

Every addition was learned the hard way — a real bug in a real shipped app that survived the previous version of this loop.

v1.0.0 — 2026-07-17

First issue. Ported from the factory's internal skill: sanitized for general use, methodology intact, field report captured from a real run.

Every update ships free to owners — your locker always serves the latest version.

QUESTIONS

Is this only for iPhone apps?›

No. The main loop works as-is on web apps and back-end services. The iPhone simulator material sits in its own file and only gets used when you're auditing an iPhone app.

How is this different from just telling my agent to test the app?›

Ask an agent to test the app and it skims, then marks its own homework. This is a loop with written stop rules, a pass that tries to prove its own fixes wrong, and a list of specific bugs to hunt for — each one found the hard way in a real shipped app.

Are the bug examples real?›

Every one. Each bug in the list names the real incident behind it, with the app details stripped out. Some of them got through several earlier audits before the check that catches them was written.

Does it fix the bugs or just list them?›

It fixes as it goes — that's what the later passes are for. You get the exact file and line, how bad it is, and the fix. Then a pass that checks the fix didn't break something else.

one-time · v1.1.0 · updated 2026-08-25

Instant download (.zip) · works with Claude Code, claude.ai projects & Codex · 7-day refund, no forms.

WHAT'S IN THE CRATE

▸Four passes in a fixed order: a wide sweep, then a walk through every screen and flow, then a pass that tries to prove the fixes wrong, then a re-check of only what changed. Eleven written rules say when the agent is allowed to stop.
▸Seven things it checks: missing pieces, parts that don't talk to each other properly, wrong numbers on screen, crashes and security holes, confusing flows, a run of the live app, and — for phone apps — the app store's rules.
▸A list of real bugs it has caught before, each with the code search, or the live tap-through, that finds the same bug in your app. A goal that counts as hit when it was never set. A save screen that says 'done' over a write that never happened.
▸The habit that keeps fixes honest: the agent has to argue against its own repairs. A check that comes back quiet is treated as a check that may have failed, not as good news.
▸For iPhone apps: a separate guide to the simulator — the fake phone your Mac runs. How to tap through the app, the SwiftUI traps that bite, the ways saved data goes missing, and when you have to switch to a real phone.
▸Apple's store rules checked inside the loop, not bolted on at the end — including a sweep for store text that promises more than the app does.
▸Which model to use where. A cheap one for the plain code searches. Your best one for the pass that tries to break the fixes.
▸A closing scorecard: what was found, what was fixed, what's left. Plus a list of features worth adding, for when the code comes back clean.
▸New in 1.1: a stop rule that checks the build you sent to Apple holds the code that got audited. Three bugs from apps that call an AI model: what the app asks the model for and the answer shape it accepts drift apart, so the feature returns nothing at all; a deadline no code ever checks against today's date; a screen that only looks right because the demo data filled it in. Plus rules that stop two agents writing over each other's files.

ISSUED TO

▸You build with an AI agent, it tells you QA passed, and a user finds a bug an hour later
▸You're about to send an iPhone app to Apple and want the store-rules check done first
▸You want a web app gone over properly: every page, every add, edit and delete, every blank screen

Also issued in:

The App Store Kit — $29 The iOS Factory — $69