Very Good Plugins · working notes
How WP Fusion turns a customer feature request into a reviewed, tested, documented pull request without a developer in the loop — and, just as important, exactly where it stops and hands the work back to a human.
First, the correction
I gave a fairly breathless account of this at the last mastermind, and the version that reached you is more finished than the thing I actually run. Before your Head of Development starts building toward it, here is the honest ledger — because the gap is the useful part, not the embarrassing part.
| What I said it does | Reality | Who does it |
|---|---|---|
| Looks at the feature request list | Automated | A triage agent queries the request board, scores candidates and returns a ranked shortlist. |
| Chooses a feature | Human gate | I pick from the shortlist. The agent is explicitly forbidden from selecting its own work. |
| Analyses the customer requests | Automated | The scoper reads the full request thread and turns it into a bounded build spec — or refuses. |
| Builds and tests it | Automated | Implementer writes the code; static analysis, unit tests and a live-site runtime harness all run unattended. |
| Does the release | Not built | Nothing merges itself. The run ends at an open, reviewed PR. I merge, tag and ship. |
| Updates the documentation | Draft only | A docs agent writes the page and takes its own screenshots, but is hard-blocked from publishing. |
| Emails the waiting customers | Not built | Still me, at release time. This is the next thing I want, not a thing I have. |
The other correction: I described this as running weekly. The agents are written for a weekly tick — the triager's own documentation has a “scheduled weekly feature-request review” entry point — but the cron is not currently armed. Today I kick each run off by hand. Arming it is a scheduling problem, not an architecture problem, and I have deliberately not rushed it: an unattended loop is only as safe as its worst gate, and I have been busy building the gates.
The shape of it
A train cannot enter a block of track until the previous block is proven clear. Not assumed clear — proven, by a physical interlock that refuses to throw the signal otherwise. That is the whole design. Each stage of the pipeline is a block, each block has an interlock, and no stage can proceed on a previous stage's optimism.
This matters because the failure mode of an AI coding agent is not that it writes bad code. It is that it writes plausible code and then tells you, sincerely and at length, that it works. Your Head of Development put it perfectly back in June — that he never really trusts the first answer. Every interlock below is an answer to that.
Note: this is a faithful diagram of the real stage order and the real halt conditions, animated for readability. It is not a live connection to anything.
The interlocks
That claim sounds like marketing, so here is the argument in full. I am a competent developer with fifteen years in this codebase, and I am also a human being who at 11pm will edit a shared base class “just this once”, skip the manual test because the unit test passed, and paste a log into a GitHub issue without reading it closely. Each interlock below removes one of those from the table permanently. Every single one exists because it already went wrong once.
Born from: run 1, where a spec proposed a change I would never have approved.
The scoper's only output is a build spec, and the spec is invalid unless the work is purely additive. “Additive” is defined by files, not by line count — a new integration directory, new JS components, generated assets are all fine; touching a shared base class is not. If the feature genuinely cannot be built additively, the agent is required to stop rather than write a smaller-sounding spec.
# .claude/agents/wpf-feature-scoper.md blast_radius MUST be `additive`. Additive means NO edits to includes/crms/class-base.php, includes/integrations/class-base.php, any shared base class, or the singleton bootstrap beyond a single registration line — it is NOT a file count. […] If the only viable implementation breaks this, output `ESCALATE: <reason>` and STOP. Do not emit a spec. Core Feature Enhancements (e.g. "Expiring tags") usually escalate — that is correct, not a failure.
That last line is doing real work. Without it, an agent measured on completion rate learns to squeeze core changes into an additive-shaped spec.
Born from: run 1, three separate confidently-wrong “this hook never fires” claims.
The calibration run for this pipeline was to point the brand-new scoper at an
integration I had already built by hand, blind, and diff its spec against my real
implementation. It got five things wrong. Two were ordinary misreadings. The other three
were all the same species: the agent asserted something did not exist, having looked
somewhat but not systematically. So now every negative claim has to carry its receipt
— the search pattern, the scope searched, and the number of hits. “I found no
renewal hook” is inadmissible. “grep -rn 'renewal' includes/ → 0 hits”
is admissible. That one rule removed the entire class.
Born from: run 1, a hand-fired hook that passed while the real code path was broken.
This is the interlock I would most encourage your team to copy, because it is where
almost all fake green comes from. It is trivially easy for an agent to “test” an
integration by calling do_action() with hand-rolled arguments. That proves the
listener runs. It proves nothing about whether the host plugin ever calls it, with those
arguments, in that order.
# .claude/agents/wpf-integration-verifier.md Drive the host plugin's real machinery — its own status setters, processors, renewal/payment paths. NEVER fire the host's hooks or WP Fusion's integration hooks manually: hand-rolled do_action() calls fake the argument shapes and ordering where real bugs live. Every test-plan item ends OK-with-evidence or BLOCKED-with-reason. Evidence is captured output: a tags dump, log rows, a screenshot path, a response body. No partial pass — "3 of 4" is not a pass. Ambiguous output is a failure with the raw capture attached; don't interpret it away.
“Don't interpret it away” is the most valuable four words in our entire agent corpus. Ambiguity resolves to failure, never to success.
Born from: run 3, a suite that exited 0 having executed nothing.
PHPUnit will happily exit successfully when a configuration change means it collected zero tests. An agent reading only the exit code reports a pass, truthfully and uselessly. So we stopped reading the exit code alone.
# tests/run-phpunit.sh
if ! grep -Eq 'OK \([1-9][0-9]* tests?|Tests: [1-9][0-9]*' "$OUTPUT_FILE"; then
echo "PHPUnit exited successfully but did not execute any tests." >&2
exit 1
fi
Four lines. Closes an entire category of false confidence.
Born from: run 1, where composer test turned out to be destructive.
Our PHPUnit config points at the live local development database — there is no
separate test database. We discovered this the hard way when a tearDown()
blanked a setting and silently disconnected the dev site's CRM connection. The fix was two
rules (capture in setup, restore in teardown, never blank) and one interlock: the runtime
harness boots WordPress and then checks where it landed before doing anything.
// tests/runtime/wp-run.php
$_SERVER['HTTP_HOST'] = 'dev.local';
require dirname( __DIR__, 5 ) . '/wp-load.php';
if ( 'https://dev.local' !== untrailingslashit( home_url() ) ) {
fwrite( STDERR, "wp-run.php: refusing to run outside https://dev.local.\n" );
exit( 1 );
}
I cannot type a production hostname into this by mistake at midnight. It simply exits.
Born from: the same incident — state left behind between runs.
Verification necessarily changes things: it flips settings, creates users, writes tags. Every option the harness touches is backed up first and restored at the end of the round. More importantly, a new round refuses to begin if a previous round left backups unrestored, rather than starting on top of unknown state.
// tests/runtime/lib.php function wpf_rt_begin_round( $round_id = '' ) { if ( wpf_rt_has_pending_option_backups() ) { return new WP_Error( 'wpf_rt_pending_option_backups', 'Runtime verification has pending option backups. Run tests/runtime/run.sh --restore-options before starting a new round.' ); } // ... fresh round state }
Fails closed. The interesting design decision is that it refuses rather than auto-cleaning — auto-cleaning would hide the fact that the last round died badly.
Born from: run 1, a bearer token that appeared in a log dump.
The verifier's evidence includes raw rows from our logging table, which contain API
traffic. During the first run an Authorization header made it into a report.
Nothing bad came of it, but “the agent was careful” is not a security control, so
redaction became a function that every log dump passes through.
// tests/runtime/lib.php function wpf_rt_redact_sensitive_log_data( $message ) { $message = preg_replace( '/\b(Authorization)\b(\s*[:=>]\s*)[^\r\n<]+/i', '$1$2[REDACTED]', $message ); $message = preg_replace( '/\b(Basic|Bearer|OAuth|Token)\s+[A-Za-z0-9+\/=._~:-]+/i', '$1 [REDACTED]', $message ); // ... further passes strip cookies, access_token, api_key, // client_secret, password and friends from query strings and JSON return $message; }
This is the concrete answer to “is it more secure than you?”. I redact when I remember to. The function redacts every time.
Standing policy, and the part I would least want to give up.
Once the branch is pushed, control passes to a review loop that requires an OpenAI Codex review on the current head of the PR. Claude wrote the code; Codex looks for what is wrong with it. Findings must be fixed or answered on the thread with a concrete reason — and crucially, a fixed thread is not sufficient on its own: Codex must have reviewed the current head, CI must pass, and the PR must be non-draft and conflict-free before the gate opens.
Two models from two vendors with different training data disagree in useful places. A model reviewing its own output mostly agrees with itself. This is also, in practice, where the genuine bugs get caught — run 3's most instructive defect was found here.
Design constraint from day one.
The orchestrator ends with a request for review from a named human, and then a single instruction with no exceptions clause:
# .claude/skills/wpf-feature-pipeline/SKILL.md
Never merge. Never mark a request `done`; those are operator-only actions.
The docs agent has the same shape — it can create drafts and propose diffs against
live pages, but status: publish is forbidden outright. Both gates cost me a few
minutes per run and have repeatedly been worth it.
How it got here
This did not arrive as a design. It arrived as three runs over six weeks, each of which added exactly one thing, and each of which added it because the previous run had already failed without it. If your Head of Development takes one process idea from this document, I would suggest that one: do not build the gate until you have been burned by its absence. You will build the wrong gate otherwise.
I had just built a subscriptions integration by hand. Rather than trust the new agents on live work, I pointed the scoper at that same feature blind and diffed its spec against what I had actually written. It got five things wrong — an impossible registration guard, a missed renewal-tag suppression, and three unverified claims that something did not exist. Each error became a rule. Nothing shipped from this run; that was the point.
The first genuine end-to-end run: a community-badges integration, request to PR, with zero writes to a live CRM. It worked — and then immediately showed me that it only worked on my machine in my state. So the hardening pass taught it to boot the local development environment itself, to stop assuming one particular CRM was connected, and to take screenshots as a matter of course.
An affiliate-platform integration, run as a deliberately narrow canary. The code
defects it produced were ordinary and got caught. The interesting failure was different in
kind: the docs agent reported the documentation as ready while the page
still contained the literal text SCREENSHOT NEEDED. Nothing lied. The agent
simply had no mechanism that could distinguish “I delivered the image” from
“I intended to”. That became a fail-closed delivery gate with a machine-readable
result and an independent re-fetch to confirm.
For the developer
This is the part that took the longest and gets discussed the least. The agents are prompts; anyone can write prompts. What makes them useful is that they are wired to a real WordPress site, a real database, a real browser session and a real queue — and each of those connections was its own small engineering problem.
The queue lives in post meta on our own site, behind a small REST API. Claiming the next
item is a rank-ordered SELECT … FOR UPDATE inside a transaction, so two
runners can never claim the same request. The state machine is a deliberately narrow
whitelist — in_progress can only become blocked or
pr_opened, and only pr_opened can become done. There
is no auto-requeue from blocked. A stuck item stays stuck and visible.
One MCP server talks to the production site — the request board, the docs. A second, entirely separate one talks to the local development database. Run 2 taught us why: a scoper with production credentials in scope will reach for them when it wants a schema. Splitting the servers makes the wrong call impossible rather than discouraged.
Our dev sites run under Local by Flywheel, which starts a MySQL instance per site on a
unix socket in a per-site directory with a generated name. For an agent to query that
database it has to find the socket. We wrote a small MCP server that scans the process table
for a mysqld whose config file sits under Local's run directory, and falls back
to globbing the run directory for the newest live socket. Unglamorous, and it removed an
entire class of “the agent can't see the database” failures.
Run 2 died because Local wasn't running and the agent had no way to start it. Preflight
now launches the desktop app, waits on its internal API using curl --retry
rather than a blind sleep, starts the site, then waits for the site to actually
serve a page — a served page implies MySQL is up. It then probes whichever CRM is
configured and reports health as evidence rather than as a blocker.
Documentation needs screenshots of logged-in admin screens. The usual answer — drive a headless browser and script the login — means putting real credentials somewhere an agent can reach, and modern Chrome will not let you attach a debugger to your normal profile anyway. Our tool takes the other route: a small extension inside the browser I am already signed into, relaying commands over localhost. No credentials in the agent's world, no separate login, and the screenshots look like a real site because they are one.
Every agent reads the repository's own CLAUDE.md files — root plus one
per subsystem. Version lives only in readme.txt; new code is tagged
@since x.x.x and rewritten at release; the text domain is fixed; never log
credentials. Conventions I would otherwise repeat in every code review are stated once and
apply to every agent that touches the repo.
Corrections go into a persistent memory service rather than into a prompt. When run 3 hit a variant of run 1's “fix the instance, miss the class” mistake, that pattern was already recorded — which is how I know it is a recurring species and not bad luck. This is also the honest limit of the approach: memory helps me diagnose repeats, it does not yet prevent them.
Judgement stages — scoping, verification, orchestration — run on the most capable model available. Mechanical stages — triage scoring, documentation drafting — run on a faster, cheaper one. The model is declared in each agent's front matter, so re-pointing the whole pipeline at a newer model is a one-line edit per file, not a migration.
One more piece of structure worth stealing: every agent file carries machine-readable front matter naming its pipeline, its stage, and its human gate.
# .claude/agents/wpf-docs-writer.md model: sonnet metadata: pipeline: "feature-request-to-pr" stage: "3-document" human_gate: "draft review — operator publishes at release time"
Which means the pipeline can be enumerated, ordered and checked by a test. We have contract self-tests that read these files and fail the build if a stage loses its gate — the guardrails are themselves under test.
Worked example
Rather than describe this in the abstract, we ran it. I generated the scoper definition using the configurator further down this page — not a hand-tuned one — pointed it at two public Barn2 repositories, and gave it four feature requests written the way a customer would write them.
The instruction to each run was explicit: reach whatever verdict is true; escalating is a correct outcome, not a failure. I did not tell any of them what I expected. I had guessed the file-size column would sail through.
Every file:line reference below was
re-checked by hand against the source before publishing. Roughly thirty citations across four
runs; all of them held.
The request that looked easiest. A column showing how big each download is — obviously additive, surely. The scoper disagreed, and the reason is subtle enough that I had to read the source twice to confirm it.
There is a public filter for table columns,
document_library_table_column_defaults, applied at
Simple_Document_Library.php:368. It looks exactly like the extension point
you would want. It is a trap. The filter decorates the column metadata array,
but both gates that decide whether a column actually survives call the
unfiltered Options::get_allowed_columns():
// Simple_Document_Library.php:374 — the header gate if ( ! in_array( $column, Options::get_allowed_columns(), true ) ) { continue; } // Simple_Document_Library.php:397 — the column-list gate $columns = array_intersect( $columns, Options::get_allowed_columns() ); // Util/Options.php:218 — and that reads the raw defaults, unfiltered public static function get_allowed_columns() { $allowed_columns = array_keys( self::get_column_defaults() ); return $allowed_columns; }
So a developer can hook the documented filter, supply a heading, a width and a
priority for file_size, and render nothing at all. The scoper worked that
out from a cold read and refused to write a spec that depended on it.
| Claim | Command | Hits |
|---|---|---|
| The defaults registry is unfilterable | grep -c 'apply_filters' src/Util/Options.php | 0 |
| No filter inside the cell emitter | sed -n '405,453p' … | grep -c 'apply_filters' | 0 |
| Only two consumers of the allow-list, both unfiltered | grep -rn "get_allowed_columns" src/ templates/ assets/ | 3 |
| No file-size support exists today | grep -rniE "filesize|file_size|size_format" src/ | 0 |
It also checked whether the JavaScript layer offered a way around, and closed that
off too: the DataTables column list is built from the same already-filtered array, and
injecting a column client-side without a matching <th> produces a
column-count mismatch.
Three options for a human, not a refusal to engage: accept a small two-file core edit; or add the missing extension point first as its own change, which would make this feature and every future column genuinely additive; or a reduced-scope version appending size text to an existing cell — which it labelled “NOT the request” rather than quietly delivering it and calling the ticket done.
This is the run that changed my mind about what these things are for. It opened by arguing against its own verdict:
It had found a clean seam — intercepting the shortcode before attribute parsing, reusing the existing query methods, rendering cards from a new class, zero edits to any render file. It could have shipped it. Then it escalated anyway, because it noticed grid layout is deliberately gated in three independent places:
Util/Options.php:31 — 'layout' is the first entry in $readonly_settingsAdmin/Settings_Tab/Display.php:104 — the grid section renders “Pro version only” copyAdmin/Wizard/Steps/Layout.php:65 — the grid radio option is flagged 'premium' => trueAnd concluded:
That is not a code judgement. That is an agent declining to build away the upsell that funds the product, and routing the question to a human as a pricing decision. Nobody designed that behaviour in. It falls out of one rule — escalate when it is not yours to decide — pointed at a codebase that was honest about its own tiering.
Same shape, sharper. The plugin already ships a metabox for exactly this feature
— as a paywall. Admin/Metabox/Document_Expiry.php:63 contains a near
verbatim restatement of the customer's request, followed by a UTM-tagged upgrade button
and a dedicated promotional graphic:
// Admin/Metabox/Document_Expiry.php:63
'Upgrade to the Document Library Pro to set expiry dates, in which
documents will automatically be removed from the library on the
date you choose.'
The scoper found it, cited it, and escalated on two independent grounds — the commercial one, and a genuine blast-radius one (the only service-registration surface is a forbidden file). It also caught something I had got wrong.
My prompt told it the plugin exposed three relevant filters, and asked it to verify rather than take my word. It came back:
// Simple_Document_Library.php:292 and :296 do_action( 'document_library_before_posts_query', $this ); do_action( 'document_library_after_posts_query', $this ); // only this one is real: // Simple_Document_Library.php:309 return apply_filters( 'document_library_table_query_args', $query_args, $this );
Two of the three were actions, not filters — return values discarded, useless for altering a query. I had asserted them casually from a grep of hook names. The citation rule caught the orchestrator's error, which is the part of this I did not expect and now value most.
Its closing suggestion was to tell the customer their actual need is solved today without touching the plugin at all — a small site-specific snippet on the one real filter, or the tier that already ships it.
A different repo — a single-tier free plugin with a subclass-shaped architecture — and the pipeline behaved completely differently. It studied the closest existing feature, found the real seam, and produced a spec:
blast_radius: additive
New file: classes/Integrations/NoIndex.php (extends Service)
Registration: one line after classes/Plugin.php:26
Primary seam: classes/Collections/Posts.php:48
apply_filters( 'llms_txt_generate_get_posts_args', $query_args )
It confirmed each hook was genuinely apply_filters and flagged that
llms_txt_settings_ready is an action, usable only for registering the
settings checkbox. It argued for filtering at query level rather than per item, because
excluded posts must never enter the result set or the link quota silently shrinks.
It could not reach a database this session, so it could not confirm which meta key the site's SEO plugin actually uses. Rather than guessing a plausible one — which would have produced a spec that looked complete and failed on contact with reality — it marked the key unverified and made confirming it acceptance item one:
It also declared a limitation it could not fix within its own mandate: pinned items are built by a separate query that would bypass the exclusion, and closing that gap means editing a collection file. So it stated the leak, recommended accepting it, and explained what it would cost to close.
Five acceptance items, each written so it can only pass with captured evidence — including one that sets a link quota of five with three excluded posts in the top eight, and asserts exactly five links survive. That test distinguishes query-level exclusion from line-level filtering. It is the difference between a feature that works and one that looks like it works.
Three refusals out of four looks like a poor showing until you read your own release notes.
From readme.txt:180 in the Lite repository:
Every one of the three requests I invented is named, verbatim, as a paid-tier feature. I did not know that when I chose them — I picked what sounded like ordinary customer asks. The pipeline reconstructed the pricing tiers from the code, independently, three times, and the first refusal came on pure architecture without ever noticing money was involved.
The practical lesson for pointing this at your own catalogue: aim it at the tier where the features are meant to be built. Run it against a free edition and it will correctly refuse most of what you ask, because a free edition is a funnel and its extension points end where the paid version begins. That is the pipeline working, but it is not throughput.
Porting it
Almost none of the above is WP Fusion-specific. The stage order, the gates and the front-matter contract are generic; what changes per repo is where the code lives, how the tests run, what “additive” means in that codebase, and where the docs go. Fill in those four things and this generates a starting scaffold — the agent definitions and the orchestration checklist — adapted to the plugin you name.
It generates a starting point, not a finished pipeline. The honest sequence is: scaffold, then do a calibration run against a feature you already built by hand, then let the errors it makes tell you which gates your codebase needs. Ours took three runs.
Caveats
The economics are good but not magic. A run costs me real money in tokens and roughly twenty minutes of my attention — picking the feature, reading the PR, merging, releasing. Against a small plugin's feature backlog that is transformative. Against a plugin with no users, it is still not worth doing.
It is confidently good at the shape of work it was built for — a new, self-contained, additive integration in a codebase with a strong existing pattern to copy. It escalates on anything touching shared code, which is correct and also means it cannot do a meaningful fraction of my actual backlog. Your smaller plugins may skew more toward core changes than ours do; that is worth checking before investing a fortnight.
And the gates are the product. If your Head of Development builds the five agents and skips the interlocks, he will get a system that produces a confident PR every week and a slow accumulation of subtle bugs, which is strictly worse than nothing. The value is not that the agent writes the code. It is that when the agent cannot prove the code works, the pipeline stops and says so.