How to Evaluate a Marketing Engineer's Work: Qualified Replies, Not Messages Sent

Four thousand emails is not a result

The dashboard says 4,000 emails went out this month. It looks like work, and it is.

It is also the easiest number in marketing to grow, and one of the least useful. Point an agent at a list and it will send as many as you let it.

Greg Isenberg drew the line in his episode on the role: messages sent is activity, and qualified replies are the signal. A marketing engineer is paid to build systems that create pipeline, so the card you grade them on has to count pipeline, not motion.

That sounds obvious. Most reports still lead with the activity number, because it is the one that always goes up.

Activity numbers still have a job. They explain a bad result after the fact; they cannot tell you whether the result was good.

So start with the split.

Activity versus signal: one question sorts every number

Every system a marketing engineer builds throws off two kinds of numbers. Activity says the system did something: posts published, emails sent, ads launched, pages rewritten. Signal says a buyer did something because of it: replied, booked, bought, or cited you.

One question sorts them. Could this number go up while revenue goes down? If yes, it is activity.

Eric Ries made the same cut for startups, separating actionable metrics from vanity metrics, and the summary of lean startup on Wikipedia keeps the definition short. An actionable metric changes what you do next week. A vanity metric changes how the meeting feels.

The figure below puts the split next to each of the six systems. The right-hand number is the one that goes on the card.

Then define "qualified" before anyone counts it. The Harvard Business Review study of online sales leads used a strict version: a meaningful conversation with a key decision maker.

Write yours in one sentence, and test it the way Anthropic's guide to agent evals tests a task. Two people who know your market should read the same reply and reach the same verdict. If they do not, the definition is not finished.

Six signal numbers, and where each one comes from

One number per system. Pick it once, write down where it comes from, and do not change it in the middle of the quarter.

Outbound signal engine: qualified replies per week, read from the inbox and tagged in the CRM by the definition above. The build is in the outbound signal engine; messages sent stays on the card only as a clue.

Creative testing engine: cost per booked call or per purchase, not cost per click. The booking or the sale has to flow back to the ad platform, which Google supports through offline conversion imports, so the platform learns from revenue instead of form fills.

On Meta, read the Results column for the event each ad set was built to produce, the habit covered in read Results, not Leads. The system itself is in the creative testing engine.

Customer truth: the share of new ads, pages and emails that came from a finding in the weekly memo, and how they did against the rest. A memo nobody acts on scores zero, however good it reads.

AI search visibility: the share of a fixed set of buyer questions where ChatGPT, Gemini or Perplexity cite your pages, rerun on the same questions each week. The set stays fixed so the trend means something; the method is in the AI search visibility system.

Founder content engine: inbound conversations where the buyer names a post, an episode or an email. Growth cockpit: meeting hours it replaced, and whether a decision was made from the memo that week.

For anything on a website, the step you care about has to exist as a tracked event first. Google Analytics calls the important ones key events, and a funnel exploration shows where people drop between them.

One more number belongs on any system that touches leads, because it moves faster than anything else: time to first response. In the HBR audit of 2,241 US companies, 37% answered a web lead within an hour and 23% never answered at all.

A second study by the same authors, of 1.25 million leads, found that firms trying to reach a lead within an hour were nearly seven times as likely to qualify it as firms that waited one more hour. The build for that is in speed to lead.

The health line: a system that does not run scores zero

A great signal number from a system that ran twice is luck. So every system gets a health line under its signal.

Assured's posting for an AI Marketing Engineer makes the hire accountable for whether the system is on, and asks for weekly dashboards the CEO can read in under five minutes. That is the health line, and the length limit, in one posting.

Software teams already measure this. Google's DORA metrics guide tracks how often a team ships, how often a change fails, and how long recovery takes, and it says the metrics work best for one application or service at a time.

For a marketing system, three health numbers are enough. Runs on schedule: how many of the planned runs happened. Time to fix: when it broke, how many days until it ran again.

The third is approval rate: the share of outputs a person approved without rewriting them. Owners skip it, and it tells you the most.

Say the approver rewrites 60% of drafts in week two and 15% in week eight. The system is learning. If the rate stays flat, the corrections are not being written down as rules.

Profound's sample job description asks the engineer for evals that catch drift, which is the same idea in engineering words. The long version, including why an agent that is right 75% of the time passes three runs in a row only about 42% of the time, is in how to evaluate AI agents.

Hours returned, counted so a finance person would accept them

Hours saved is the number most likely to be inflated. It still belongs on the card: Figma's Marketing Engineer, AI Deployment posting measures success by agents live, hours saved and revenue driven.

Clay says its GTM engineers are measured the same way, by meetings booked and hours saved. So the number stays; it just needs rules.

Count only work someone was already doing, timed before the build. Take the minutes per task and how often it ran over two weeks, then time the same task after the system is live.

Hours for work nobody did before are not saved. They are new work, and they go under signal if they go anywhere.

Google's site reliability engineers have a name for the work being removed. Their book defines toil as work that is manual, repetitive and automatable, and that grows as the service grows, and the team aims to keep it below half of each engineer's time.

Turn that rule on the marketing engineer's own week. If more than half of it goes to keeping old systems alive, the role has slid into ops, and the next system will not get built.

Put the running cost on the same line as the hours. A system that returns 10 hours a week for $900 a month is a different decision from one that returns 10 hours for $40.

The weekly card: five minutes, one screen, one system

Everything above fits on one card per system, read every Monday. If it takes longer than five minutes to read, it has too much on it.

The card has seven lines: signal, health, hours, cost, clues, a decision, and the next test. The figure below is a filled-in example you can copy line for line.

Two reads matter each week. First, whether the signal moved against its baseline. Second, whether last week's change moved the number it was meant to move.

If the signal fell, open the clues line. That is the only time activity numbers earn a look, because a drop in replies with steady sends points at the message, and a drop in sends points at the system.

If you sell on calls, put the four rates from book-a-call math under the card. The marketing engineer's signal numbers should show up in those rates within a quarter, or the signal was further from revenue than it looked.

Goodhart's law is the warning label for all of this. As usually stated, when a measure becomes a target, it stops being a good measure, so agree on the definitions before any pay rides on the card, and check a sample of qualified replies by hand each month.

What good looks like at day 30, 60 and 90

The card tells you different things as the quarter goes on. Reading it too early is how good systems get killed.

Day 30: the card exists, the signal is defined in one written sentence, and there is a baseline from real data. No result yet, and that is correct.

Day 60: the system runs on schedule, health is green most weeks, and the approval rate is climbing. The signal sits next to its baseline, but three or four weeks of data is not a verdict.

Day 90: the signal has moved against baseline or you know why it has not, the system ran a full week without its builder, and hours and cost sit on the same line. That is when you decide on system two.

If day 90 arrives and the signal has not moved, the card already tells you which kind of failure it is. Health red means the system did not run enough to judge, so fix it and give it four more weeks. Health green with a flat signal means it ran fine and buyers did not care, and that is a kill, not a fix.

Where you can, test instead of comparing with last quarter. Google Ads lets you run a custom experiment that splits traffic between the old setup and the new one, which answers "did this system cause it" better than a chart of two quarters.

We call the whole thing the Signal Card. One signal, one health line, honest hours, a cost, and a decision every week.

"You cannot pin pipeline on one person"

True, and the critics are right about more than that. Revenue has many parents: the sales team, the season, the offer, a competitor who raised prices.

The card does not assign revenue to a person. It asks a smaller question about one system: did its signal move against its own baseline, and did the next step in the funnel move with it.

Hours saved is soft, as we said above. Approval rate can be gamed by an approver who stops reading. That is why the card has several lines instead of one; it is harder to fool all of them in the same week.

Some value will never show up on a weekly card, like a sharper message the founder starts using on sales calls. Write it in the notes. Just do not grade on it.

How we measure our own systems

We grade our own systems on the same lines. Our citation checker either confirms every source link on an article loads or blocks it from publishing, so its health line is a yes or a no.

Our AI search tracking is a signal system. Two runs on September 22, 2026 read 409 AI answers to unbranded buyer questions and logged 4,243 citations; company pages were 65% of them and Reddit 1.0%, which changed what we write.

Our customer-language scan carries its cost line in the code: a cap of $0.50 a run, about $0.02 on a typical run. Every ad render is scored by scripts and then by a separate critic agent before a person sees it, so the builder never grades its own work.

If you want systems built with the card attached from day one, that is how our marketing engineering engagements run. The CRM side, where replies and booked calls get tagged so the card can count them, is our CRM and RevOps work.

What to read next, and why

The First 90 Days With a Marketing Engineer [plan]. The day-by-day calendar this card is read against. Set the baseline in month one or the day-90 read is guesswork.

The Growth Cockpit [system]. Where the Signal Cards for every system end up on one Monday memo. Build it second.

The Marketing Engineering Audit [checklist]. Twenty questions that find which number is worth a system at all. Run it before you hire.

Reading Ad Data [diagnosis]. When the signal moves the wrong way on paid traffic, this is the order to check the numbers in.

So the simple takeaway is this: count what buyers did, not what the system did, and read it against a baseline you set before the build.