Local AI for Marketing Engineers: When Customer Data Should Never Leave the Building

Most of your data can go to the cloud. Some of it cannot.

Every marketing team now sends data to an AI model somewhere. The question worth asking is which rows.

Greg Isenberg made the case in his episode on the marketing engineer role: local AI matters when the data is sensitive, like private customer transcripts, regulated notes or pricing plans, and when a job would cost too much in a cloud tool. That is the right list, and it is shorter than the local-AI crowd makes it sound.

A marketing engineer does not pick a side in cloud versus local. They sort the data, send most of it to the best model available, and keep a small, specific part at home.

We call that sorting the Building Test. Three questions, asked about each data set, before anything leaves the building.

The three cases where local wins

Case one: the data is regulated. For a clinic, the federal line is drawn by HHS.

Its guidance on online tracking says vendors that receive protected health information for a provider are business associates and need a signed agreement. The legal definition of a business associate is broader than trackers: any outside company that creates, receives, maintains or transmits that information for the provider.

The cloud is not banned for health data, but it is narrow. Anthropic says it signs a business associate agreement for its API and Enterprise plans, and its own table marks features such as the Batch API and third-party connectors as not covered. You have to check the exact feature, not the brand.

The other route is to remove what makes the data regulated. HIPAA's safe harbor rule lists 18 kinds of identifiers to strip, from names and phone numbers to IP addresses and full-face photos. A local model that does that first pass is often the cleanest way to use a cloud model at all.

Case two: the data is private, even if no law says so. Sales calls, churn interviews, a pricing plan for next quarter. In 2023 Samsung found staff had uploaded sensitive code to ChatGPT and banned the tools on company devices.

That story is about people pasting into consumer apps, which is a governance problem as much as a tooling one; shadow AI guardrails covers that side. Local AI answers a narrower question: what if the data should never be on anyone else's computer, under any agreement.

Case three: the job runs all day. This is the weakest case for most companies, and it is worth saying so. Anthropic lists its smallest current model at $0.10 per million input tokens for shorter prompts, with a 50% discount for batch jobs.

At that price, cost alone rarely justifies a server. The cost case is real for constant, heavy work, such as transcribing every call or embedding every support ticket, every day.

What runs locally today

A laptop now runs useful models, and the main tools are open source. The work is choosing the job, not fighting the install.

Ollama is the simplest starting point. Its FAQ is plain about the privacy line: Ollama runs locally and does not see your prompts when you use local models.

The models are good enough for the jobs marketing data needs. OpenAI says its open-weight gpt-oss-20b runs with 16 GB of memory and delivers results similar to o3-mini on common benchmarks, under an Apache 2.0 license; the larger gpt-oss-120b needs a single 80 GB GPU.

Three kinds of work fit local models especially well.

Extraction, where Ollama's structured outputs force the answer into a JSON schema you define. Search over your own files, where embeddings turn text into vectors you can store and compare. And transcription, where OpenAI's Whisper runs on a single machine, as the chart below shows.

Under the friendlier tools sit engines like llama.cpp, which runs models in C and C++ on ordinary hardware, and model files in the GGUF format built for fast loading. You rarely touch them directly. It helps to know they are there when something breaks.

Your scripts talk to a local model the way they talk to a cloud one. Ollama's API answers on your own machine at port 11434, so moving a job from cloud to local is often a change of address, not a rewrite.

One default is worth knowing. Ollama keeps a model in memory for 5 minutes after the last request, then unloads it, so the first call after a pause is slower; the FAQ shows how to change that.

What does not run well locally

The honest gap is judgment over long, multi-step work. Sebastian Raschka tested open-weight models inside coding agents on a Mac Mini and wrote that the top hosted models are currently better than smaller open-weight models.

He also found the newer 30B-class models capable enough for a lot of tasks, using up to 30 GB of memory at a 50,000-token context and writing at about 40 tokens a second. One model he tested scored 3 out of 5 on his agent tests; he called that usable but not fully reliable for autonomous tool use.

Read that as a job description. Local models are good clerks: extract, tag, redact, summarize. They are weaker strategists.

Local is also not automatically private. Raschka noted that one coding harness could send usage telemetry to its maker even while the model ran locally, unless you turned it off.

Ollama itself offers cloud models next to local ones, and its FAQ says those prompts are processed on its servers. The same page shows the switch, OLLAMA_NO_CLOUD=1, and notes the server listens only on 127.0.0.1 by default; change that and anyone on your network can reach it.

The split: a local clerk, a cloud strategist

The pattern that works is not either-or. A local model reads the raw file, removes what should not travel, and fills in a fixed set of fields.

Only that clean output goes to a cloud model for the hard thinking. Raw audio and raw transcripts never leave.

Take a sales call. Locally, Whisper transcribes it and a local model extracts the objection, the competitor named, the price mentioned, and a summary with the names removed. In the cloud, the best model you have reads 200 of those summaries and writes next week's positioning memo.

This is how a customer truth system can run on sensitive calls without the calls going anywhere. When an assistant needs to query the structured results later, give it read-only access, the approach in MCP for marketing data.

One rule keeps the split honest. The schema is the contract: if a field is not in it, it does not leave the machine.

Three local jobs worth building first

Most teams should start with one job, not a platform. These three pay back fastest, and each one has a clear finish line.

Call and interview extraction. Whisper turns the recording into text, and a local model fills the schema: objection, competitor, price mentioned, next step, a summary with names removed. The finish line is a weekly file of clean rows that sales and marketing both read.

Ticket and review tagging. Every support ticket and every complaint gets a topic, a sentiment and a product area, and its embedding goes into a local index. Then anyone can ask which complaints look like this one, without a vendor ever holding the tickets.

Redaction before the cloud. For a clinic or a law firm, the local model's only job is to remove the identifiers before a stronger model sees the text. That turns a no into a yes for a whole class of analysis.

Pick a bar before you start and write it down. How many of 20 hand-checked files must be right, field by field, before the job runs on its own?

Then keep checking a sample every week, because a model that was right in testing can drift when the inputs change. The method is the same one you would use for any agent.

"Is the cloud not private enough already?"

Often, yes. Anthropic says it deletes API inputs and outputs within 30 days by default, keeps flagged content longer for policy enforcement, and offers zero data retention by agreement.

It also says it does not train on commercial inputs and outputs by default. For most marketing work, that is enough, and running your own hardware adds a machine to patch and a model that is weaker than the one you replaced.

So the case for local is narrower than its fans claim. It wins when no agreement can cover the data, when a client contract forbids sending it out, or when the volume never stops.

It also wins on a quieter point: you can switch providers without moving your most sensitive data again. That portability argument is in staying model-agnostic, and the decisions about who may send what where belong in an AI governance framework written on one page.

Setup: a private box in one afternoon

This is a half-day build, not an infrastructure project. Start with one job, usually call or interview extraction, and one machine.

The checklist below is the order. Write the schema before you pull a model, and test with files you invented before any real customer data touches it.

Then decide where the clean output lives and what reads it. That is the marketing engineering stack question: storage, schedule, logs, and who gets an alert when a run fails.

If your data is regulated, the setup starts with a review, not a download; our healthcare and med spa work begins there. If you want the box and the split pipeline built for you, it is part of our AI automation work and our marketing engineering engagements, and you keep the machine and the code.

What to read next, and why

Server-Side Conversion Tracking [system]. The other place customer data leaves the building: the events you send to ad platforms. Same instinct, different pipe.

AI Cost Management: The Token P&L Method [method]. Run this before you buy hardware to save money. Most teams find the cloud bill is smaller than they feared.

How to Evaluate AI Agents [method]. A local model needs the same tests as a cloud one, and probably more. Do not trust a field you never checked.

What Is an Agent Harness [explainer]. The model is the small part. The schema, the checks and the logs around it decide whether the private box works.

So the simple takeaway is this: send your data to the best model you can, except the rows you could not explain to the customer they came from.