Reducing Cost After Launching Your Agent

Getting an agent into production is its own project. Then it ships, it holds up, and everyone gets on with their lives.

That’s usually when the bill starts creeping up in the background. Every conversation costs what it costs, and once the thing is working, nobody’s staring at it anymore.

Cost work is a different kind of work than building, and there’s an order to it. I did some of it out of order and paid for that.

First, get a baseline

You can’t improve what you haven’t measured. You want two baselines: what it costs and how it behaves.

Cost is the easy one. Pull the numbers: per conversation, per run, per whatever unit the business actually cares about.

Behavior is trickier, because the system isn’t deterministic. Same input, different output. So “did this break anything?” isn’t something you can answer by eyeballing a single run and squinting at it. You need a test harness. Write down the user journeys people really take, then baseline those.

And you want the whole team on the same harness. Three of us were working on this, and we each made our own. That sounds reasonable right up until you compare results and realize my number and your number are measuring different things. One change looked like a win in one harness and basically nothing in another.

So: one harness, agreed on before anybody starts optimizing, and fast enough that people will actually use it. Because they’re going to run it constantly. Every cost change is a behavior change until it proves otherwise.

Write down your assumptions

People like to say first principles. What I mean is less glamorous: write down the box you’re working inside.

• What models can we use?
• What models can’t we use?
• Can we run open-weight models on our own infrastructure?
• Are there APIs we can’t send data to for reasons that have nothing to do with engineering?

Those are constraints. Once they’re written down, you stop burning a week on an option that was never available in the first place.

They also change. Vendors reprice things. Model lineups shift. Your own system picks up new habits. You can only revisit assumptions you bothered to record.

Lever one: the cheapest model that can still do the job

This was the biggest lever I found, and it’s not just flipping a dropdown.

Smaller models infer less. They need the task spelled out more clearly. A prompt that works fine on an expensive model will usually underperform on a cheaper one, so most of the work in “switch to the cheaper model” is really prompt work.

Treat it like a config swap and you’ll walk away thinking the cheap model can’t do the job. All you’ve really proven is that your prompt can’t.

Lever two: make the prompt smaller

Fewer words in means less money out. On every call. Forever.

Look at what you’re actually sending. There’s usually more in there than you think, and some of it is hanging around because it was there last month and nobody questioned it.

Lever three: organize the prompt for caching

Vendors will discount content they’ve already seen, but only if it’s reusable: same content, same position, every time. If something near the front changes for each customer, everything after it gets billed at full price.

So this turns into a structure problem. Stable parts first. Variable parts last.

What made that click for me was reading the assembled request top to bottom. Not the code that builds it. The actual thing going over the wire. We were sending the same tool descriptions again and again for different instances of the same tool, and nobody had read the whole document as one piece.

Where I lost time

I lost time trying different models mostly because they existed. Reasoning modes. Configurations I wanted to be clever about. That’s the part that feels most like engineering, and it returned the least.

The savings came from reading my own prompt and noticing I’d sent the same paragraph four times.

Why I bother

An agent that works is a fixed cost you already agreed to without much negotiation. This is the work that decides whether it’s still affordable when volume goes up 10x.

Baseline, then the box, then the levers in order.