Red flags when hiring an AI automation agency: the checklist

"We use AI" has become the easiest sentence to say and the hardest to verify. As every consultancy repositions as an "AI automation agency", the number of teams that can build a working system has not kept pace with the number that can sell one. For an SME owner, the risk isn't paying for automation. It's paying for a slide deck, a half-finished integration and a monthly retainer that never quite ends.

This is a buyer's guide, not a sales pitch. Below is the checklist of red flags to watch for when hiring an AI automation agency, how to vet one before you sign, and an honest look at when an agency is the right call and when a different model fits better.

The biggest red flags, one by one

The demo never touches your data

The single most reliable tell. A polished demo on the agency's own sandbox proves nothing about your CRM, your invoicing, your messy real-world data. If they won't run a small proof on your actual process —even a narrow one— before you commit, assume it doesn't work yet. Automation lives or dies on integration, and integration only shows up against real systems.

No numbers on the proposal

If the proposal talks about "efficiency" and "transformation" but never states how many hours a week the automation saves, what it costs to run, and in how many months it pays back, that's a red flag. A serious partner puts the business case in writing: hours saved × cost per hour versus implementation cost. If you want the maths, we broke it down in how much AI automation really costs. An agency that can't produce that math either hasn't done it or doesn't want you to.

"We use AI" with no named tools

Ask which models, which platforms, which integrations. A credible answer names them and explains how they connect to your stack. A vague one —"proprietary AI", "our secret sauce"— usually hides either a thin wrapper over someone else's API or nothing at all. You don't need to become an engineer, but you should be able to write down the tools on a napkin after the call.

No one owns the result after go-live

Who maintains the automation when a supplier changes a form, an API updates, a rule shifts? If the answer is "that's a new project", you've bought a demo, not a system. Ask, in writing, who owns the accounts, the code and the credentials, and what happens on day 91. Automations that nobody tends quietly rot.

No human in the loop where errors are expensive

Good automation knows what to hand back to a person. If an agency proposes letting AI act unsupervised on refunds, pricing, contracts or anything that touches money or a key client, that's not ambition —it's risk. The right design automates the repetitive volume and routes the sensitive exceptions to a human. Speed without control isn't efficiency; it's exposure.

Lock-in by design

If the system only runs on accounts you don't control, with data you can't export and logic you can't see, you don't own an asset —you own a dependency. A fair partner leaves you able to operate, or hand over, what they built. Deliberate opacity is a commercial strategy, not a technical necessity.

Selling the tool before understanding the problem

The tell here is a fixed product pitched before anyone has asked what process is costing you the most hours and errors. Technology is the last step, not the first. If the conversation starts with "here's our platform" instead of "show us where the time goes", the fit is being forced.

How to vet an agency before you sign

Turn the red flags into questions and make them answer in writing:

1. Run a proof on our data. A small, real slice of our process —not your sandbox.
2. Put the ROI in the proposal. Hours saved, monthly run cost, payback in months.
3. Name the stack. Which tools, and exactly how they connect to our CRM and invoicing.
4. Define ownership. Who holds the accounts, the code and the data —and can we run it without you?
5. Show the maintenance plan. What happens when something upstream changes.
6. Give us one reference. A client doing the same kind of work we do.

If those six get clear answers, you're likely talking to a builder. If they get slides, you're talking to a seller.

Agency vs venture builder: a fairer comparison

Not every red flag is the agency's fault —some are baked into the model. A traditional agency is paid to deliver a project and then move on. That's fine for a one-off integration. But automation isn't a deliverable you file away; it's a process that has to keep working, keep paying back, and adapt as the business changes. The agency model has little incentive to stay accountable once the invoice clears.

A venture builder works the other way round. It doesn't hand you a project and leave —it operates the system alongside you, with skin in the game, measured on whether the process actually keeps saving time and money. The same logic applies to the classic build-or-buy question we cover in automate vs hire: the point isn't the tool, it's who's accountable for the outcome. When the partner only wins if the automation keeps working, most of the red flags above simply can't survive the relationship.

That's the difference between renting a demo and owning a result.

How Obsidy works

We don't pitch a platform. We look at where your team's time actually goes, tell you which process has the highest return, what it would cost and what it would save —in writing— before anything is built, and then we run it with your team in the loop where judgement matters. We name the tools, you own the result, and we're measured on whether it keeps paying back. It's the opposite of most of this checklist. See what that looks like on our services page.

Thinking of hiring an AI automation agency and not sure how to tell the builders from the sellers? Let's talk. Write to us at hola@obsidy.com or visit obsidy.com and in a twenty-minute call we'll tell you what we'd automate first, what it would cost and how we'd prove it works.