7 Questions to Ask Before Signing with an AI Vendor—and the Answers That Should Worry You

Most AI vendor guides focus on models, safeguards, and compliance. These seven questions help you assess something just as important: judgment, accountability, delivery discipline, and whether the vendor can solve the business problem in real conditions.

Why Standard Vendor Checks Fall Short in AI

When you evaluate software, you usually ask what the product can do. When you evaluate an AI vendor, you need to ask how they work under real conditions, how they measure quality, and how they keep the system aligned with business goals over time.

AI capability is different from software capability

In software projects, deterministic behavior carries much of the load. A feature either works or it does not. In AI projects, output quality depends on things like data quality, process fit, evaluation design, monitoring, integration, and ongoing iteration.

That changes the way buyers need to assess capability.

A good AI vendor should be able to explain how the solution fits your data, systems, review process, and business goals. They know where AI works well, where it struggles, and where a rules-based approach or a simpler workflow change would deliver a better result.

The strongest signal in judging AI capability is the vendor’s ability to connect that model to your actual operating environment.

This is why AI vendor evaluation should test judgment as much as capability. Most of all, it should test accountability and willingness to give honest feedback and push back when the proposed solution is weak.

But that’s not always easy to assess. Every vendor will tell you they are the right fit; their website full of polished claims and case studies, their sales deck filled with percentages and delivery promises.

So how do you separate the pitch from the real signal?

You ask better questions.

AI Vendor Due Diligence: 7 Questions to Ask First

Many companies today are trying to assess potential AI partners and struggling to tell strong vendors apart from well-prepared ones.

That is why we drew on our AI project experience to put together a set of questions that goes beyond the standard features conversation, and focuses on something just as important as technical skill: judgment, accountability, and the ability to deliver in real conditions.

These seven questions help decision-makers evaluate how vendors operate beyond the pitch and spot the delivery risks that polished sales decks can hide.

For each one, we describe what a strong answer sounds like—and what answers, including some that sound good, should make you pause.

Question 1: Can you show me this working with real-world data?

What you are testing: Whether the vendor can move from a proof of concept to a system that works under real-life conditions.

This is one of the best opening questions because it moves the conversation out of the demo environment and into reality. Clean sample data can make almost any solution look reliable.

A worrying answer stays in the demo environment. The vendor points to polished case studies or a walkthrough on clean, well-behaved data. When you push further, they fall back on broad claims about “robust preprocessing pipelines” or “handling edge cases through our proprietary framework.”

A strong answer sounds specific. The vendor can explain how they assess data readiness, what preprocessing they expect, and what limitations they would raise before promising results. They can also tell you what usually breaks first when real data enters the system.

Question 2: Who owns the integration between the AI component and the rest of the system?

What you are testing: Whether the vendor can define ownership clearly at the point where the model meets the wider system and take responsibility for the delivery layer.

This question matters for two reasons. First, responsibility should always be clear. Second, the answer shows whether the vendor is willing to take ownership and define their scope properly.

A worrying answer sounds cooperative but vague. Phrases like “we’ll coordinate with your engineering team,” “that depends on your internal setup,” or “integration is usually handled jointly” often sound good, but in practice they can signal a responsibility gap. When ownership is blurred, issues get passed around instead of solved, and the client usually ends up carrying more delivery risk than expected.

A strong answer gives you a clear picture of how responsibility is divided across the project. It should tell you who owns the integration design, who is building each part, what the vendor expects from your team, how dependencies will be handled, and how work will pass from one stage to the next. It should also make clear who is accountable if the model performs well on its own but fails in the live workflow.

Question 3: What’s your retraining and model lifecycle plan after deployment?

What you are testing: Whether the vendor treats AI as a product or a project.

MLOps is the operational backbone of production AI, and this question quickly shows how mature their process is.

A worrying answer refers to “standard MLOps practices” without any detail. A vendor may say they handle monitoring after deployment and have a retraining pipeline. If they cannot explain how it works, press further. Ask what they monitor, what triggers retraining, how they version models, and what happens when performance drops.

A strong answer goes into technical and operational detail. A good vendor should be able to describe to you how they monitor performance, what triggers a retraining cycle, how models are versioned, and how they roll back when a new version performs worse than the old one in detail.

Question 4: How do you define project success, and in what terms?

What you are testing: Whether the vendor connects technical performance to business outcome.

This question separates vendors who want to help you push your business further, from the ones who just deliver a solution and don’t stay to make sure it works for you.

A worrying answer leans on model metrics alone. The vendor talks about accuracy, F1, recall, BLEU, or precision at K. Those numbers have value, but they only measure model performance. A model can score well in testing and still create no business value. If the vendor stops at technical metrics, that is a warning sign.

A strong answer defines success in your business’s language. That means specific business outcomes, not technical performance of the model. A good vendor will want to agree on a baseline before they start, because success without a baseline is untestable.

Question 5: Can you tell me about a project where the AI technically worked but didn’t deliver business value?

What you are testing: Whether the vendor has the maturity to separate technical success from business success.

This question tests honesty under pressure. It asks the vendor to speak about a failure mode that happens often in AI: technical success without business payoff.

A worrying answer pivots to a success story or denies the premise. You may hear, “That’s a great question. We’ve actually had a lot of success recently with…” Or, “We haven’t really run into that. Our projects tend to deliver ROI from month one.” A vendor who cannot name a failure is either avoiding the question or has very little real experience.

A strong answer is specific and includes a takeaway. The vendor should be able to explain what happened, what they learned, and how that lesson changed their process. The strongest vendors turn those lessons into standard practice and can point to the exact step in their methodology where a similar failure is now prevented.

Question 6: What would you tell us not to build, and why?

What you are testing: the vendor’s approach; are they a partner, or simply a contractor?

A strong vendor will not turn every idea into a build plan. That is why this question matters: it shows whether the vendor can think like an advisor, not just a supplier.

A worrying answer avoids saying a hard no. That usually signals a vendor who is comfortable turning a flawed brief into a project plan, even if the result is expensive, slow to adopt, or misaligned with the real business problem.

A strong answer names a specific type of project they would advise against, or gives a real example of steering a client away from the wrong build. For example, they may explain that AI is a poor fit for a problem a rule engine could solve or warn against automating a workflow that is still changing too fast.

At Addepto, we come back to this question whenever the scope moves. That helps keep the technical direction tied to the business outcome.

Question 7: What does your quality and testing process look like for AI-generated outputs?

What you are testing: Whether the vendor has built a reliable evaluation practice.

Last but not least, this is a strong question to ask whether the vendor has a defined method for testing quality, spotting failure patterns, and, most importantly, deciding when the system is safe to use in production

A worrying answer treats AI quality assurance as a solved extension of standard software QA. That usually signals a weak evaluation process. AI systems fail differently, and those failures often sit outside a conventional test plan. If the answer stays at the level of “rigorous testing” or “industry-standard frameworks,” ask them to show you exactly what that means in practice.

A strong answer describes a defined evaluation practice in enough detail that you can picture how the system is tested before and after launch. The vendor should be able to explain what quality means for this use case, how they measure it, what kinds of failure they actively test for, and where human review sits in the process.

How to Use These Questions: What the Answers Reveal About the Vendor

These questions are not meant to produce perfect answers. They’re not a checklist, but rather a diagnostic tool.

One weak answer does not automatically rule a vendor out. A team can be weaker in one area and strong in several others. That happens. What matters is the pattern: where they are specific, where they stay vague, where they take ownership, and where they shift responsibility away from themselves.

How to Read the Pattern

Taken together, these questions are trying to answer a bigger one: can this vendor help solve the business problem in front of us?

That matters more than any polished explanation of model choice, architecture, or framework.

Why Vendor Choice Carries So Much Weight

As Kasia Zielosko pointed out in her recent article on why AI projects fail, the wrong vendor can put a project on the wrong path early, and by the time the problem is visible, the cost is already real.

That’s part of why AI so consistently underdelivers.

More than 8 in 10 enterprise AI implementations fail to deliver measurable value, a number that reflects, in large part, initiatives that were technically competent but strategically misconceived.

 

Katarzyna Zielosko

Head of Growth Marketing at Addepto

We prepared these questions to help you reduce that risk.

Capability may get a vendor in the door, but it’s accountability that tells you whether to sign.

A vendor who can tell you what they would decline, what went wrong in a past project, and who owns the hard middle ground between AI and engineering is showing you something far more valuable than confidence.

Most vendors can describe capability. Strong partners can describe responsibility.

Look For a Partner, Not Just a Vendor

If you’re choosing an AI vendor right now, remember, make sure to ask questions that force clarity.

At Addepto, we see that as part of the consulting job itself: helping clients define the right problem, pressure-test the solution, and carry the work through implementation, deployment, and optimization with one clear line of accountability.

If you want a partner who will tell you what to build, what to cut, and why, talk to us.