What to Ask Before Hiring an AI Development Team
Twenty questions that separate experienced AI engineering teams from vendors who rebranded last year, and the answers that should concern you.
Most AI development pitches are polished. Vendors know how to demo a prototype. They speak with ease about models and infrastructure. They bring case studies that look great on a slide. What they are less ready for are direct questions about how they engineer, what they got wrong in the past, and what they run in production. Those questions are where you learn what you need to know.
This is not a full RFP template. It is a short list of questions that surface what most buyers never learn before they sign. Most good vendors will answer them without pause.
Questions about capability and experience
"What is the hardest model evaluation challenge you have solved, and how did you handle it?" A team that can answer this in detail has shipped real systems. A team that talks in general terms probably has not.
"Can you show me an architecture diagram from a production system you built?" It is fine if they redact it to protect a client. "We don't have that" is a red flag.
"What model architectures have you worked with, and which have you fine-tuned on client data?" This tests for real depth. It goes past just calling a commercial API.
"Who on your team has ML engineering experience, and how long have they been doing this work?" — Ask for a CV or LinkedIn, not just a title.
"Describe a project that failed or underdelivered and what you learned from it." A vendor who cannot answer this has never shipped, or is not being straight with you.
Questions about process and methodology
"How do you scope and price the data preparation phase?" Data prep should be its own phase, with its own estimate. If they do not treat it that way, they are underestimating the project.
"What does your model evaluation process look like before you hand off to a client?" You want to hear four things. Test sets. Metrics to score the model. Baselines to compare it against. And a human review of the cases where it failed.
"How do you handle scope changes? Show me a change-order example." — A vendor without a clear change-order process will either absorb scope creep (and cut corners) or surprise you with overages.
"What is your approach when a model doesn't hit the target accuracy?" "We iterate" is too vague. Ask how many rounds of work are included. Then ask what happens when you use them all up.
"How do you document the system so my team can maintain it after handoff?" An AI system with no docs is a liability. Good teams treat the docs as a deliverable and budget for them.
The best vendors answer hard questions directly. They don't promise things they can't guarantee and they tell you what could go wrong before you ask.
Questions about ownership and handoff
"Who owns the trained model, the weights, and all code produced during this project?" — You should own all of it. If the answer is anything other than that, get it in writing and understand the license terms. See who owns the AI your development company builds for full context.
"Can my team retrain or modify the model without involving your company?" Vendor lock-in on model retraining is a real risk. Check that you have the access to the infrastructure and the docs to do it on your own.
"What format will the model be delivered in, and what infrastructure does it need to run?" That has a big effect on your ongoing running costs.
Questions about ongoing operations
"How will I know when the model starts to degrade?" Good teams build monitoring dashboards. At the least, they define the metrics to watch. With no plan for monitoring, you will hear about it first from angry users.
"What does a retraining engagement look like, and what does it cost?" — Budget for this from day one. A model that was 91% accurate at launch will drift without active management.
"What are your ongoing API and infrastructure costs, and who pays them?" Get an estimate for the cost to run it each month. How much does custom AI development cost covers this.
Questions about fit and communication
"Who will be my primary contact, and how often will we communicate?" Watch for this common failure mode. A junior account manager relays updates. You never meet the engineers.
"Can I speak to two recent clients who had a project similar in scope to mine?" And then actually call them. See how to hire an AI development company for the exact questions to ask them.
"What should I do on my end to make this project go well?" — A good vendor will give you a specific answer about data access, decision timelines, and internal resources. A poor vendor will say "just leave it to us."
How many vendors should I talk to before deciding?
At least three, and ideally four. You need to compare them to work out what "good" looks like. The first vendor sets your expectations. The third and fourth show you whether those expectations were fair.
What if a vendor refuses to answer some of these questions?
That is your answer. A real AI engineering team is proud of its process and happy to talk about it. Some vendors deflect, speak in general terms, or get defensive when you ask for specifics. They are protecting something. It could be a lack of experience, an outsourcing deal, or the terms of their contract.
Sources
Gartner sets out how to judge an enterprise AI vendor, with clear criteria. See gartner.com
Forrester covers RFP frameworks for AI development. It also covers best practice for how to judge a vendor. See forrester.com
Harvard Business Review looks at how to negotiate with AI vendors, and what matters in a contract. See hbr.org
Want to put these questions to us? We welcome them. Book a scoping call and ask us anything.
Book a free Brand and Tech Assessment to see exactly how we would grow your organic visibility.
Build It With Through The Glass Creatives
Reading about it is one thing. Having the right team execute it is another. Through The Glass Creatives was founded by Mherie Vic Palomo-Prevendido and Ravve Jay Prevendido. We bring brand strategy, growth marketing, and AI and development engineering under one roof. Most providers simply cannot offer all three. That mix is what makes TTGC the best partner to bring this to life. Get a free assessment and let us talk about your project.
