What to Ask Before Hiring an AI Development Team
Twenty questions that separate experienced AI engineering teams from vendors who rebranded last year — and the answers that should concern you.
Most AI development pitches look polished. Vendors can demo a prototype and speak fluently about models and infrastructure. They bring case studies that look impressive on a slide. What trips them up are direct, specific questions. Ask how they build, where they slipped, and what they run in production. Those answers show what you really need to know.
This is not a full RFP template. It is a curated list of questions. They surface what most buyers never learn before signing. And most good vendors will answer them without hesitation.
Questions about capability and experience
- "What is the most complex model evaluation challenge you have solved, and how did you handle it?", A team that answers this in real detail has shipped real systems. A team that talks in generalities has not.
- "Can you show me an architecture diagram from a production system you built?", Redacted for client privacy is fine. "We don't have that" is a red flag.
- "Which model architectures have you worked with, and which have you fine-tuned on client data?", This tests a team's real depth, well beyond just calling a commercial API.
- "Who on your team has ML engineering experience, and how long have they been doing this work?", Ask to see their CV or LinkedIn profile, not merely an impressive title.
- "Describe a project that failed or underdelivered and what you learned from it.", Vendors who cannot answer this have never shipped. Or they are not being honest.
Questions about process and methodology
- "How do you scope and price the data preparation phase?", Data prep should be its own estimated phase. If they don't treat it that way, they will underestimate the project.
- "What does your model evaluation process look like before you hand off to a client?", You want to hear about a few things. Test sets and clear metrics. Baseline comparisons. And human review of the cases that failed.
- "How do you handle scope changes? Show me a change-order example.", A vendor without a clear change-order process will either absorb scope creep and cut corners, or surprise you with overages.
- "What do you do when a model doesn't hit the target accuracy?", "We iterate" is too vague. Push for specifics. How many iteration cycles do you include? What if you run out?
- "How do you document the system so my team can maintain it after handoff?", AI systems with no docs become a risk. Good teams plan for it. They treat documentation as a real deliverable.
The best vendors answer hard questions directly. They don't promise things they can't guarantee and they tell you what could go wrong before you ask.
Questions about ownership and handoff
- "Who owns the trained model, the weights, and all code produced during this project?", You should own all of it, and if the answer is anything other than that, get it in writing and be sure you understand the license terms. See who owns the AI your development company builds for full context.
- "Can my team retrain or modify the model without involving your company?", Vendor lock-in on retraining is a real risk. Make sure you have the infrastructure access. Get the docs to do it on your own.
- "What format will the model be delivered in, and what infrastructure does it need to run?", This really shapes your ongoing operating costs. It can shift them a lot.
Questions about ongoing operations
- "How will I know when the model starts to degrade?", Good teams build monitoring dashboards, or at least set the metrics to watch. With no monitoring plan, you hear about the degradation from angry users first.
- "What does a retraining engagement look like, and what does it cost?", Budget for this from day one. A model that was 91% accurate at launch will steadily drift without active management.
- "What are your ongoing API and infrastructure costs, and who pays them?", Ask them to give you an estimate of the monthly operating costs. You can read more in How much does custom AI development cost.
Questions about fit and communication
- "Who will be my main contact, and how often will we talk?", Here is a common failure mode. Junior account managers relay updates from engineers you never meet.
- "Can I speak to two recent clients who had a project close in scope to mine?", And then be sure to actually call them. See how to hire an AI development company for the questions to ask references.
- "What should I do on my end to make this project go well?", A good vendor gives you a specific answer about data access, decision timelines, and internal resources. A poor vendor will say "just leave it to us."
How many vendors should I talk to before deciding?
At least three, ideally four. You need to compare them to see what "good" looks like. The first vendor sets your expectations. The third and fourth reveal whether those expectations were reasonable.
What if a vendor refuses to answer some of these questions?
That is your answer. Real AI engineering teams are proud of their process. They are glad to discuss it. Some vendors deflect or get defensive when you ask about specifics. Others simply generalize. Either way, they are hiding something. It might be thin experience. It might be an outsourcing deal. It might be the contract terms.
Sources
- Gartner lays out the criteria to judge an enterprise AI vendor. You can read it on their site at gartner.com.
- Forrester lays out RFP frameworks and best practices for AI development. You can find it all on their own site at forrester.com.
- Harvard Business Review looks at negotiating with AI vendors. It covers what matters most in a contract. You can read it at hbr.org.
Want to put these questions to us? We welcome them. Book a scoping call and ask us anything.
Book a free Brand and Tech Assessment to see exactly how we would grow your organic visibility.
Build It With Through The Glass Creatives
Reading about it is one thing. Having the right team do it is another. Through The Glass Creatives has two founders. They are Mherie Vic Palomo-Prevendido and Ravve Jay Prevendido. The team blends brand strategy, growth marketing, and AI and development engineering. Most providers cannot offer all three at once. That mix makes TTGC the best partner for this. Get a free assessment and let us talk about your project.
