What to Ask Before Hiring an AI Development Team
Twenty questions that separate experienced AI engineering teams from vendors who rebranded last year — and the answers that should concern you.
Most AI development pitches are polished. Vendors know how to demo a prototype, speak fluently about models and infrastructure, and provide case studies that look impressive on a slide. What they are less prepared for are direct, specific questions about their engineering process, their past mistakes, and their production systems. Those questions are where you learn what you actually need to know.
This is not an exhaustive RFP template. It is a curated list of questions that surface the information most buyers never get before signing — and that most good vendors will answer without hesitation.
Questions about capability and experience
"What is the most complex model evaluation challenge you have solved, and how did you handle it?" — A team that can answer this specifically has shipped real systems. A team that talks in generalities probably hasn't.
"Can you show me an architecture diagram from a production system you built?" — Redacted for client confidentiality is fine. "We don't have that" is a red flag.
"What model architectures are you experienced with, and which have you fine-tuned on client data?" — This tests depth beyond calling a commercial API.
"Who on your team has ML engineering experience, and how long have they been doing this work?" — Ask for a CV or LinkedIn, not just a title.
"Describe a project that failed or underdelivered and what you learned from it." — Vendors who can't answer this have either never shipped or aren't honest.
Questions about process and methodology
"How do you scope and price the data preparation phase?" — If they don't mention data prep as a distinct, estimated phase, they are underestimating the project.
"What does your model evaluation process look like before you hand off to a client?" — You want to hear: test sets, evaluation metrics, baseline comparisons, human review of failure cases.
"How do you handle scope changes? Show me a change-order example." — A vendor without a clear change-order process will either absorb scope creep (and cut corners) or surprise you with overages.
"What is your approach when a model doesn't hit the target accuracy?" — "We iterate" is too vague. Ask specifically: how many iteration cycles are included, and what happens when you exhaust them?
"How do you document the system so my team can maintain it after handoff?" — Undocumented AI systems are liabilities. Good teams budget for documentation as a deliverable.
The best vendors answer hard questions directly. They don't promise things they can't guarantee and they tell you what could go wrong before you ask.
Questions about ownership and handoff
"Who owns the trained model, the weights, and all code produced during this project?" — You should own all of it. If the answer is anything other than that, get it in writing and understand the license terms. See who owns the AI your development company builds for full context.
"Can my team retrain or modify the model without involving your company?" — Vendor lock-in on model retraining is a real risk. Verify you have the infrastructure access and documentation to do it independently.
"What format will the model be delivered in, and what infrastructure does it require to run?" — This affects your ongoing operating costs significantly.
Questions about ongoing operations
"How will I know when the model starts to degrade?" — Good teams build monitoring dashboards or at least define the metrics to watch. No plan for monitoring means you'll only find out about degradation from angry users.
"What does a retraining engagement look like, and what does it cost?" — Budget for this from day one. A model that was 91% accurate at launch will drift without active management.
"What are your ongoing API and infrastructure costs, and who pays them?" — Get an estimate for monthly operating costs. How much does custom AI development cost covers this.
Questions about fit and communication
"Who will be my primary contact, and how often will we communicate?" — Junior account managers relaying updates from engineers you never meet is a common failure mode.
"Can I speak to two recent clients who had a project similar in scope to mine?" — And then actually call them. See how to hire an AI development company for the specific questions to ask references.
"What should I do on my end to make this project go well?" — A good vendor will give you a specific answer about data access, decision timelines, and internal resources. A poor vendor will say "just leave it to us."
How many vendors should I talk to before deciding?
At least three, ideally four. You need comparison to calibrate what "good" looks like. The first vendor sets your expectations; the third and fourth will reveal whether those expectations were reasonable or not.
What if a vendor refuses to answer some of these questions?
That is your answer. Legitimate AI engineering teams are proud of their process and happy to discuss it. Vendors who deflect, generalize, or get defensive when asked about specifics are protecting something — either their lack of experience, their outsourcing relationships, or their contractual terms.
Sources
Gartner — Enterprise AI vendor evaluation criteria. gartner.com
Forrester — AI development RFP frameworks and evaluation best practices. forrester.com
Harvard Business Review — Negotiating with AI vendors: what matters in a contract. hbr.org
Want to put these questions to us? We welcome them. Book a scoping call and ask us anything.
Book a free Brand and Tech Assessment to see exactly how we would grow your organic visibility.
