What procurement teams should ask every agent vendor

Why your standard RFP template will not save you

In the last eighteen months we have been brought in three times to replace an agent vendor a client had already bought. In every case the procurement team had done its job by the book: security questionnaire, SOC 2 report, reference calls, pricing comparison. And in every case the thing that killed the deployment was something none of those artifacts measure - the vendor had built a demo, not a system.

The agent market in 2026 is full of companies that are, functionally, a prompt and a UI on top of someone else's model. That is not automatically disqualifying. But it means the standard software procurement playbook, which assumes the product exists and the question is whether it is priced and secured properly, misses the real risk. With agents, the question is whether the product will still work in month seven, when your ERP schema changes, your volumes double, and the vendor's founding engineer has moved on.

The twelve questions below are the ones we ask when we audit a client's existing agent stack. They cluster into four areas: where the work happens, how correctness is measured, what happens on failure, and what happens after go-live. For each, we have included the answer that should worry you, because worrying answers are usually delivered with great confidence.

Where the work happens: questions 1-3

Question 1: Does the agent operate inside our existing systems, or do we operate inside yours? Thin wrappers almost always answer this with a portal - a new place your team logs in to watch the agent work. Durable platforms write back to your ERP, your CRM, your ticketing system, and your team never changes where they live. When a client's AP team has to check a separate dashboard to see which invoices the agent coded, adoption drops off a cliff around week six. We have measured this: portal-based deployments in our audits showed daily active usage falling below 30 percent of the target team by month two.

Question 2: Show us the exact list of API scopes and permissions you need, per workflow. A vendor who asks for admin credentials to your NetSuite instance to do three-way matching has not thought about blast radius. The right answer is a scoped service account per workflow, with write permissions limited to the specific record types the agent touches. If they cannot produce this list in the sales cycle, they will not produce it under audit either.

Question 3: What happens when we upgrade or reconfigure the underlying system? ERPs get customized. Fields get renamed. Approval chains get restructured. Ask how the vendor detects that the environment shifted under the agent. The worrying answer is silence, or worse, 'the agent adapts automatically.' The good answer involves contract tests against your instance that run on a schedule and fail loudly before the agent processes a single record against a changed schema.

How correctness is measured: questions 4-6

Question 4: What is your measured error rate on a workflow like ours, and how did you measure it? This is the single fastest filter we know. A vendor who cannot tell you their error rate does not have a low one - they have an unmeasured one. Press for specifics: error rate on what task, against what ground truth, on how many cases, from whose data. 'Ninety-nine percent accuracy' with no denominator is a marketing number, not an eval result.

Question 5: Will you run an eval on our data before we sign? Any serious vendor should be willing to take 200 to 500 historical cases - invoices you already coded, disputes you already resolved, orders you already reconciled - and show you agent output against your team's actual decisions. We do this in every engagement, and the results are humbling often enough to be worth it. A workflow that scored 97 percent on the vendor's benchmark routinely scores 85 percent on a client's real data, because the client's data has the weird subsidiaries, the legacy vendor codes, and the one approver who does everything by email.

Question 6: How do you detect quality drift after deployment? Model providers update models. Your business changes. An agent that was fine in March can be quietly wrong in August. Ask whether the vendor samples production outputs for ongoing review, what the sampling rate is, and who looks at the samples. The worrying answer is that monitoring means uptime dashboards. Uptime is not correctness. An agent can be fully operational and miscoding every intercompany invoice it sees.

What happens on failure: questions 7-9

Question 7: Where are the human review gates, and who decides where they go? Every agent system we operate has explicit thresholds where work stops and a named human approves before anything posts. The gates are configured per client, per workflow, usually by dollar amount and confidence score - for one logistics client, anything above 5,000 dollars or below 90 percent match confidence routes to a person. A vendor who pitches 'fully autonomous, no human in the loop' as the headline feature is optimizing for the demo, not for the day something goes wrong.

Question 8: Can we roll back what the agent did? Ask for the mechanics, not the promise. Every write the agent makes should be attributable, timestamped, and reversible - either through the source system's own audit trail or through a compensating transaction the vendor can generate. In one audit, we found an agent that had been applying cash to the wrong customer accounts for eleven days. The vendor's rollback plan was a CSV export and an apology. The remediation took the client's AR team three weeks.

Question 9: When the agent causes a financial error, who is liable and what does the contract say? Read the limitation of liability clause with agents specifically in mind. Most standard SaaS contracts cap liability at fees paid, which is fine for a reporting tool and absurd for software that posts journal entries. You may not get unlimited liability, and you should not expect it, but you should get an explicit conversation and a cap that acknowledges the agent touches money. A vendor who has never had this conversation before has never run agents anywhere serious.

A vendor who cannot tell you their error rate does not have a low one - they have an unmeasured one.

What happens after go-live: questions 10-12

Question 10: Who operates this after deployment, and what does that cost? Agents are not fire-and-forget software. Someone has to review the sampled outputs, tune the gates, handle the escalations, and update the workflow when your business changes. Ask whether that is you, the vendor, or an unpriced gap between you. In our own engagements this is the bulk of the relationship - the build is months one through three, the operating is everything after - and we price it explicitly because unpriced operations always become unowned operations.

Question 11: What does the rollout schedule look like, month by month? Distrust any plan that goes to full production in the first month. Our standard pattern is: month one, the agent runs in shadow mode and its outputs are compared against what your team actually did. Month two, it goes live on the lowest-risk slice with tight gates - one entity, one vendor category, one lane. Months three through six, scope expands as the eval numbers hold. Vendors who resist a staged rollout are usually resisting the measurement, not the timeline.

Question 12: If we leave, what do we keep? The prompts, workflow configurations, eval sets, and decision logs built during your deployment encode your business. Ask which of those you own and in what format they export. The worrying answer is that everything lives in the vendor's proprietary format and exports as a PDF. The good answer is that your workflow definitions and full decision history come with you, machine-readable, because a vendor confident in their platform does not need to hold your data hostage to keep you.

How to score the answers

You will not get perfect answers to all twelve questions, and you should not require them. What you are listening for is whether the vendor has actually lived these problems. Teams that have run agents in production for real finance and operations workloads answer with specifics, numbers, and the occasional war story. Teams that have not answer with adjectives.

Our rough scoring rule: a vendor should answer at least nine of the twelve with concrete mechanisms. Fewer than that, and you are not buying a platform - you are funding one, with your own operations as the test environment. Sometimes that trade is worth making for a genuinely novel capability. But make it knowingly, price it accordingly, and put the review gates, eval requirements, and exit terms in the contract before the pilot starts, because you will never have more leverage than you do right now.