Skip to content
Sam AkbariFractional CXO
Book a 30-min call

Free tool · Australia

AI vendor due diligence scorecard

Rate up to three AI vendors on 27 questions that decide whether a tool is safe to buy and likely to pay off. Your answers stay in your browser unless you ask for the report.

Short answer: before you sign with an AI vendor, get written answers on whether your data trains their models, where it is stored, how they prove security, what stops wrong answers reaching customers, who owns the outputs and how you leave. A "No" on any of those is a red flag, whatever else the vendor does well.

1. Name the vendors you're comparing

Now rating:

2. Answer for Vendor A

Answer from what the vendor has shown you in writing. If you don't know yet, leave it as "Not sure yet": it goes on the list of questions to send them.

Business fit and value

Can the vendor name the business outcome it will improve, and how it will be measured?

What good looks like: A named measure (hours saved, error rate, cycle time) with a baseline and a target, agreed before you sign.

Has it been proven with organisations like yours, of a similar size and industry?

What good looks like: Reference customers you can call, ideally in Australia, using it in production rather than in a pilot.

Will they run a time-boxed pilot on your own data before a long commitment?

What good looks like: Success criteria agreed up front, and a way out if the pilot misses them.

Data and privacy

Will your inputs, outputs or files be kept out of training or improving their models?Critical

What good looks like: Yes by default, in writing, with any opt-in controlled by you.

Do you know where your data is stored and processed, including whether it leaves Australia?Critical

What good looks like: Named regions and sub-processors. If data goes offshore, you can still meet your obligations under APP 8 (cross-border disclosure).

Is there a defined retention period, and can you export and delete your data when you leave?

What good looks like: Retention set in the contract, export in a usable format, and confirmed deletion at the end.

Do you know every third party that receives your data (sub-processors, model providers, search features)?

What good looks like: A current sub-processor list, notice of changes, and the option to switch off features that send data to third parties.

Security

Can they share current independent security assurance?Critical

What good looks like: An ISO/IEC 27001 certificate or a SOC 2 Type II report, plus recent penetration test results.

Does it support your single sign-on, role-based access and audit logs?

What good looks like: Single sign-on with your identity provider, least-privilege roles, and logs you can export.

Will they notify you of a security incident within a set time, in the contract?Critical

What good looks like: A notification period in hours, short enough for you to meet your own Notifiable Data Breaches obligations.

Accuracy and performance

Can you test accuracy on your own data and edge cases before go-live?

What good looks like: Their evaluation results, plus a test on a sample of your real cases.

Is there a control that stops wrong or made-up answers reaching customers or decisions?Critical

What good looks like: Confidence thresholds, links back to the source, and human review for anything high-stakes.

Will you be told before a model update changes how it behaves?

What good looks like: Release notes, advance notice of material changes, and a way to re-test first.

Responsible AI and governance

Can they explain what the system does, its limits and the data it relies on, in terms your board would accept?

What good looks like: Plain-English documentation of how outputs are produced and where the system should not be used.

Have they tested for unfair outcomes that matter for your customers and staff?

What good looks like: Testing results for the groups relevant to your use, and a process to report and fix issues.

Can you keep a person in control of important decisions?

What good looks like: Approval steps you configure, and an easy way to override or switch off automated actions.

Will they give you what you need for your own AI register, impact assessment and policy?

What good looks like: The information Australia’s Guidance for AI Adoption expects you to hold, provided without a fight.

Commercials and lock-in

Will the cost stay predictable as usage grows?

What good looks like: Clear unit pricing, usage alerts or caps, and a cost modelled at two to three times today’s volume.

Do you own your data, outputs, prompts and configurations?Critical

What good looks like: Ownership stated in the contract, including anything built or tuned for you.

Can you leave without a penalty or a rebuild?

What good looks like: A documented exit path, data export, and no punitive minimum terms or automatic renewals.

Do the liability terms match the risk, including claims about generated content?

What good looks like: A liability cap that reflects the exposure, and an IP indemnity for outputs where it matters.

Vendor strength and support

Is the vendor financially sound, and is there a plan if it is acquired or closes?

What good looks like: Evidence of funding or profitability, and an exit or escrow plan for anything critical.

Is support available in your hours, with service levels in the contract?

What good looks like: At least Australian business-hours support, with uptime and response targets you can enforce.

Does what you need exist today, rather than on the roadmap?

What good looks like: Buy for what works now. Treat roadmap promises as a bonus, not the reason to sign.

Integration and adoption

Does it connect to your core systems through supported APIs or connectors?

What good looks like: Documented integrations for the systems you named, tested during the pilot.

Do you know what implementation will take from your own team?

What good looks like: A plan that names your people’s time and roles, not just the vendor’s.

Is training and change support included, so people actually use it?

What good looks like: Role-based training, adoption measures and a named customer success contact.

3. Your comparison

VendorScoreRed flagsNot sure yetVerdict
Answer a few questions to see the comparison.
    Get the full report

    Email me the full comparison report

    A Word report with the weighted scores, every answer side by side, the red flags to resolve and the follow-up questions to send each vendor.

    I'll use your details to send the report, and to send my notes only if you tick the box. See the privacy policy.

    Method

    How the scorecard works

    Each answer scores 2 for yes, 1 for partly and 0 for no or not sure. Each area is scored out of 100%, then weighted by how much it matters to you (normal, high or critical) to give the overall score.

    Six questions are marked critical: training on your data, where data lives, security assurance, incident notification, controls on wrong answers, and ownership of outputs. A "No" on any of them is a red flag, because a high score elsewhere doesn't make it safe to sign.

    The privacy questions follow the OAIC's guidance on privacy and commercially available AI products. The governance questions line up with the National AI Centre's Guidance for AI Adoption. This is general information, not legal advice.

    Fractional Chief AI Officer AI governance checklist AI Readiness Scorecard

    Want an independent view before you sign?

    Thirty minutes on the vendors you're weighing up, what to ask them, and whether the business case stacks up.

    Book a callSend a message