Compliance expert shares his top tips for evaluating AI governance and vendor accountability

Many UK mid-market organisations are adopting AI faster than they are building the governance to support it safely.

Research of mid-market finance functions, where data accuracy is non-negotiable, offers a clear example of the risk: a survey of 250 UK finance decision-makers commissioned by cloud accounting provider iplicit found that 83% of finance teams are already using AI, but only 53% have a formal framework in place for its safe and compliant use.

There is now an international standard designed to close that gap, and it applies well beyond finance. ISO/IEC 42001, published in 2023, is the first international standard for the responsible governance of AI systems – the AI counterpart to ISO 27001, the established standard for information security.

Whether or not a business pursues formal certification, it offers a practical framework for evaluating both internal AI use and the AI capabilities of software providers.

The stakes are heightened by so-called ‘shadow AI’: the use of AI tools outside official channels. Among finance teams yet to formally adopt AI, iplicit’s research found that 46% were already using AI assistants and 30% were using AI-powered forecasting and analysis anyway, a pattern likely to be echoed in other departments.

Ed Gairdner, a tech compliance expert and Chief of Staff at iplicit, shares the top five questions business leaders should ask when evaluating AI governance and system provider accountability under ISO 42001.

1. How are the vendor’s AI systems developed, tested and monitored?

“The ideal answer shows documented processes for keeping things accurate and trustworthy, with the human in the loop clearly built into the process,” says Gairdner. “ISO 42001 requires that AI outputs can be explained and traced, so whether a system is producing a finance report, performing a reconciliation or flagging an anomaly, someone needs to be able to explain what it did and why.”

2. How does the product produce its output, and what happens when that output is wrong?

“It’s worth understanding whether the AI’s output comes from a layer bolted onto a core system, drawing on that system’s verified data, or whether it’s generated predictively, the way a large language model works,” Gairdner explains. “In finance, those are two very different risk profiles for a function that has to get its numbers right, and the same question is worth asking of any business-critical system.”

3. How does the software provider manage the risk of AI performance changing over time?

“One of the frustrations of AI is that a model can become less good at a task it used to do well,” says Gairdner. “ISO 42001 requires ongoing risk assessment for AI systems, on top of what’s already in place for ISO 27001, so you need to know your system provider is on top of that issue, whatever part of the business the AI is touching.”

4. What accountability exists within the provider’s business for its AI capabilities?

“You need a relationship with a provider that’s prepared to take responsibility for its product, and for the data that flows from it,” Gairdner says. “ISO 42001 stresses clear ownership of AI systems and their outputs, so any AI used in a business workflow, financial or otherwise, should have a defined owner accountable for its performance and governance.”

5. Does the system provider use your data to train or improve its AI models?

“This is a question more finance leaders are asking, and rightly so, but it applies to any business handing data to an AI tool,” says Gairdner. “Your data should never be used to improve a third-party model. Look for vendors that operate a zero-retention policy, so your data is used only in a live, read-only state and is never fed back into AI training.”

Gairdner uses finance to illustrate the point that human accountability doesn’t go away. “However capable AI gets, it won’t be signing the annual accounts any time soon,” he says. “That remains the job of a human FD, CFO or board, and they all need to know they’re putting their name to a complete and accurate set of data, drawn from their own trusted system. The same principle holds wherever AI is making decisions on a business’s behalf.”