Q&A: Guardrails for Third-Party AI Tools

Companies don’t often have the internal resources needed to develop their own AI tools, leading many to leverage third-party solutions and fine tune them to their needs. What are some of the steps organizations should take internally before beginning to evaluate outside providers?

Nevin: Conducting an internal assessment is an important step. Often, companies want to go straight to uplifting their third-party evaluation program, but should first examine their existing internal landscape regarding AI and risk management. Doing so will help teams to understand the use cases already happening, where AI is being deployed, which tools offer AI options that are not yet deployed, and what data is in use. Taking inventory of AI already in the company will reveal existing risk, and in turn, help refine assessment criteria and governance controls needed to implement additional tooling. 

Bordignon: Additionally, companies often underestimate the complexity of integrating a third-party AI tool. Having the house in order first, with clarity about requirements and existing diligence processes, will help to avoid surprises and project delays when implementation challenges arise.   

Who should be at the table through these processes? And do you have any advice for how they should work together within the organization?

Nevin: This process is complicated because many large organizations have procurement and commercial teams that are typically involved in the buying process for new technology. In addition to those stakeholders, risk functions including business continuity, cybersecurity, data privacy and legal, as well as IT, are all at the table. Together, these groups tackle the two key components of AI governance: the policy and risk framework and the technical and data elements.

In an efficient process, stakeholders will collaborate to develop the criteria for acceptance of outside AI tools and layer that onto existing practices for onboarding new technology. 

Once internal assessment and alignment has been achieved, and the organization is ready to conduct diligence on an AI vendor, what are some of the critical questions that need to be covered? 

Bordignon: Building on questions that are already part of existing diligence processes, organizations will want to add several AI-specific considerations. Fully understanding the technology is key — so, for example, whether it’s proprietary to the third party or the third party is facilitating a technology built by another vendor. Additionally, it’s important to assess how the provider has controlled the data used to test and train the tool. Has the company upheld and documented data privacy compliance with its data? How has data quality been managed? Are there any intellectual property or copyright concerns within the data?

Documentation that details AI output and model explainability should also be available. This component can be easily overlooked but is important. Stakeholders should ask whether the product documentation  includes explanation of the model in a transparent manner and if the vendor monitors its AI system methodically to mitigate bias, drift, etc.

Should organizations test and validate third-party AI systems before implementation? 

Bordignon: Yes, we advise clients to conduct pilot or proof of concept exercises with a limited set of users, in a contained environment, and with compliance documentation to guide and benchmark testing. This step helps reveal how the AI is performing in reality, without exposing the entire organization. It also provides insight into whether the organization’s data is quality and ready to be used within an AI system.

All in all, testing modalities are more complex with AI but are essential to highlight usability, the limits of the tool and any issues with the underlying datasets.  

Nevin: In many of our proofs of concept with clients, we’ve seen that this testing provides a reality check on the efficacy of governance and compliance controls. It can also help validate the business case before all the work is done in implementation. Eventually, this validation can feed into change management efforts by giving insight into potential pitfalls that may need to be resolved or areas where users need additional training to use the tool as intended. 

An important note is that additional care needs to be given to these validation exercises due to limitations of testing in the vendor environment, where it may not be possible to assess performance against company data (vs. a vendor’s sample data). An AI model may work in the test environment but because the company’s data may differ materially from the data the tool was trained on, results may differ between pilot testing and post implementation. Therefore, teams will need to evaluate to consider how to deal with that, such as by externalizing sample data into the pilot or conducting additional testing and diligence during implementation.  

Can you talk about the best practices related to monitoring after a tool is implemented? 

Bordignon: Privacy and governance professionals are familiar with the importance of ongoing monitoring of vendors and procedures. AI must be addressed in a similar way —  it’s very important to not just turn it on and forget it. A continuous cycle of testing and revalidating the vendors and the data must be established at the outset and diligently upheld. Standards frameworks such as NIST and ISO are helpful in guiding organizations in monitoring efforts, with third parties and internally. 

Nevin: Ongoing monitoring also bolsters readiness for any errors or incidents that may occur with a vendor or internally. For example, usage failures that may constitute the need for improved user training, awareness and prompt engineering. Or, more seriously, model failures, and ensuring that vendors are prepared to respond to a systematic failure of their model.

More information about vendor risk management for AI is covered in a recorded webcast Dera and Tracy hosted with IAPP, here.

Related topics:

The views expressed herein are those of the author(s) and not necessarily the views of FTI Consulting, its management, its subsidiaries, its affiliates, or its other professionals.