Unpacking the Black Box: Why Explainable AI is Critical for Trust and Accountability

Transparency is one of those buzz words associated with AI. It can mean so many different things to different people that it is important to define what it covers, what it doesn’t cover, and why certain things are not covered.

Some define “explainable” AI as AI that is transparent, which is why these two terms often are used synonymously or jointly when describing responsible AI methods. But what does that actually mean? Transparency is by definition a state between opaque and clear, which means there are various levels of transparency. In data governance, transparency refers to open data practices, while in data architecture it means to fully understand how something works. In AI, it’s a bit of both where being transparent means the model is ethical and the company using or designing it has documented how they have used and designed the AI. Transparency can be characterized as:

• Interpretable, being able to explain or tell the meaning of how something was created, selected, improved, maintained, or discontinued, presented in understandable terms;
• Available, being able to access information in order to make decisions on the information;
• Comprehensive, being able to explain something from beginning to end, from different facets, and from various levels of depth;
• Accountability, ability to follow, trace, verify, or review the processes, actions, use of, and decisions throughout on a regular and repeatable cadence. 

The different layers in AI architectures can have different aspects of transparency, depending on what is being used in the AI architecture. There are traditional components in AI architecture such as databases, search engines, continuous integration and continuous delivery/deployment (CICD) tooling, but there are also components more common to AI architectures such as vector databases, prompt engineering, agents, and more. One common area in AI architectures is AI training, but this is an often conflated term for many different things in AI. While EBSCO does not train AI, it is good to understand the different aspects to how AI models work when reviewing documentation about AI.

The different ways AI models can be interacted with are:

• Information used to create word, input, or vector embeddings (traditionally what is meant by AI training)
• Information and/or embeddings from Information used for training transformer models (traditionally what is meant by AI training)
• Information and/or embeddings from Information used in model inferencing and predicting (traditionally what is meant when text, images, or steps in a process are generated by the AI, the direct output from AI training based on a prompt)
• Information and/or embeddings from Information used for grounding (not part of AI training, grounding is a supplemental data source for AI to consult after the generation step to increase its accuracy and context relevancy)
• Information and/or embeddings from Information used for fine-tuning (this is usually heuristic, or rule based guidance added to refine the AI models processing and output)
• Information and/or embeddings from Information used for inferring based on schema like an ontology or knowledge graphs (this is usually guidance given to an AI model to make it more relevant for a specific organizational need)
• Information and/or embeddings from Information used for feature or entity extraction (not usually part of training, but a form of text and data mining).
• Data parsing (when an AI is used to understand the structure and context of text or images. Not part of training.)
• Information retrieval (when an AI is used to search and retrieve information based on a query or prompt. Not part of training.)

From this list, we can see that AI has many elements, and while not all may be relevant to each use case or product, these are good elements to ask about or review provided documentation for, while assessing the transparency and the metrics transparency is needed for. 

Having explainable AI means the metrics used to assess the AI and the way the AI is used, designed, and maintained is also transparently presented to end-users. Explainable and transparent AI allows users to determine if the quality, ethics, security, privacy, environmental impact, and other metrics meet their expectations to be used and if the AI feature will meet their needs.

In order to be transparent, EBSCO always notes when AI is used on product. We also have product pages for each AI feature that explains which model is being used, why it was selected, what guardrails are in place for quality, security, and environmental impact, what quality criteria is used, and how the model is grounded or maintained. EBSCO does not train AI models, but if we were to do so, that also would be documented on a dedicated product page.

View the AI Transparency documents for AI Insights and Natural Language Search at EBSCO Connect. We are dedicated to providing an AI Transparency document for every AI feature we release, so make sure to visit EBSCO Connect for updates.