Using AI Agents to improve Relevancy and Accuracy of your Enterprise Content Solutions

Image generated with Microsoft Designer

Co-authors: Luigi Pichett, Pierre Feillet

In previous discussions, we explored the limitations and risks of designing Generative AI (GenAI) solutions that depend solely on a single Large Language Model (LLM) interaction. These systems typically rely on complex prompts containing detailed instructions and examples to guide the generation of the most accurate output possible. However, this single-step generation approach often results in inaccuracies, known as “hallucinations,” which produce outputs with unpredictable accuracy and inconsistent relevance. This is because LLMs are statistical models that predict the most probable sequence of tokens based on the input and their training data.

Transitioning to a Modular System Approach
To overcome these challenges, AI solutions are evolving toward a modular system design. Instead of relying solely on a single LLM, this approach integrates the LLM into a broader system to:

• Break Down Problems: Decompose complex tasks into simpler, manageable steps.
• Optimize Task Handling: Determine when the LLM can provide answers and when external tools or systems are more appropriate, such as accessing personalized data from a backend database.
• Map Resources: Identify the tools, functions, and data sources required for external assistance, along with their appropriate usage.
• Validate Outputs: Implement quality assurance steps to ensure reliable and accurate answers.

The recent advancements in LLM capabilities now allow them to break down complex prompts or problems into logical steps (referred to as “LLM chaining”). When combined with checks, loops, and validation steps in a modular system, this approach introduces what is known as the AI Agentic Approach.

What is AI Agent and Agentic approach?

AI Agent refers to autonomous agent— entity that is capable of perceiving its environment, making decisions, and taking actions to achieve specific objectives.

Unlike systems governed by rigid, hard-coded logic flows, AI agents leverage dynamic control logic to solve problems through a process of reasoning and acting, often referred to as the ReAct pattern. Their level of autonomy can be fine-tuned based on the complexity of the task and user preferences, such as the extent of reasoning or the number of fail/retry iterations the system is allowed to perform.

At its core, this approach enables the LLM to dynamically orchestrate tasks, incorporate validation steps, and adapt in real time, providing a flexible and robust mechanism for achieving desired outcomes in a wide range of scenarios.

Key features of AI agents:

• Autonomy: Operates independently of human input once deployed.
• Goal-Oriented: Works towards achieving predefined or evolving goals.
• Adaptability: Adjusts to new information or changing environments.
• Interactive: Engages with its surroundings or other agents to effect change.

Core technologies powering AI agentic systems include:

1. Reinforcement Learning (RL):Enables agents to learn optimal behaviors through trial and error. Example: AlphaGo learning to play Go by playing against itself.
2. Multi-Agent Reinforcement Learning (MARL): Extends RL to environments with multiple interacting agents. Example: AI teams competing in strategy games like StarCraft.
3. Planning and Search Algorithms: Utilize methods like A* or Monte Carlo Tree Search for decision-making. Example: Pathfinding in robotics.
4. Natural Language Processing (NLP): Enables understanding and generation of human language. Example: Virtual assistants like Siri or Alexa.
5. Knowledge Representation: Uses ontologies and knowledge graphs for reasoning and context-awareness. Example: AI reasoning about medical symptoms for diagnosis.

Applying an agentic approach to improve accuracy and reduce hallucination

Applying an agentic approach to design a system that is more accurate in its response generation. An “agentic” AI, in this sense, doesn’t just generate responses passively but actively engages in steps to verify, retrieve, and confirm information before and during response generation. These approaches are complementary to the different knowledge grounding techniques discussed before and in fact, we make use of those technique to make the answer more precise and accurate.

Here are some examples on how we can apply an agentic approach to Q&A solutions:

Incorporate Knowledge Retrieval with RAG or Similar Frameworks

• Use a Retrieval-Augmented Generation (RAG) or similar framework that actively pulls relevant, up-to-date information from a reliable corpus or database before generating responses.
• Set up a self-reflection loop, where the AI agent verifies the retrieved documents for accuracy or relevance to the user query. If the retriever doesn’t find relevant documents, the agent can either re-query or request clarification from the user.
• For example, if a user asks a complex question about recent events, the agent could query reliable news sources and verify the relevance of retrieved documents before generating a response.

Figure 1. RAG with Reflection Loop

Integrate Knowledge Graphs for Contextual and Factual Verification

• Use a Knowledge Graph to structure and verify entity relationships, which can help disambiguate terms and confirm factual correctness. Before finalizing a response, the AI agent can check the generated output against this graph to avoid contradictions and ensure grounding.
• For example, if the AI generates a response about “London,” it can check with the knowledge graph to confirm if it’s referring to the city in the UK or in Canada, based on context.

Implement Dynamic Fact-Checking Modules

• Add a fact-checking module that evaluates generated answers against external knowledge bases or online sources (if accessible). The agent can detect potential hallucinations or gaps in knowledge by flagging unverified statements and either correct them or present a disclaimer.
• This could involve integrating APIs that allow for real-time cross-checking with databases like Wikipedia, official databases, or specific domain sources (e.g., PubMed for medical questions).

Figure 2. Q & A with multiple agents to improve answer accuracy

Set Response Confidence Thresholds and Use Disclaimers

• Design the agent to assess its own confidence in the response based on the availability and quality of retrieved or verified information. If confidence is low, the agent can either flag it for clarification or provide a disclaimer, indicating the response might be speculative.
• For example, in answering a question about an emerging topic with little available data, the agent might say, “Based on current knowledge, here is the answer; however, this topic is evolving.”

Incorporate function calling to retrieve context specific information

• Design the agent to check if the question is related to information that’s context specific and dynamic in nature
• Instead of retrieving data from static knowledge bases, it can create a SQL query or API query to retrieve the necessary information and combine that together with knowledge from other sources to form a comprehensive response.

Summary

The above provides a further look into how enterprise can make use of AI Agents to build their own content management solution based on grounding the knowledge and cross-checking the answers against different accuracy and relevancy criteria. Using the technique above, we will be able to provide answers that are based on known facts, and by leveraging an agentic approach, we can further evaluate and provide additional feedback to improve the quality of the answers.

Pros:

1. Higher Accuracy: By incorporating active retrieval, clarification, and verification steps, the solution reduces the chances of providing incorrect information.
2. Reduced Hallucinations: Dynamic fact-checking and multi-step verification help catch potential errors.
3. Enhanced User Trust: Confidence-based responses and context-aware memory improve response reliability, fostering user confidence in the system’s accuracy.
4. Flexible Solution: The control logic flow to resolve a problem is not statically defined but generated by an AI Agent that “reasons” and whose level of autonomy can be adjusted.

Cons:

1. Incorporating one or more “fact-checking” or “feedback” agents into the agentic pipeline will increase the time it will take the system to response to the query.
2. Systematically evaluating the improvement in results across different approaches is challenging.

By implementing these techniques, enterprises can leverage Generative AI more effectively, improving decision-making and content management while minimizing risks.