Tobias Zwingmann’s Post

I’m curious with the use case here. Where there are clearly defined “right” answers (“what’s my holiday allowance?”, “what’s my remaining holiday?”), would we not be better off transforming the question using NLP in to a direct API call to guarantee correctness and only move to probability if we aren’t aware of a concrete data source? We literally have the right data and are then introducing probability, this seems retrograde. I full support the intent of the article and RAG approach, cleaner, accurate, localised data is the only way enterprises can adopt at scale, I just get concerned that we see AI/GenAI as the solution when there are often simpler ways from A to B! Am I missing the point, or are we trying to solve everything with the current fashionable tool?

This is a great article with a very interesting internal use case example of RAG and how it all hinges on good data quality - especially when subjects as important as HR documents that could affect an employee’s career and life choices #geico #geicoproduct #data #aiml #dataquality

While AI—driven by large language models (LLMs)—has emerged as a groundbreaking innovation for streamlining workflows, its current limitations are becoming more apparent, including inaccurate responses and weaknesses in logical and mathematical reasoning. To address these challenges, Wolfram Research has developed a suite of tools and technologies to enhance the capabilities of LLMs. Wolfram’s technology stack increases the productivity of AI applications in multiple enterprise environments. By leveraging Wolfram’s extensive experience in computational intelligence and data curation, organizations can overcome LLM limitations to achieve greater accuracy and efficiency in AI-driven workflows. https://lnkd.in/daz38qb4 #machinelearning #SoftwareDevelopment #ArtificialIntelligence

It is the era of AI Agents ! There is considerable reason to believe that GPT-4omni is an AI Agent more than a Language Model. GPT-4o leverages an AI architecture that connects an LLM with various Machine Learning Models and operates coherently as a software architecture/ platform over just a simple language model. The renowned, Andrew Ng made a compelling presentation on AI Agents. He abstracts the fundamental function of an AI Agent in the steps of Self Reflection, Planning, Tool-use, and Multi-Agent Collaboration. 1. Self Reflection: Self reflection is like hitting back space after you've typed something and iteratively going through a feedback loop of Thought - Refinement - Expression until you are satisfied.  Check out this paper on Self-Refine (https://lnkd.in/gsTZFDGj) 2. Planning: Most human inputs are broken or verbose. Capturing the intent from such human essentially means to breakdown the human input into its intermediary reasoning. And solving for such intermediaries using tools: These papers on Chain-of-thought prompting (https://lnkd.in/g4BHDKck) and HuggingGPT helps understand this ( https://lnkd.in/gw-wk6FZ) 3. Tool-use: For a Language Model to pick up Multi-modal inputs and reason on it requires it to be capable of interfacing with other Machine Learning and Deep Learning tools. This is achieved seamlessly through a middle ware that is configured using Language (model descriptions). The way "3x+1 = 4" was solved in GPT-4o announcement video is the first example in the paper MM-REACT (https://lnkd.in/gUWCkVAH). Similarly a try on advanced analytics on GPT-4o calls function from scikit-learn to solve for the uploaded datasets. This potentially means that GPT-4o is an Agent and the future of LLM based architectures are moving steadily and surely into AI Agents. (at least till AGI arrives) We could soon see most computerized systems to become AI Agents based solutions. For instance engaging announcements where Governments are understaffed, A Point of Sales Machine help you decide your Subway meal, or a calendar that helps you organize yourself over a month. 

The future of data integration is here! See how AI, ML, and NLP are reshaping the landscape, reducing human error, and paving the way for a more efficient and dynamic data ecosystem. #DataIntegration

The future of data integration is here! See how AI, ML, and NLP are reshaping the landscape, reducing human error, and paving the way for a more efficient and dynamic data ecosystem. #DataIntegration

Synthetic data is the only route to better models. But you have to do it well— "If you don't do this well, you don't get much more than you started with." - Dario Amodei https://lnkd.in/dkhJTz3n Xuhui Zhou et al. pointed out that "synthetic data, which is artificially generated rather than collected from real-world sources, may not accurately represent the nuances and complexities of human values and preferences." https://lnkd.in/d_KnuN7t This is why synthetic data generation is not just a task, but an information retrieval problem. To make synthetic data truly representative of the real world, grounding examples act as anchors, guiding large language models (LLMs) to generate data that faithfully mirrors real-world patterns, distributions, and relationships. To solve this retrieval challenge, we built The Librarian - a Composite Neural Information Retrieval service that intelligently retrieves relevant real-world examples from multiple sources for the same task. This allows the grounding of a synthetic dataset from multiple real-world sources, providing LLMs with highly pertinent and representative examples that serve as a strong grounding for data generation. By combining state-of-the-art synthetic data generation techniques with real-world grounding from The Librarian, we are leveraging the best of both worlds to create more accurate and representative AI models. https://dria.co/whitepaper

The Art of Chunking: Boosting AI Performance in RAG Architectures The Key to Effective AI-Driven Retrieval Smart people are lazy. They find the most efficient ways to solve complex problems, minimizing effort while maximizing results. In Generative AI applications, this efficiency is achieved through chunking. Just like breaking a book into chapters makes it easier to read, chunking divides significant texts into smaller, manageable parts, making them easier to process and understand. Before exploring the mechanics of chunking, it’s essential to understand the broader framework in which this technique operates: Retrieval-Augmented Generation or RAG. What is RAG? What is Retrieval Augmented Generation Retrieval-augmented generation (RAG) is an approach that integrates retrieval mechanisms with large language models (LLM models). It enhances AI capabilities using retrieved documents to generate more accurate and contextually enriched responses. Introducing Chunking What is chunking Chunking is breaking down large pieces of text into smaller, more manageable chunks. This process has two main phases: Data Preparation: Reliable data sources are segmented into chunked documents and stored in a database. The database can be a vector store if you generate embeddings within the chunks. Retrieval: When a user asks a question, the system searches through the document chunks using vector search, full-text search, or a combination of both. This process identifies and retrieves the chunks most relevant to the user’s query. Why Chunking is Crucial in RAG Architectures Chunking is absolutely essential in RAG architectures because it is the first element that decides the accuracy of your Gen AI application. Chunks should be small to increase accuracy: Chunking enables the system to index and search through smaller text segments, increasing the accuracy of finding relevant documents. When a query is made, the system can quickly pinpoint the most pertinent chunks, improving the precision of the retrieval process. Chunks should be big to enhance Contextual Generation: Not chunks should be small. By working with smaller chunks, the generative model can better understand and utilize the context provided by each segment. This results in more coherent and contextually accurate responses, as the model can draw on specific, relevant information rather than sifting through a large, undivided document. Scalability and Performance: Chunking allows for more scalable and efficient processing of large datasets. It reduces the computational load by breaking down the data into manageable pieces, which can be processed in parallel, enhancing the overall performance of the RAG system. However, scalability should be ensured by Chunking is a technological necessity and a strategic approach to ensuring robust, efficient, and scalable RAG systems. It enhances retrieval accuracy, processing efficiency, and resource utilization, playing a crucial role in the su...

The future of data integration is here! See how AI, ML, and NLP are reshaping the landscape, reducing human error, and paving the way for a more efficient and dynamic data ecosystem. #DataIntegration #Data

Retrieval Augmented Generation (RAG) is transforming how AI systems fetch answers, offering more reliable and accurate responses for businesses. By integrating context-specific data with large language models (LLMs), RAG reduces errors and enhances relevance, making it a game-changer for customer service, research, and more. As we address AI challenges, RAG is poised to become a crucial tool in real-world applications.   #genai #aiinnovation #rag #futuretech #customerservice #aiaccuracy