Jose Lasa’s Post

Scale3's LangTrace.ai SDK now supports native open telemetry(o11y) tracing for LlamaIndex This allows developers to utilize o11y tracing to ship, debug, and evaluate AI applications more efficiently across various observability tools such as SigNoz, Grafana Labs Labs, Datadog, and honeycomb.io 🔍 Why is o11y tracing crucial? With open telemetry (a CNCF project), engineering teams gain a standard data model for telemetry that supports high cardinality. This is vital as it provides flexibility, prevents vendor lock-in, and helps in quicker debugging and insights, especially important in systems utilizing LLMs and VectorDBs. 👨💻 Meet Langtrace.ai  Langtrace.ai from Scale3 helps you gain confidence with LLMs before taking your application to production. If you are using OpenAI's APIs or vector DB's for any of your products, this could help with: **Tracking the responses made by the LLM **Evaluating and measuring the accuracy of responses **Identify bottlenecks. Best part - It's full Open Source and has native Open Telemetry support. https://langtrace.ai/ https://lnkd.in/grYfWsbP We’re thrilled to roll out native o11y support for LlamaIndex, making it easier for developers to harness the power of AI while maintaining transparency and control over their applications. Interested in o11y tracing for LLM applications or want to contribute to Langtrace? Let’s connect! 🚀 https://langtrace.ai/ # OpenTelemetry #ai

If you are using LlamaIndex to build AI applications, We just added support for native open telemetry(o11y) tracing for LlamaIndex in the Langtrace SDK. This enables you to ship traces, debug and evaluate not just to Langtrace but to also to all the other popular observability tools like SigNoz, Grafana Labs, Datadog, honeycomb.io etc. that support the visualization of o11y traces and spans. What is LlamaIndex? In simple terms, LlamaIndex is a framework for building AI applications. LlamaIndex has clean interfaces and constructs for - Data connections and ingestion - Data indexing - Querying And LlamaIndex has support for the popular LLMs and VectorDBs so you can swap in and out of different models and vector stores with ease without worrying about a code migration. Why o11y tracing matters? Open telemetry is a CNCF project that aims to have a standard data model for telemetry data. The open telemetry spans and traces are designed for high cardinality that can be ingested and visualized in any number of o11y supported observability solutions. This provides optionality for engineering teams and prevents vendor lock in. Teams can visualize the same data in different tools to debug and gain insights better and fix issues faster. With more products starting to leverage the incredible power of LLMs and VectorDBs to build novel features, we are moving from a world of mostly deterministic, well defined systems that can be reasoned through logic to non deterministic LLM inference endpoints that are primarily black boxes. In order to effectively debug, reason and fix issues around these new systems, engineering teams need access to deep non-intrusive tracing and high cardinality metrics from the AI framework layers to vectorDBs down to the LLM layers. What is Langtrace? This is the problem we are solving with Langtrace, a fully open source o11y compatible SDK for tracing any LLM stack. With Langtrace, you can not only trace the calls made at the framework layer, but also tap into vectorDBs and LLMs. We have support for the most popular LLMs, VectorDBs and Frameworks. Today, we are happy to announce native o11y support for tracing LlamaIndex. More details on how to set it up and get started can be found in the link on the first comment of this post. If you are curious about o11y tracing for LLM applications or would like to contribute to the project, reach out to me.

Hey Connections! Following up on my previous post about Retrieval-Augmented Generation (RAG), today lets dive into its architecture and workflow. 🌟 As discussed earlier, RAG is made up of two main components: the Retriever and the Generator. The Retriever hunts down the right documents based on your input, while the Generator crafts responses using those documents and your questions. They work together seamlessly to make RAG function like a charm! 👐 𝗛𝗼𝘄 𝗥𝗔𝗚 𝗪𝗼𝗿𝗸𝘀: 1. User Query: The process starts with a user submitting a query. 2. Retrieval Process: - The Retriever scans a large corpus of documents that you’ve provided. 📚 - It converts both the query and documents into vector embeddings (numerical representation of data along with its meaning for efficient similarity search) using models like all-MiniLM-L6 v2 or similar from the Hugging Face leaderboard(https://lnkd.in/gGYusNws). - Using similarity search techniques, it retrieves the top-k relevant documents with respect to the query. 3. Generator Process: - These top-k documents are fed into the Generator. - The Generator, using models like GPT-3, Llama2-7B, Mistral7B, etc., integrates the context from these documents with the user’s query. You can use hosted models(GPT-3.5) via their APIs or have a local LLM(for more privacy) installed on your system. - It then generates a response with its Gen-AI architecture that’s both relevant and well-structured. 4. Final Response: The user receives a well-crafted answer based on the most relevant information from the documents. ✅ 𝗣𝗿𝗼 𝗧𝗶𝗽: In case of repetitive usage or large datasets, it is recommended integrating a persistent vector database like Pinecone or ChromaDB. This way, you store the embeddings there as a one-time process(unless you want to change your data) and not every time you run a session. Next time a user queries something, you create real-time embeddings only for the query and perform a similarity search within the vector database. This saves a lot of time. ⏳ RAG’s combination of retrieval and generation makes it a powerful tool for many applications, from customer support to research. Stay tuned for the next post with more technical toppings to perform hands-on! Feel free to share your thoughts or ask any questions in the comments. Let’s keep the conversation going! 😊 #AI #RAG #MachineLearning #LLM #NLP #Innovation #TechTrends #ArtificialIntelligence #DataScience

This will set you up for successfully operationalizing GenAI apps. So, What Are Your Expectations From A Well-Operationalized GenAI Application? A successful GenAI application seamlessly integrates generative models into production workflows, driving performance, accuracy, and efficiency. This is a complex undertaking, demanding careful attention to three fundamental prerequisites: 1. Secure Data Pipeline Design: ⇰ Identify and map data sources for ingestion and preprocessing. ⇰ Leverage vector databases like Pinecone or Weaviate for efficient data retrieval. ⇰ Ensure data security and privacy compliance throughout the pipeline. 2. Strategic Model Selection & Architecture: ⇰Choose models based on specific application needs and performance metrics. ⇰Decide trade-offs between speed, accuracy, and computational cost. ⇰Utilize tools like MTEB and TruthfulQA for benchmarking models and understanding their limitations in addition to provider's benchmarks. ⇰ Choose architecture like prompt engineering, RAG, fine-tuning, or agentic systems. 3. Stable Monitoring & Governance Framework: ⇰Establish a robust monitoring system to track model performance and identify potential issues. ⇰Implement governance frameworks to promote ethical AI development and mitigate risks. ⇰Log and track model parameters, performance metrics, and data usage for transparency and accountability. Additional Considerations for Building Production-Ready GenAI Applications: ⇰Managing Model Ambiguity: - Understand the inherent stochastic nature of generative models and design workflows that can handle output variations. - Employ prompt engineering techniques to guide models toward desired outcomes and improve consistency. - Explore architecture choices that balance accuracy and consistency, such as Retrieval Augmented Generation (RAG), which combines LLM generation with information retrieval. ⇰Selecting the Right Model: -Choose models based on your specific needs for tasks like embedding generation, query completion, and content creation. -Consider computational cost, latency, and benchmark performance when making model choices. ⇰Optimizing Prompts: -Use Mlflow Prompt Engineering UI or Azure Prompt Flow to iterate and refine prompts ⇰LLMOps Orchestration: - Leverage tools like Langchain, LlamaIndex, haystack, and Semantic Kernel to seamlessly integrate GenAI models into production pipelines. ⇰Deployment and Inference: -Design scalable architectures to ensure smooth performance and user experience. -Consider deployment strategies like GPU clusters or hosted models for efficient inference Deal with ethical concerns: -Bias and Fairness: Models trained on biased data can perpetuate harmful biases. -Misinformation and Manipulation: GenAI models can create convincing but inaccurate content, raising concerns about spreading misinformation and manipulation. What's your experience in dealing these challenges? Comment below. #GenAI #Operationalization #DigitalTransformation #AIEthics

Unlock the potential of your documents with Azure AI's innovative approach to managing cross-page tables. Here's how it simplifies data processing and analysis: 🌐📊 - Prepares documents for analysis, handling various formats like PDFs and images. - Extracts basic table info, identifying page numbers and span offsets. - Identifies mergeable table candidates across pages using smart algorithms. - Merges tables vertically or horizontally, maintaining original data semantics. - Enhances large language model (LLM) comprehension for improved data processing. - Streamlines complex document processing tasks, making data analysis more efficient. For more insights, visit: https://lnkd.in/dyH5D2Eg #AzureAI #DataArchitecture #DocumentIntelligence

DataStax updates tools for building gen AI applications: DataStax is updating its tools for building generative AI-based applications in an effort to ease and accelerate application development for enterprises, databases, and service providers. One of these tools is Langflow, which DataStax acquired in April. It is an open source, web-based no-code graphical user interface (GUI) that allows developers to visually prototype LangChain flows and iterate them to develop applications faster. To read this article in full, please click here #MachineLearning #ArtificialIntelligence #DrivenByData #BigData #DataAnalytics #ITDirector #AI #DataDriven

EasyOCR text extraction from images is complete for MittaAI. Efficient Resource Utilization: By leveraging on-demand spot instances, ensures the OCR processing is cost-effective and scalable. The instances only run when needed, automatically scaling up when there are tasks in the queue. This optimizes resource utilization and minimizes idle time. High-Quality OCR: EasyOCR, backed by GPU instances, provides fast and accurate text extraction from images. Rivaling the quality of the Google Cloud Vision API, this integration enables MittaAI to deliver high-quality OCR results efficiently. Seamless Integration with SlothAI Pipeline: The integration of EasyOCR into the aivision processor in the SlothAI pipeline framework allows for smooth integration with other pipeline operations. This includes combining OCR with object and scene detection, as well as processing extracted images from PDFs that lack or have incomplete OCR data. Enhanced Weaviate Vector Store Integration: With the recent integration of Weaviate vector store, the addition of EasyOCR opens up new possibilities. The system can extract, chunk, embed, and store vectors for texts from various formats and sizes efficiently. This unlocks powerful search and retrieval capabilities for textual data. Upcoming Features: Support for Anthropic's Claude model for scene and object extraction from images, auto DAG-pipeline splitting for large embeddings, vector projection to lower dimensions in Weaviate (and a special trick I have up my sleeve for adjusting the vector space), and auto-detection of mixed-mode PDFs will continue the improvement and expansion of MittaAI's capabilities.

#Topics Auto ELT with Rivery's Generative AI Copilot, Blueprint AI [ad_1] 181 zettabytes. That’s the amount of data expected to be created, captured, copied and consumed by 2025, according to Accenture. If raw data is the lifeblood, ELT (extract, load and transform) is the skeleton, giving structure to data for the businesses powering our data consumption needs. Moving data from one place to another, as trivial as it sounds, has always been a laborious process; adjusting for changes in structures, arranging connections to new data sources, and fitting them into existing flows. That process is getting face-lift with Generative AI and copilots, courtesy of Rivery. Rivery Blueprint is an AI-powered copilot designed to simplify and accelerate data integration at scale. “We’re here to make data integration simple and flexible for the changes that happen every week,” says Itamar Ben Hemo, CEO of Rivery. “Our vision is to democratize data integration, and this AI-powered copilot is a significant step towards that goal.” Blueprint’s technology can analyze and understand the structure of new data sources at the user’s request, generate YAML configuration files for seamless integration, and enable users to quickly tap into new data sources and integrate them into thei...

Actually, there is a a large unknown in the AI and API management debate. One side you have what I call the side of the "Developers Mindset", who doesn't really care about having an API gateway to validate requests from LLM to internal database/datastores/Apps, and prefer framework like Langchain/Langsmith. They will use API gateway features (AuthN, AuthZ, rate limiting, throttling, caching, analytics, role management etc) only when they will be easy to use and included in such framework. They don"t care about "APIs and API management" per se, they ingest everything they can in whatever format (text, files, scraping and APIs of course) with what LLM framework give to them. On the other side, you have the Architect Mindset, who loves well governed AI and love AI gateways features (Prompt security, validation, PII removal etc) on top of API gateways because all the policy (AuthN, AuthZ, rate limiting, throttling, caching, analytics, role management etc) is already in production the API Management/API gateway layer. Then AI and LLMs are just another type of API consumer, and LLM just another type of applications consuming APIs. For the architect side, AI MUST consume and be consumed only by APIs. Because it is the proper way to do it, when you think about it, and the teams are already in place and ready to manage it. there is not enugh AIOps/MLOps on the market to make an other IT department, managing AI governance and AI gateways competing with the existing API management team, and its Center of Enablement. Then, these 2 options collide. The Developer way, to make it work as long it works, and the architect way, that want to build AI apps on top on what is already structured and governed, via APIs.  I prefer the second option, but most of the time the first one wins. As Alberto Brandolini states "It is not the domain experts knowledge that goes in production, it is the assumption of the developers that goes into production" With the success of Langchain and other framework, we can consider these are the current defacto "Developer assumption". As long as they don't implement API-first AI, AI implementation may not happen "the proper API way".

Is the secret sauce behind LLaMA-3 a new training algorithm / model architecture? According to Meta's technical report, training a top LLM like LLaMA-3 is all about data quality: 1. 15T tokens of pretraining data 2. More code examples in pretraining data (for better reasoning capabilities) 3. Relentlessly studying the mixture of data sources 4. Algorithmic data quality filtering using LLMs 5. Heavy quality filtering of remaining training data post pre-training (for SFT/RLHF/DPO) Most technical innovations in Meta's write up are about how to better curate data for pre/post-training! This demonstrates how data curation is the most important aspect of modern AI. Some quotes from Meta's article: "Some of our biggest improvements in model quality came from carefully curating this data and performing multiple rounds of quality assurance on annotations provided by human annotators" “We found that previous generations of Llama are surprisingly good at identifying high-quality data, hence we used Llama 2 to generate the training data for the text-quality classifiers that are powering Llama 3.” Some of the data filters applied include: NSFW, semantic near duplicates, and low-quality text. With Cleanlab Studio, your team can automatically utilize such AI-based data curation to perform all of the same steps in a no-code interface that doesn't require any ML expertise. Use Cleanlab to obtain quality assurance on human annotators, data sources, and automatically find+fix data issues like: NSFW, semantic near duplicates, low-quality text, outliers, mis-annotated data, and more.