Stefano Fiorucci’s Post

🛠️ Adding external tools to LLMs/VLMs provides exciting opportunities for models to perform various tasks. A great example that I’d like to highlight today is the recently published paper — 'LLaVA-Plus: Learning to Use Tools for Creating Multimodal Agents.' The authors extended the capabilities of the LLaVa multimodal model by adding a set of different tools accessible by the main model. This turned out to be quite interesting. Now, user interaction with the model consists not just of a question (text + picture) and an answer, but of 4 steps: ✅ The person provides a task (X_q) and a picture (I_q). ✅ The assistant processes all the information and generates X_skill_use, a set of tools that is needed to accomplish the task (this can be empty). ✅ Once a particular tool is used, the X_result is fed back to the assistant as input. ✅ Aggregating all the available information, the model produces the final result. In the accompanying diagram you can see this interaction. Sequences, where loss is computed during training, are marked in green, i.e., the model learns to predict the set of tools and the final answer. As for the tools themselves, many options are added: generation via Stable Diffusion, OCR, segmentation via SAM, ControlNet, Pix2Pix, and many more. Everything from the model to the training code and a new dataset with instructions is publicly available. #llm #vlm #multimodalai #nlp #llava #computervision

📈 BERTScore - Evaluating Text Generation with BERT 📊 BERTScore leverages the pre-trained BERT model to score the semantic similarity between two pieces of text. It uses cosine similarity over the pooled output, cls token, and average of all tokens. This approach provides a more accurate representation of human judgment compared to traditional methods like ROUGE, METEOR and CIDEr. 🎯 BERTScore calculates precision and recall through a process known as token matching as below: ✅ Contextual Embeddings ✅ Cosine Similarity ✅ Token Matching for Precision and Recall ✅ Importance Weighting ✅ Baseline Rescaling ⚠️ Limitations and potential biases ➡️ It can be biased towards models that are more similar to its own underlying model. This is because the metrics can favor their own outputs and other outputs which are more similar to them. ➡️ The ability of reference-free metrics to evaluate other models is inherently limited by the qualities of their pseudo-references. If a system outputs a translation or a summary which is higher-quality than the pseudo-reference, it will be incorrectly penalized because it is different from the pseudo-reference, even though those differences are actually improvements. ➡️ BERTScore doesn't take into account the syntactic structure of the sentence. It can lead to incorrect evaluations in cases where the syntactic structure of the sentences is different but they convey the same meaning. ➡️ It might perform poorly on tasks that require understanding the context beyond the individual words, such as idiomatic expressions or cultural references 📌 For further insights, please refer to the complete article and tutorial: https://lnkd.in/eTUzYPuG 📌 Paper: https://lnkd.in/eF-FgK4r 📌 Github: https://lnkd.in/expsDDPZ #LLM #GenAI #BERTScore #NLP #MachineLearning #Research

Confused about what RAG—Retrieval Augmented Generation—actually does? In essence, RAG is a technique that supercharges text generation models by providing them with more contextually rich prompts. The mechanics involve two pivotal components: 📚 Step 1: Retrieve pertinent chunks of text from a vast pool of documents based on your initial query. 🔄 Step 2: Compile a prompt that incorporates the original context, the retrieved documents, and the user's query. This approach leverages the large context windows that advanced LLMs (like GPT-4 with 32k tokens and Claude with 100k tokens) possess. It fetches closely related documents to enhance the quality of the model's output. The result? More precise and contextually relevant answers. RAG is particularly useful in handling cases such as data falling beyond the model's cutoff date, or for augmenting domain-specific knowledge that the model hasn't previously encountered. So, can you envision other scenarios where RAG could be a game-changer? ------------------------------ Did I oversimplify too much? How would you explain it? Let me know if you have any analogies! #NLP #GenAI #Embeddings #DataScience #RAG #LanguageModels

I'm meeting with a client tomorrow morning who has some pretty specific—and somewhat lofty—ambitions. As I prepare a solution, I keep coming back to the 'Hyena Hierarchy' paper as a potential pathway. I've read this paper multiple times, and even after simplifying it as much as I can, I still have questions. I'll end this post with one particular question that's on my mind. For those unfamiliar with it, the paper explores the limitations of large Transformers, specifically the quadratic computational cost associated with their attention mechanisms. The authors introduce a new architecture known as the Hyena operator, designed to serve as a subquadratic drop-in replacement for attention. 🔑 Key Insights: 1️⃣ Hyena Operator: Defined by a recurrence of two efficient subquadratic primitives—an implicit long convolution and multiplicative element-wise gating of the input. 2️⃣ Performance: In tests involving sequences with hundreds of thousands of tokens, Hyena outperforms other operators by over 50% in accuracy. It sets a new state-of-the-art for dense-attention-free architectures on standard datasets like WikiText103 and The Pile. 3️⃣ Efficiency: Achieves up to a 100x speedup over standard attention implementations in PyTorch for sequence lengths of 64k. 4️⃣ Generalization: Hyena is designed to generalize existing subquadratic approaches by introducing a recurrence of gates and implicit long convolutions. 🤔 My Question: How do you think the Hyena operator's subquadratic computational complexity could impact the scalability of future language models? Could this architecture be generalized to other domains beyond NLP?" #NLP #MachineLearning #Transformers #HyenaOperator #ComputationalEfficiency #VirtualReality #MixedReality Michael Poli Stefano Massaroli Eric Nguyen Dan Fu Tri Dao Stephen Baccus Yoshua Bengio Stefano Ermon @christopher re

Fine-tine Idefics2 for image to json !! •HuggingFaceM4 introduced Idefics 2, a multimodal LLM that has quickly become a favorite in the open-source community. • The Idefics family of models has impressed me with its robust capabilities, and Idefics 2 is no exception. •The growing interest in vision-language models (VLMs) has been fueled by advancements in large language models and vision transformers. • With its vision capabilities, it excels at understanding scanned images, enhancing OCR, and extracting data from tables. •Despite being 8B in size, like #llama3, #idefics2 seems to fly under the radar amidst the usual PR frenzy. • Idefics2 attains state-of-the-art performance within its size category across diverse multimodal benchmarks, often rivaling models four times its size. Abs: https://lnkd.in/evscPxin Paper : https://lnkd.in/e7Zt-ZtY Blog : https://lnkd.in/exMjeG5v #OpenIA #hugginface #OCR #tesseract #json #computervision #nlp #LLM #cvpr #cvpr2024

I'm thrilled to share my latest project with you all: the RAG System! 📝💡 Imagine a large language model (LLM) that can access and understand information beyond its training data! I've been working on a project that does just that. LLMs like me (Gemini) can be amazing, but we sometimes lack specific details. This project uses a system called RAG ( Retriever-Augmenter-Generator) to bridge that gap. The result? More accurate and insightful answers based on both the LLM's inherent knowledge and external data! 🌟 Finding the answer: The system searches the paper (stored as a text file) for relevant information. Combining knowledge: The LLM use the retrieved information to understand the context and answer your question in an informative way. The outcome? More accurate and insightful answers based on both the LLM's knowledge and external data! This project explores how LLMs can be enhanced by accessing external sources. It's a step towards building even more powerful and versatile language models in the future! Huge thanks to Kanav Bansal and Innomatics Research Labs Team for their invaluable support and knowledge! 🙌 Check out the project on GitHub here , https://lnkd.in/g7j_fNQi and watch the demo video. #LLM #AI #NLP #MachineLearning #RAG #LeaveNoContextBehind

JINA EMBEDDINGS 2 - Open Source Text Embeddings for Long Documents 1️⃣ Text embedding models are powerful tools for representing text as fixed-sized vectors. 2️⃣ Most existing open-source models, especially those built on architectures like BERT, struggle to represent lengthy documents.  3️⃣ Jina Embeddings v2, an open-source text embedding model addresses this issue. 4️⃣ Jina Embeddings v2 is capable of encoding long documents of up to 8192 tokens. 5️⃣ Jina Embeddings v2 not only achieves state-of-the-art performance on MTEB benchmark. 6️⃣ Jina Embeddings v2 matches the performance of OpenAI’s proprietary text-embedding-ada-002 model. ➡️ Jina Embeddings v2 (base model) link: https://lnkd.in/gYv_Xnhq ➡️ Jina Embeddings v2 (small model) link: https://lnkd.in/gSektkWa ✔️ For complete details, refer the paper (paper link in the comments) #nlproc #nlp #deeplearning #datascience #ai #generativeai #embeddings

Open source embedding for long document. Strong research in sentence embedding called JiNA embedding with 8k seq length. Earlier we used use Bert sentence embedding but embedding paradigm are changing. Good part is performance is equal to open AI text embedding ada-002.