Ingy Y.’s Post

"Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction" Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng, Liwei Wang GPT series and other autoregressive (AR) large language models (LLMs) have exhibited unprecedented abilities in generalization and versatility, and despite limitations like hallucinations and token-limits, are considered state-of-the-art approaches towards generalized language models. In addition, large AR models in computer vision have also shown promising advances, with models such as VQGAN and DALL-E showcasing unprecedented potential for customized image generation, although their latency still significantly lags in comparison with diffusion-only models. All of these models base their output on the paradigm of next-token prediction. To address the issue of scalable image generation, the authors of this research present Visual AutoRegressive modeling (VAR), a new framework that redefines AR learning on images as coarse-to-fine "next-scale prediction" or "next-resolution prediction". This method improves AR transformers by allowing them to learn visual distribution faster with better generalization, and for the first time making them surpass diffusion transformers for the task of image generation. Experimental results demonstrate the potential of VAR models in scaling and zero-shot generalization, and they surpass the Diffusion Transformer (DiT) in several cases such as image quality, inference speed, data efficiency, and scalability. This is a very exciting step towards the development of AR models, as for the first time GPT-style AR models are able to surpass strong diffusion models in image synthesis, and offer a whole range of possibilities for model scaling. #llm #gpt #diffusion #imagegeneration #researchpaper https://lnkd.in/ghRwAcKN

𝐓𝐡𝐞 𝐄𝐫𝐚 𝐨𝐟 1-𝐛𝐢𝐭 𝐋𝐋𝐌𝐬 𝐏𝐫𝐨𝐛𝐥𝐞𝐦𝐬 𝐰𝐢𝐭𝐡 𝐟𝐮𝐥𝐥-𝐩𝐫𝐞𝐜𝐢𝐬𝐢𝐨𝐧 𝐋𝐋𝐌𝐬 Large Language Models (LLMs) have achieved remarkable results in natural language processing, but this comes at ever-increasing model size. The size of LLMs poses issues for real-world deployment. 𝐐𝐮𝐚𝐧𝐭𝐢𝐳𝐚𝐭𝐢𝐨𝐧 𝐢𝐬 𝐚 𝐬𝐮𝐛-𝐨𝐩𝐭𝐢𝐦𝐚𝐥 𝐬𝐨𝐥𝐮𝐭𝐢𝐨𝐧 Post-training quantization is a popular technique used to make LLMs smaller and more efficient for inference. It compresses the model by reducing the precision of parameters. Post-training quantization, even though popular in the industry, is not the perfect solution. The process of compressing the model can sometimes hurt the LLM performance. 𝐓𝐡𝐞 𝐬𝐨𝐥𝐮𝐭𝐢𝐨𝐧 𝐢𝐬 1-𝐛𝐢𝐭 𝐋𝐋𝐌𝐬 1-bit model architectures, such as BitNet are promising to reduce the cost of LLMs without any impact on the performance. 1-bit LLMs are as efficient as full precision LLMs and also suitable for resource-constrained platforms like edge and mobile devices. 𝐁𝐢𝐭𝐍𝐞𝐭 𝐛1.58 𝐋𝐋𝐌 𝐀𝐫𝐜𝐡𝐢𝐭𝐞𝐜𝐭𝐮𝐫𝐞 BitNet b1.58 is based on BitNet architecture and restricts its weights to -1, 0 or 1. The original BitNet architecture uses just -1 and 1 while BitNet 1.58b uses 0 in addition to -1 and 1. The inclusion of '0' allows for feature filtering within the model. This means it can selectively ignore certain features, boosting its performance compared to pure 1-bit LLMs. 1-𝐛𝐢𝐭 𝐋𝐋𝐌𝐬 𝐯𝐢𝐝𝐞𝐨 𝐭𝐮𝐭𝐨𝐫𝐢𝐚𝐥 𝐥𝐢𝐧𝐤 - https://lnkd.in/gsR_SncH #llms #generativeai #ai #1bitllms #nlproc #deeplearning #microsoft

Building generative AI apps is largely about trying to figure out whether one can accomplish key downstream tasks by simply engineering the heck out of a prompt into a huge pretrained model, or implementing some document/content retrieval to augment the prompt with key supporting information, or maybe incorporate fine-tuning for particular qualitative aspects, but the reality is it's often some blend of all three. I posted a couple months ago about a paper concluding that much of AI's emergent capability (i.e. the cool stuff!) [https://lnkd.in/e67rEkGZ] is due to in-context learning! So, in-context learning is crucial and that's why we see leading models like GPT4 and Llama 2 releasing longer and longer context windows, going from 4k tokens to 32k tokens and beyond. Last month NVIDIA AI researchers released a paper- https://lnkd.in/g4EgU7KG - that dug into how long context models perform relative to shorter context alternatives, but also how combining a long context window with retrieval impacts performance on key tasks. I'll do a subsequent post here soon on some of my own performance benchmarking of prompt engineered GPT4 vs GPT4 + retrieval vs fine-tuned GPT3.5 Turbo (spoiler: my early returns confirm the research which is cool). The key takeaway from this paper for me is that retrieval + long context can produce better results than than a small context window + retrieval, even when fed with the same top 5 chunks of context! That last part implies that long context LLMs have a more effective attention mechanism for the first 4k tokens than their shorter context brethren. Long context is also expensive (GPT4 32k is twice as expensive as GPT4 4k, and if you're running your own hardware for inference than I definitely don't need to tell you this 😃 ) - so there is a real cost/benefit analysis that should go into deciding on the architecture.

Deductive Friday Morning Coffee Reasoning: "What will happen to GPT-{n} once LLMs contribute much of the language found online? We find that use of model-generated content in training causes irreversible defects in the resulting models, where tails of the original content distribution disappear. We refer to this effect as model collapse and show that it can occur in Variational Autoencoders, Gaussian Mixture Models and LLMs." - https://lnkd.in/d-idr5fT Technological advancements in software development require substantial effort and cannot emerge from thin air. Promoting the widespread notion among AI-[PoP]ulists that AI will replace developers essentially acknowledges contentment with current achievements and implies a diminished interest in further technological progress.

Exciting News from NVIDIA and Georgia Tech! Researchers have introduced an innovative framework called RankRAG, designed to significantly enhance the capabilities of large language models (LLMs) in retrieval-augmented generation (RAG) tasks. RankRAG uniquely instruction-tunes a single LLM to perform both context ranking and answer generation, making it a game-changer in the field. This framework expands on existing instruction-tuning datasets by incorporating context-rich question-answering, retrieval-augmented QA, and ranking datasets. The goal is to improve the LLM's ability to filter out irrelevant contexts during both retrieval and generation phases. RankRAG introduces a specialized task focused on identifying relevant contexts or passages for given questions, structured for ranking but framed as regular question-answering with instructions. During inference, the LLM reranks retrieved contexts before generating answers based on the refined top-k contexts. This versatile approach can be applied to a wide range of knowledge-intensive NLP tasks, offering a unified solution for enhancing RAG performance across diverse domains. Read more about this groundbreaking development - https://lnkd.in/gM3veuze #NVIDIA #AI #MachineLearning #LLM #RankRAG #Innovation #TechNews

#GenerativeAI #AI #Learning Using #LLM for #embedding #retrieval and #textGeneration Contextual AI explores #GRIT (Generative Representational #InstructionTuning) 🔥 #performance in both generative and embedding tasks. 🤯 

Can we use an LLM for embedding retrieval and text generation at the same time? New research from Contextual AI explores this in generative representational instruction tuning (GRIT), achieving state-of-the-art performance in both generative and embedding tasks. 🤯 The paper also open source GritLM7-B & GritLM-8x7B, which can generate embeddings and text! 🔥 𝗜𝗺𝗽𝗹𝗲𝗺𝗲𝗻𝘁𝗮𝘁𝗶𝗼𝗻 1️⃣ Select pre-trained LLM, e.g. Mistral or Mixtral and adapt the model to use different attention mechanism 2️⃣ Create a mixed dataset with instruction (Tülu) and retrieval tasks (E5) 3️⃣ Trains LLM on the dataset using contrastive loss for embedding samples and Language Modelling Loss for instruction samples. Separation is achieved by different “BOS” tokens, e.g. <|embed|>. 𝗜𝗻𝘀𝗶𝗴𝗵𝘁𝘀 🏆 GritLM 7B scores #1 on MTEB with 66.8 and keeps a MMLU score of 57.6 🧠 Uses bidirectional attention (BERT) for embedding and causal attention (GPT) for generation 🤝 Matches the performance of embedding-only models 🐌 Embedding creation should be slower compared to S-BERT models 🔍 RAG can be sped up by caching the query or key-value states of the attention. Only works with single document retrieval (top 1). 🥇 Can be used for reranking too 🤗 Models on Hugging Face Arxiv: https://lnkd.in/e_63Y5nT Github: https://lnkd.in/enNw--pc Models: https://lnkd.in/ezzrk7nd Huge Congrats to Niklas Muennighoff, Douwe Kiela and the whole Contextual AI team on this release and research!🤗

Hi everyone! In this post, I want to talk about what commonsense reasoning is. Commonsense reasoning is the ability to use general knowledge and common sense to make inferences and decisions that are not explicitly stated or given. For example, if you see a picture of a person holding an umbrella, you can infer that it is raining or that the person is expecting rain. Commonsense reasoning is essential for humans to understand and interact with the world. Commonsense reasoning is also a challenging problem for machines, especially for natural language understanding. Natural language is often ambiguous, incomplete, or implicit, and requires commonsense knowledge and reasoning to resolve. For example, if you read a sentence like “He put out the candle with his fingers”, you can use your commonsense knowledge to understand that he extinguished the candle by pinching the wick, and not by dipping his fingers in the wax. Achieving commonsense reasoning in ML is an active area of research that involves various tasks, techniques, and resources. 1. Knowledge-based methods: These methods use structured knowledge bases or ontologies that store facts and relations about the world, such as ConceptNet or Cyc. These methods can query or reason over the knowledge bases using logical rules or inference engines to answer commonsense questions or generate commonsense explanations. 2. Learning-based methods: These methods use machine learning models, such as neural networks or transformers, that learn from large amounts of natural language data, such as text or speech. These methods can leverage pre-trained language models, such as BERT or GPT-3, that capture some aspects of commonsense knowledge and reasoning from the data. These methods can also fine-tune or adapt the models to specific commonsense tasks or domains using supervised or unsupervised learning techniques. 3. Hybrid methods: These methods combine knowledge-based and learning-based methods to leverage the strengths of both approaches. For example, some methods use knowledge graphs to augment or guide the learning models, while some methods use learning models to enrich or update the knowledge bases. I hope this post has given you some useful information on commonsense reasoning in ML. If you have any questions or comments, please feel free to share them below. Thank you for reading! #softwareengineer #datascientist #machinelearningengineer #artificialintelligence

It is good to have a single model for both embedding and text generation!