Olimpiu P.’s Post

LLMs as Compilers This work proposes a think-and-execute framework to decompose the reasoning process in language models. This helps to improve algorithmic reasoning in LLMs. - It first THINKS to discover a task-level logic to solve a task and express logic in pseudocode - It then EXECUTES to simulate the execution of the code with language models Apparently, the process of discovering task-level logic behind a task improves performance on algorithmic reasoning tasks. It also outperforms instance-specific reasoning approaches like chain-of-thought and program-of-thoughts. Pseudocode understanding and generation is powerful and this paper shows how it can improve reasoning in LLMs. This is fascinating as pseudocode is one of my favorite approaches to solving really complex problems and build complex programs. Similar to the paper I shared yesterday on visualization-as-thoughts, I think this trend continues where we borrow ideas from how humans solve problems to improve LLMs and how we interact with them. Paper: https://lnkd.in/eTNJtVWv -- You can also track my weekly summary of LLM papers and research developments here: https://lnkd.in/e6ajg945

LLMs as Compilers Simulating Pseudocode Execution Improves Algorithmic Reasoning in Language Models This work proposes a think-and-execute framework to decompose the reasoning process in language models. This helps to improve algorithmic reasoning in LLMs. - It first THINKS to discover a task-level logic to solve a task and express logic in pseudocode - It then EXECUTES to simulate the execution of the code with language models Apparently, the process of discovering task-level logic behind a task improves performance on algorithmic reasoning tasks. It also outperforms instance-specific reasoning approaches like chain-of-thought and program-of-thoughts. Pseudocode understanding and generation is powerful and this paper shows how it can improve reasoning in LLMs. This is fascinating as pseudocode is one of my favorite approaches to solving really complex problems and build complex programs. This throws ideas from how humans solve problems to improve LLMs and how we interact with them. Paper :https://lnkd.in/gdPAYVsE

Most of the work done in tools supporting a language consists in manipulating the AST. In this article, we are going to see how to perform transformation and processing on the Abstract Syntax Tree through model-to-model transformations. Read more: https://lnkd.in/dV9ic5Ti #oldbutgold #strumentablog #compiler

Most of the work done in tools supporting a language consists in manipulating the AST. In this article, we are going to see how to perform transformation and processing on the Abstract Syntax Tree through model-to-model transformations. Read more: https://lnkd.in/d26FazSk #oldbutgold #strumentablog #compiler

There are many LLMs in the market. GPT4, Claude, Gemini, Mistral, etc. But which LLM suits your needs the best and how to choose it? LLM benchmarks are often confusing. As a user which benchmark score should you be concerned about while choosing the right LLM? Here I am curating a list of popular(not exhaustive) benchmarks and what they aim to measure. 1. SQuAD (Stanford Question Answering Dataset): Tests reading comprehension and answering questions based on given passages. 2. RACE: Evaluate understanding of passages from real-world scenarios through multiple-choice questions. 3. HellaSwag: Tests common sense by predicting what could happen next in a situation. 4. LAMBADA: Checks if a model can predict the right final word in a passage by understanding the context. 5. BoolQ: Yes/no questions to be answered based on short passages. Tests query understanding. 6. MultiRC: Requires answering multiple choice questions by reasoning over multiple passages. 7. ARC: Multiple-choice reasoning questions requiring gathering information from a science database. 8. CommonsenseQA: Tests commonsense reasoning through multiple-choice questions about everyday scenarios. 9. OpenBookQA: Science multiple-choice questions to be answered using an open book of facts. 10. GSM8K: Tests advanced math word problem solving needing multi-step reasoning. 11. CodeXGLUE: Evaluates code understanding and generation abilities across programming languages. 12. APPS: Assesses program understanding skills for Python code. 13. HumanEval: Tests coding abilities by having the model write functional programs. 14. HumanEval-X: An extension of HumanEval with more challenging coding tasks. 15. PIQA: Questions about diagrams and situations to test physical reasoning abilities. 16. VCR: Tests understanding of situations depicted in images through question-answering. 17. NLVR2: Determines if a statement accurately describes a set of images. 18. MMLU: Combines text with images/videos to test multimodal language and vision skills. So choose your LLM by looking at their score on your desirable benchmark.

A Journey Through the World of Algorithms: The post provides an overview of algorithms, their importance in problem-solving and efficiency improvement. It discusses different methods of specifying algorithms, such as natural language, pseudocode, flowcharts, and programming language code. The post also covers the fundamentals of algorithm analysis, Big-O notation, and various algorithm design techniques. Additionally, it presents examples of well-known algorithms, including divide and conquer, dynamic programming, greedy algorithms, and brute force algorithms. #generalprogramming #algorithms #datastructures https://lnkd.in/gxpZXbeS

ggml is a machine learning (ML) library written in C and C++ with a focus on Transformer inference. The project is open-source and is being actively developed by a growing community. ggml is similar to ML libraries such as PyTorch and TensorFlow, though it is still in it's early stages of development and some of it's fundamentals are still changing rapidly. Strongly recommended read. People who are coming from high level abstractions like PyTorch and Tensorflow, might face some challenges to understand the core concepts of ggml, but in short this is the base for Ollama, llama.cpp, Jan etc. Go read it! https://lnkd.in/eZVenPiG

Gemini 1.5 Release 🎯 Key Takeaways 🔍 Needle in a haystack benchmark is when there's a hidden message randomly somewhere in the input and the LLM has to extract it. - 1M token context length (8x GPT4 Turbo). 10M was tested successfully in research. - Needle in a haystack benchmark > 99.7% across all modalities (text, audio, video). Beats GPT 4 Turbo! - > 70% for 100 needles. Same as a needle in a haystack, except there are 100 needles, and the model has to retrieve them all in one go. Why are these benchmarks relevant? LLMs are capable of in-context learning, whereby they can use the prompts you give for information and instructions, allowing them to learn in real time. The benchmarks show Gemini 1.5 is far more capable than GPT4 at extracting relevant information for in-context learning. 📚In an effort to further showcase the ability for in-context learning, Gemini 1.5 is evaluated in the MTOB (Machine Translation One Book). The setup is a translation between Kalamang and English. Kalamang has fewer than 200 speakers and virtually no online presence. Gemini is given a book describing how to translate English <-> Kalamang. Gemini shows superior performance to both GPT4 and Claude 2.1 and almost rivals human translation for English to Kalamang. Note that GPT4 and Claude can't even fit the entire book in the context since they are limited to 128k and 200k tokens. 💡Applications: Prior to this, you had to heavily rely upon LLM's knowledge of the training dataset and "guide" it to output what you wanted. Now you don't have to worry about any of that. Let's take code generation for data frames as an example. Current renditions heavily rely on documented uses of the Pandas library on multiple internet sources. But what about a library like https://pola.rs/? It provides improvements over Pandas but has a significantly smaller user base and, therefore, not nearly as many documented use cases. Well, now that may not be an issue. Give the LLM the polar.rs documentation from the codebase and let it rip. ⚠️ Caveats: The pricing and response time for approaches remain unclear, so this new paradigm may favour use cases where the user doesn't require a fast response but can wait for their request to be processed and then notified. Particularly considering the 100 needles haystack evaluation, it may be a good idea to chain 10 questions in one request rather than 1 question per request since the bulk of the cost will be in the "context" tokens, which remain static for each request.

This is Amazing…4bit LLM Variant When Recently I was looking and exploring the Code base LLMs model quantisation models….search engine provide me different links and after number off efforts what I found is on The Hugging Face page for mlx-community/starcoder2-15b-4bit presents a model converted to MLX format from the original bigcode/starcoder2-15b. This model is part of a project aimed at advancing and democratizing artificial intelligence through open source and open science. The model is trained for text generation, leveraging transformers and safetensors within the MLX framework. It was originally trained on a dataset named bigcode/the-stack-v2-train-full-ids, indicating its broad applicability in generating code across various programming languages. Key Features: 🔸Model Size: It has 1.31 billion parameters, indicating its complexity and potential for understanding and generating intricate patterns in data. 🔸 Tensor Type: Utilizes FP16 and U32 tensor types for operations, optimizing for performance and memory usage. 🔸 Usage: Demonstrated through a Python example, where the model and tokenizer are loaded for generating text in response to a prompt. This illustrates the model's practical application in generating code or textual responses within a given context. 🔸 Inference API: The model supports loading on an Inference API, making it accessible for serverless applications and facilitating easy integration into various projects. Evaluation Results: The model showcases strong performance across several benchmarks: ◾️ pass@1 on CruxEval-I: 48.1% ◾️ pass@1 on DS-1000: 33.8% ◾️ Accuracy on GSM8K (PAL): 65.1% ◾️ pass@1 on HumanEval+ and HumanEval: 37.8% and 46.3%, respectively ◾️ edit-smiliarity on RepoBench-v1.1: 74.08% These results highlight the model's proficiency in understanding and generating code, making it a valuable tool for developers and researchers interested in leveraging AI for coding tasks and beyond in less memory.

My lastest post on using open source large language models using Ollama. In a few steps and less than 75 lines of code I show how to build a Streamlit chat application where open source models or proprietary models can be switched in and out with just a couple of changes. You can read it here https://lnkd.in/g3GpqYxS or here https://www.billparker.ai/