Hicham Badri’s Post

I think it is not too much to say that LoRA adapters have been one of the most important inventions to enable LLM finetuning for people with limited GPU resources like myself. This is also evident by the countless number of variants it inspired. As someone who has used LoRA countless times for various projects I always wanted to do my own minimal implementation in PyTorch . The most elegant way to do it in my opinion is via PyTorch Parametrizations which allow you to define operations for weights which are then executed before each forward pass.   I am now happy to share the first minimal version of this: pytora https://lnkd.in/dGRP-hyv   I also wrote a short medium article giving a brief introduction for how it works https://lnkd.in/dkUgzU95   I hope this is useful to people looking for a vanilla PyTorch implementation of LoRA!

I updated my LLMs repo with a 5th way to implement multihead attention using PyTorch's torch.nn.MultiheadAttention class. • It's 3.5x slower than scaled_dot_product_attention (which uses flash attention v2 in PyTorch 2.2, and flash attention v1 in PyTorch 2.1). • And it's even 1.2x slower than my own from-scratch implementation. So, maybe don't use that one. Link to the code: https://lnkd.in/gcKY-yMd

#PyTorch schedule up, Chainguard presenting on eliminating known vulns in PyTorch images. Cool lineup, I see stuff on PyTorch for hieroglyphics (cool) and supply chain security from #redhat among many others. #python #ai #datascience #conferences

Now we were talking NeRFs on steroids! This is getting better day by day! #nerf #smerf #gaussiansplatting #photogrammetry #realtimerendering #ai #vr #xr #360video

Just stumbled upon the most exhaustive LLM (Large Language Model) course on GitHub! This comprehensive resource includes articles, roadmaps, Colab notebooks, and various learning materials, equipping you to become an expert in the field. 🚀 Topics covered: ➡ The LLM architecture ➡ Building an instruction dataset ➡ Pre-training models ➡ Supervised fine-tuning ➡ Reinforcement Learning from Human Feedback ➡ Evaluation ➡ Quantization ➡ Inference optimization Repo : https://lnkd.in/g-42Yjgu

Video Lectures on CUDA: https://lnkd.in/eaxYNCJV The supplementary material for the lecture series, titled "Programming Massively Parallel Processors: A Hands-on Approach," provides a thorough guide for both beginners and advanced learners interested in CUDA programming and PyTorch integration. Spanning various topics, the lectures cover profiling and integrating CUDA kernels in PyTorch, an introduction to compute and memory architecture, advanced quantization, CUDA performance checklists, and building production-ready CUDA libraries. Each lecture includes videos, notebooks, slides, and code available in designated folders, ensuring comprehensive learning resources. Notable lectures feature speakers such as Mark Saroufim, Jeremy Howard, Thomas Viehmann, and Andreas Koepf, with dates ranging from January to May 2024. Key sessions include optimizing PyTorch optimizers, GPU collective communication with NCCL, and working with fused kernels. These materials are designed to equip learners with practical skills and deep understanding, enabling effective use of CUDA and PyTorch for enhanced programming capabilities and performance optimization. The supplementary material for the lecture series, based on the "Programming Massively Parallel Processors: A Hands-on Approach" book, offers a comprehensive guide for both beginners and advanced learners interested in CUDA programming and PyTorch integration. Below is a detailed breakdown of the lecture content: Lecture 1: Profiling and Integrating CUDA Kernels in PyTorch Lecture 2: Recap Ch. 1-3 from the PMPP book Lecture 3: Getting Started With CUDA Lecture 4: Intro to Compute and Memory Architecture Lecture 5: Going Further with CUDA for Python Programmers Lecture 6: Optimizing PyTorch Optimizers Lecture 7: Advanced Quantization Lecture 8: CUDA Performance Checklist Lecture 9: Reductions ...

Read this article by Bob Chesebrough to discover the rules of fancy slicing and broadcasting with NumPy and PyTorch. Follow along to understand how to execute these techniques for great performance benefits with code readability and maintainability. https://intel.ly/3TaCjG3 #ArtificialIntelligence #MachineLearning #oneAPI

just reinstall Pytorch using pip... to make Cuda available.

🚀 Join us for Round II of DSPy: Advanced RAG! 🚀 Following our first sesh on DSPy: Advanced Prompt Engineering, we're diving deeper with DSPy: Advanced RAG! 🌟 That is, Declarative Self-improving Language Programs, pythonically, for RAG. ⛓️ In other words, join us learn how to automatically optimize a LangChain RAG pipeline! See you there! RSVP: https://bit.ly/dspyRAG

Check out what's new in PyTorch Documentation. Featuring 3 new tutorials and PyTorch 2.2 documentation + tutorials! #pytorch Check out here : https://lnkd.in/dehHw3Y2