vickie li’s Post

PyTorch Internals! A comprehensive guide to how you can understand the PyTorch codebase & contribute to it. Here's the link: https://lnkd.in/dnDsJTQG

PyTorch gets sophisticated with every release. It is the biggest enabler in democratizing AI.

PyTorch 2.0 succeeds its predecessor, PyTorch 1.x, with improved performance and efficiency. Here’s a rundown of these new features and capabilities: Backward Compatibility: - Code, model and API compatibility with PyTorch 1.x - Maintained compatibility with libraries and dependencies Fusing Operations: - Fuses multiple operations into a single GPU kernel - Reduces kernel launching overhead Reducing Type Conversion Overhead: - Identifies and minimizes unnecessary type conversion code - Avoids redundancies Reusing Buffers: - Reduced memory allocation - Lower memory consumption during model training Read on to learn about all the improvements PyTorch 2.0 brings and how they work: https://bit.ly/47QsWR2

PyTorch 2.3 is here 😎🔥 PyTorch 2.3 offers support for user-defined Triton kernels in torch.compile, allowing for users to migrate their own Triton kernels from eager without experiencing performance regressions or graph breaks. Details: https://hubs.la/Q02tY8PB0

This is a win for #sustainability. In effect, ML applications may better leverage GPU capabilities, saving more time and energy 🚀

PyTorch 2.3 is now released. I'm delighted to see the many improvements, such as the ability to include Triton kernels directly in torch.compile() optimized code. To my mind, this will become a game changer for users that want to get the best performance from their system. I'm comparing this to inline assembly in traditional programming languages -- you want to use it sparingly and benefit from the higher-level abstractions wherever possible, but if you find a massively performance critical bottleneck in your program that can't be expressed efficiently (or at all, in some instances), this is a game changer! Other cool new features include: * tensor parallel API: express tensor-level parallelism natively to train large models such as LLMs more efficiently; * structured sparsity support. to take advantage of support for 2:4 sparsity in GPUs. * torch.export improvements for support of dynamic shapes; * asynchronous checkpoint generation to continue training while checkpoints are being generated, essentially offloading much of the checkpointing cost; * support for weight-only quantization on CPUs to improve performance for CPU-based inference. Check out the official announcement below, and give the new release a try! We hope you'll love the many improvements that the open source community and the PyTorch team have created. #PyTorch #AcceleratedAI #accelerators #LLMs #MetaAI #Meta

I think it is not too much to say that LoRA adapters have been one of the most important inventions to enable LLM finetuning for people with limited GPU resources like myself. This is also evident by the countless number of variants it inspired. As someone who has used LoRA countless times for various projects I always wanted to do my own minimal implementation in PyTorch . The most elegant way to do it in my opinion is via PyTorch Parametrizations which allow you to define operations for weights which are then executed before each forward pass.   I am now happy to share the first minimal version of this: pytora https://lnkd.in/dGRP-hyv   I also wrote a short medium article giving a brief introduction for how it works https://lnkd.in/dkUgzU95   I hope this is useful to people looking for a vanilla PyTorch implementation of LoRA!

Code for Number Recognition Using TensorFlow....

I updated my LLMs repo with a 5th way to implement multihead attention using PyTorch's torch.nn.MultiheadAttention class. • It's 3.5x slower than scaled_dot_product_attention (which uses flash attention v2 in PyTorch 2.2, and flash attention v1 in PyTorch 2.1). • And it's even 1.2x slower than my own from-scratch implementation. So, maybe don't use that one. Link to the code: https://lnkd.in/gcKY-yMd

New post on DebuggerCafe - Fine-Tuning FCOS using PyTorch https://lnkd.in/g6Cd9JHK #ObjectDetection #PyTorch #FCOS #ComputerVision #DeepLearning #FineTuning #DeepLearningSmokeDetection