Meta for Developers’ Post

PyTorch has a new attention function for LLMs called FlexAttention that supports various attention variants. I just took it for a spin and compared it to other implementations of multi-head attention, and it seems to work great in combination with torch.compile. However, if you are not compiling your models (e.g., due to compatibility issues), it may be better not to use it for multi-head attention. The notebook with the benchmark code is available here: https://lnkd.in/gcKY-yMd

PyTorch has a new attention function for LLMs called FlexAttention that supports various attention variants.

PyTorch 2.3 is here 😎🔥 PyTorch 2.3 offers support for user-defined Triton kernels in torch.compile, allowing for users to migrate their own Triton kernels from eager without experiencing performance regressions or graph breaks. Details: https://hubs.la/Q02tY8PB0

This is a win for #sustainability. In effect, ML applications may better leverage GPU capabilities, saving more time and energy 🚀

PyTorch gets sophisticated with every release. It is the biggest enabler in democratizing AI.

Simple and built-in to PyTorch, TorchTune emphasizes components that are simple to use, comprehend, and expand. This demonstrates exacting criteria for demonstrating the accuracy of parts and recipes. Based on PyTorch, it simply functions on various hardware right out of the box....https://lnkd.in/gWVcG7fZ

Day 97 of #100DaysOfCode - learned how to train a multi-layer perceptron(NN) in TensorFlow - learned more on activation funcs, specifically ReLU - dived into code today and spent time understanding each step in depth

just reinstall Pytorch using pip... to make Cuda available.

PyTorch 2.3 is now released. I'm delighted to see the many improvements, such as the ability to include Triton kernels directly in torch.compile() optimized code. To my mind, this will become a game changer for users that want to get the best performance from their system. I'm comparing this to inline assembly in traditional programming languages -- you want to use it sparingly and benefit from the higher-level abstractions wherever possible, but if you find a massively performance critical bottleneck in your program that can't be expressed efficiently (or at all, in some instances), this is a game changer! Other cool new features include: * tensor parallel API: express tensor-level parallelism natively to train large models such as LLMs more efficiently; * structured sparsity support. to take advantage of support for 2:4 sparsity in GPUs. * torch.export improvements for support of dynamic shapes; * asynchronous checkpoint generation to continue training while checkpoints are being generated, essentially offloading much of the checkpointing cost; * support for weight-only quantization on CPUs to improve performance for CPU-based inference. Check out the official announcement below, and give the new release a try! We hope you'll love the many improvements that the open source community and the PyTorch team have created. #PyTorch #AcceleratedAI #accelerators #LLMs #MetaAI #Meta

PyTorch Internals! A comprehensive guide to how you can understand the PyTorch codebase & contribute to it. Here's the link: https://lnkd.in/dnDsJTQG