I’m currently working on a startup (Reka.ai) where I am co-founder and chief scientist. Reka is an AI research and product company that builds state-of-the-art multimodal language models.
Before founding Reka.ai, I spent a wonderful 3.3 years at Google Research & Google Brain where I made contributions to many industry-defining LLMs such as PaLM, UL2, Flan-2 and Bard and multimodal models such as PaLI-X and ViT-22B. Notably, I was also the co-lead of modeling on PaLM-2 and PaLM-2 API.
During my time as a research scientist at Google, most of my published work revolved around Transformers, especially pertaining to efficiency, scaling and architecture research.
![What happened to BERT & T5? On Transformer Encoders, PrefixLM and Denoising Objectives](https://images.squarespace-cdn.com/content/v1/636628d6a9d980577a9faf5b/1721069203358-QT6Q8C8AA2WCUPRA37FN/image-asset.jpeg)
What happened to BERT & T5? On Transformer Encoders, PrefixLM and Denoising Objectives
A Blogpost series about Model Architectures Part 1: What happened to BERT and T5? Thoughts on Transformer Encoders, PrefixLM and Denoising objectives
![Training great LLMs entirely from ground up in the wilderness as a startup](https://images.squarespace-cdn.com/content/v1/636628d6a9d980577a9faf5b/1709663769227-EYG21H9BTX2CXT4VOJAM/Screenshot+2024-03-06+at+2.35.43+AM.png)
Training great LLMs entirely from ground up in the wilderness as a startup
Chronicles of training strong LLMs from scratch in the wild
![Leaving Google Brain](https://images.squarespace-cdn.com/content/v1/636628d6a9d980577a9faf5b/1680191976052-FG99EWL56VL6U90KGXL9/PXL_20230327_025947068.jpg)
![A New Open Source Flan 20B with UL2](https://images.squarespace-cdn.com/content/v1/636628d6a9d980577a9faf5b/1680192164341-PR8WGTUP0T5PYYA0299Z/Screenshot%2B2023-03-31%2Bat%2B12.01.32%2BAM.jpg)
![2022 in Review: Top language AI research papers + interesting papers to read](https://images.squarespace-cdn.com/content/v1/636628d6a9d980577a9faf5b/1718359186441-229Q6P1DNFE21MWH0101/image-asset.jpeg)
2022 in Review: Top language AI research papers + interesting papers to read
Here are some of the best language AI / NLP papers of 2022!
![On Emergence, Scaling and Inductive Bias](https://images.squarespace-cdn.com/content/v1/636628d6a9d980577a9faf5b/1680192279660-MJIMTAQ55IKF8TRJ2L3X/Screen%2BShot%2B2022-11-13%2Bat%2B3.05.52%2BAM.jpg)
On Emergence, Scaling and Inductive Bias
Some thoughts on emergent abilities and scaling language models.