Flamingo: a Visual Language Model for Few-Shot Learning
Authors:
Jean-Baptiste Alayrac,
Jeff Donahue,
Pauline Luc,
Antoine Miech,
Iain Barr,
Yana Hasson,
Karel Lenc,
Arthur Mensch,
Katie Millican,
Malcolm Reynolds,
Roman Ring,
Eliza Rutherford,
Serkan Cabi,
Tengda Han,
Zhitao Gong,
Sina Samangooei,
Marianne Monteiro,
Jacob Menick,
Sebastian Borgeaud,
Andrew Brock,
Aida Nematzadeh,
Sahand Sharifzadeh,
Mikolaj Binkowski,
Ricardo Barreira,
Oriol Vinyals
, et al. (2 additional authors not shown)
Abstract:
Building models that can be rapidly adapted to novel tasks using only a handful of annotated examples is an open challenge for multimodal machine learning research. We introduce Flamingo, a family of Visual Language Models (VLM) with this ability. We propose key architectural innovations to: (i) bridge powerful pretrained vision-only and language-only models, (ii) handle sequences of arbitrarily i…
▽ More
Building models that can be rapidly adapted to novel tasks using only a handful of annotated examples is an open challenge for multimodal machine learning research. We introduce Flamingo, a family of Visual Language Models (VLM) with this ability. We propose key architectural innovations to: (i) bridge powerful pretrained vision-only and language-only models, (ii) handle sequences of arbitrarily interleaved visual and textual data, and (iii) seamlessly ingest images or videos as inputs. Thanks to their flexibility, Flamingo models can be trained on large-scale multimodal web corpora containing arbitrarily interleaved text and images, which is key to endow them with in-context few-shot learning capabilities. We perform a thorough evaluation of our models, exploring and measuring their ability to rapidly adapt to a variety of image and video tasks. These include open-ended tasks such as visual question-answering, where the model is prompted with a question which it has to answer; captioning tasks, which evaluate the ability to describe a scene or an event; and close-ended tasks such as multiple-choice visual question-answering. For tasks lying anywhere on this spectrum, a single Flamingo model can achieve a new state of the art with few-shot learning, simply by prompting the model with task-specific examples. On numerous benchmarks, Flamingo outperforms models fine-tuned on thousands of times more task-specific data.
△ Less
Submitted 15 November, 2022; v1 submitted 29 April, 2022;
originally announced April 2022.
Using Unity to Help Solve Intelligence
Authors:
Tom Ward,
Andrew Bolt,
Nik Hemmings,
Simon Carter,
Manuel Sanchez,
Ricardo Barreira,
Seb Noury,
Keith Anderson,
Jay Lemmon,
Jonathan Coe,
Piotr Trochim,
Tom Handley,
Adrian Bolton
Abstract:
In the pursuit of artificial general intelligence, our most significant measurement of progress is an agent's ability to achieve goals in a wide range of environments. Existing platforms for constructing such environments are typically constrained by the technologies they are founded on, and are therefore only able to provide a subset of scenarios necessary to evaluate progress. To overcome these…
▽ More
In the pursuit of artificial general intelligence, our most significant measurement of progress is an agent's ability to achieve goals in a wide range of environments. Existing platforms for constructing such environments are typically constrained by the technologies they are founded on, and are therefore only able to provide a subset of scenarios necessary to evaluate progress. To overcome these shortcomings, we present our use of Unity, a widely recognized and comprehensive game engine, to create more diverse, complex, virtual simulations. We describe the concepts and components developed to simplify the authoring of these environments, intended for use predominantly in the field of reinforcement learning. We also introduce a practical approach to packaging and re-distributing environments in a way that attempts to improve the robustness and reproducibility of experiment results. To illustrate the versatility of our use of Unity compared to other solutions, we highlight environments already created using our approach from published papers. We hope that others can draw inspiration from how we adapted Unity to our needs, and anticipate increasingly varied and complex environments to emerge from our approach as familiarity grows.
△ Less
Submitted 18 November, 2020;
originally announced November 2020.
