In the following overview, we will detail how two of these models — a ~3 billion parameter on-device language model, and a larger server-based language model available with [Private Cloud Compute](https://security.apple.com/blog/private-cloud-compute/) and running on Apple silicon servers — have been built and adapted to perform specialized tasks efficiently, accurately, and responsibly. These two foundation models are part of a larger family of generative models created by Apple to support users and developers; this includes a coding model to build intelligence into Xcode, as well as a diffusion model to help users express themselves visually, for example, in the Messages app. We look forward to sharing more information soon on this broader set of models.

## Our Focus on Responsible AI Development
Apple Intelligence is designed with our core values at every step and built on a foundation of groundbreaking privacy innovations.

Additionally, we have created a set of Responsible AI principles to guide how we develop AI tools, as well as the models that underpin them:

1. Empower users with intelligent tools: We identify areas where AI can be used responsibly to create tools for addressing specific user needs. We respect how our users choose to use these tools to accomplish their goals.
2. Represent our users: We build deeply personal products with the goal of representing users around the globe authentically. We work continuously to avoid perpetuating stereotypes and systemic biases across our AI tools and models.
3. Design with care: We take precautions at every stage of our process, including design, model training, feature development, and quality evaluation to identify how our AI tools may be misused or lead to potential harm. We will continuously and proactively improve our AI tools with the help of user feedback.
4. Protect privacy: We protect our users' privacy with powerful on-device processing and groundbreaking infrastructure like Private Cloud Compute. We do not use our users' private personal data or user interactions when training our foundation models.

These principles are reflected throughout the architecture that enables Apple Intelligence, connects features and tools with specialized models, and scans inputs and outputs to provide each feature with the information needed to function responsibly.

In the remainder of this overview, we provide details on decisions such as: how we develop models that are highly capable, fast, and power-efficient; how we approach training these models; how our adapters are fine-tuned for specific user needs; and how we evaluate model performance for both helpfulness and unintended harm.

<Figure alt="A diagram that shows the training steps for the Apple foundation models." id="figure1" caption={<>Figure 1: Modeling overview for the Apple foundation models.</>} animate="fade-up">
 <Image src="https://mlr.cdn-apple.com/media/Fig1_Responsible_AI_8bf7727ab5.png" alt="Modeling overview" />
</Figure>

## Pre-Training
Our foundation models are trained on [Apple's AXLearn framework](https://github.com/apple/axlearn), an open-source project we released in 2023. It builds on top of JAX and XLA, and allows us to train the models with high efficiency and scalability on various training hardware and cloud platforms, including TPUs and both cloud and on-premise GPUs. We used a combination of data parallelism, tensor parallelism, sequence parallelism, and Fully Sharded Data Parallel (FSDP) to scale training along multiple dimensions such as data, model, and sequence length.

We train our foundation models on licensed data, including data selected to enhance specific features, as well as publicly available data collected by our web-crawler, AppleBot. Web publishers have [the option to opt out](https://support.apple.com/en-us/119829) of the use of their web content for Apple Intelligence training with a data usage control.

We never use our users’ private personal data or user interactions when training our foundation models, and we apply filters to remove personally identifiable information like social security and credit card numbers that are publicly available on the Internet. We also filter profanity and other low-quality content to prevent its inclusion in the training corpus. In addition to filtering, we perform data extraction, deduplication, and the application of a model-based classifier to identify high quality documents.

## Post-Training
We find that data quality is essential to model success, so we utilize a hybrid data strategy in our training pipeline, incorporating both human-annotated and synthetic data, and conduct thorough data curation and filtering procedures. We have developed two novel algorithms in post-training: (1) a rejection sampling fine-tuning algorithm with teacher committee, and (2) a reinforcement learning from human feedback (RLHF) algorithm with mirror descent policy optimization and a leave-one-out advantage estimator. We find that these two algorithms lead to significant improvement in the model’s instruction-following quality.

## Optimization
In addition to ensuring our generative models are highly capable, we have used a range of innovative techniques to optimize them on-device and on our private cloud for speed and efficiency. We have applied an extensive set of optimizations for both first token and extended token inference performance.

Both the on-device and server models use grouped-query-attention. We use shared input and output vocab embedding tables to reduce memory requirements and inference cost. These shared embedding tensors are mapped without duplications. The on-device model uses a vocab size of 49K, while the server model uses a vocab size of 100K, which includes additional language and technical tokens.

For on-device inference, we use low-bit palletization, a critical optimization technique that achieves the necessary memory, power, and performance requirements. To maintain model quality, we developed a new framework using LoRA adapters that incorporates a mixed 2-bit and 4-bit configuration strategy &mdash; averaging 3.5 bits-per-weight &mdash; to achieve the same accuracy as the uncompressed models.

Additionally, we use an interactive model latency and power analysis tool, [Talaria](/research/talaria), to better guide the bit rate selection for each operation. We also utilize activation quantization and embedding quantization, and have developed an approach to enable efficient Key-Value (KV) cache update on our neural engines.

With this set of optimizations, on iPhone 15 Pro we are able to reach time-to-first-token latency of about 0.6 millisecond per prompt token, and a generation rate of 30 tokens per second. Notably, this performance is attained before employing token speculation techniques, from which we see further enhancement on the token generation rate.

## Model Adaptation
Our foundation models are fine-tuned for users’ everyday activities, and can dynamically specialize themselves on-the-fly for the task at hand. We utilize adapters, small neural network modules that can be plugged into various layers of the pre-trained model, to fine-tune our models for specific tasks. For our models we adapt the attention matrices, the attention projection matrix, and the fully connected layers in the point-wise feedforward networks for a suitable set of the decoding layers of the transformer architecture.

By fine-tuning only the adapter layers, the original parameters of the base pre-trained model remain unchanged, preserving the general knowledge of the model while tailoring the adapter layers to support specific tasks.

<Figure alt="A diagram of the Apple Foundation Model and Adapters." id="figure2" caption={<>Figure 2: Adapters are small collections of model weights that are overlaid onto the common base foundation model. They can be dynamically loaded and swapped &mdash; giving the foundation model the ability to specialize itself on-the-fly for the task at hand. Apple Intelligence includes a broad set of adapters, each fine-tuned for a specific feature. It’s an efficient way to scale the capabilities of our foundation model.</>} className="overflow-visible" animate="fade-up">
 <Figure2 src="https://mlr.cdn-apple.com/video/Fig2_Adapters_9f087bc5c5.mp4" />
</Figure>

We represent the values of the adapter parameters using 16 bits, and for the ~3 billion parameter on-device model, the parameters for a rank 16 adapter typically require 10s of megabytes. The adapter models can be dynamically loaded, temporarily cached in memory, and swapped — giving our foundation model the ability to specialize itself on the fly for the task at hand while efficiently managing memory and guaranteeing the operating system's responsiveness.

To facilitate the training of the adapters, we created an efficient infrastructure that allows us to rapidly retrain, test, and deploy adapters when either the base model or the training data gets updated. The adapter parameters are initialized using the accuracy-recovery adapter introduced in the Optimization section.

## Performance and Evaluation
Our focus is on delivering generative models that can enable users to communicate, work, express themselves, and get things done across their Apple products. When benchmarking our models, we focus on human evaluation as we find that these results are highly correlated to user experience in our products. We conducted performance evaluations on both feature-specific adapters and the foundation models.

To illustrate our approach, we look at how we evaluated our adapter for summarization. As product requirements for summaries of emails and notifications differ in subtle but important ways, we fine-tune accuracy-recovery low-rank (LoRA) adapters on top of the palletized model to meet these specific requirements. Our training data is based on synthetic summaries generated from bigger server models, filtered by a rejection sampling strategy that keeps only the high quality summaries.

To evaluate the product-specific summarization, we use a set of 750 responses carefully sampled for each use case. These evaluation datasets emphasize a diverse set of inputs that our product features are likely to face in production, and include a stratified mixture of single and stacked documents of varying content types and lengths. As product features, it was important to evaluate performance against datasets that are representative of real use cases. We find that our models with adapters generate better summaries than a comparable model.

As part of responsible development, we identified and evaluated specific risks inherent to summarization. For example, summaries occasionally remove important nuance or other details in ways that are undesirable. However, we found that the summarization adapter did not amplify sensitive content in over 99% of targeted adversarial examples. We continue to adversarially probe to identify unknown harms and expand our evaluations to help guide further improvements.

<Figure 
 id="figure3" 
 caption={<>Figure 3: Ratio of "good" and "poor" responses for two summarization use cases relative to all responses. Summaries are classified as "good", "neutral", "poor" given the grader's scores across five dimensions. A result is classified as "good" if all of the dimensions are good (higher is better). A result is classified as "poor" if any of the dimensions are poor (lower is better). Our models with adapters generate better summaries than a comparable model.</>}
 animate="fade-up">
 <Figure3 title="Human Satisfaction Score on Summarization Feature Benchmark" />
</Figure>

In addition to evaluating feature specific performance powered by foundation models and adapters, we evaluate both the on-device and server-based models’ general capabilities. We utilize a comprehensive evaluation set of real-world prompts to test the general model capabilities. These prompts are diverse across different difficulty levels and cover major categories such as brainstorming, classification, closed question answering, coding, extraction, mathematical reasoning, open question answering, rewriting, safety, summarization, and writing.

We compare our models with both open-source models (Phi-3, Gemma, Mistral, DBRX) and commercial models of comparable size (GPT-3.5-Turbo, GPT-4-Turbo)<a href="#footnotes">1</a>. We find that our models are preferred by human graders over most comparable competitor models. On this benchmark, our on-device model, with ~3B parameters, outperforms larger models including Phi-3-mini, Mistral-7B, and Gemma-7B. Our server model compares favorably to DBRX-Instruct, Mixtral-8x22B, and GPT-3.5-Turbo while being highly efficient.

<Figure 
 id="figure4" 
 caption={<>Figure 4: Fraction of preferred responses in side-by-side evaluation of Apple's foundation model against comparable models. We find that our models are preferred by human graders.</>}
 animate="fade-up">
 <Figure4 title="Apple Foundation Model Human Evaluation" />
</Figure>

We use a set of diverse adversarial prompts to test the model performance on harmful content, sensitive topics, and factuality. We measure the violation rates of each model as evaluated by human graders on this evaluation set, with a lower number being desirable. Both the on-device and server models are robust when faced with adversarial prompts, achieving violation rates lower than open-source and commercial models.

<Figure 
 id="figure5" 
 caption={<>Figure 5: Fraction of violating responses for harmful content, sensitive topics, and factuality (lower is better). Our models are robust when faced with adversarial prompts.</>} 
 animate="fade-up">
 <Figure5 title="Human Evaluation of Output Harmfulness" />
</Figure>

Our models are preferred by human graders as safe and helpful over competitor models for these prompts. However, considering the broad capabilities of large language models, we understand the limitation of our safety benchmark. We are actively conducting both manual and automatic red-teaming with internal and external teams to continue evaluating our models' safety.

<Figure 
 id="figure6" 
 caption={<>Figure 6: Fraction of preferred responses in side-by-side evaluation of Apple's foundation model against comparable models on safety prompts. Human graders found our responses safer and more helpful.</>}
 animate="fade-up">
 <Figure6 title="Human Preference Evaluation on Safety Prompts" />
</Figure>

To further evaluate our models, we use the Instruction-Following Eval (IFEval) benchmark to compare their instruction-following capabilities with models of comparable size. The results suggest that both our on-device and server model follow detailed instructions better than the open-source and commercial models of comparable size.

<Figure 
 id="figure7" 
 caption={<>Figure 7: Instruction-following capability (measured with IFEval) for Apple's foundation models and models of comparable size (higher is better).</>}
 animate="fade-up">
 <Figure7 title="IFEval Benchmarks" />
</Figure>

We evaluate our models’ writing ability on our internal summarization and composition benchmarks, consisting of a variety of writing instructions. These results do not refer to our feature-specific adapter for summarization (seen in [Figure 3](#figure3)), nor do we have an adapter focused on composition.

<Figure 
 id="figure8" 
 caption={<>Figure 8: Writing ability on internal summarization and composition benchmarks (higher is better).</>}
 animate="fade-up">
 <Figure8 title="Writing Benchmarks" />
</Figure>

## Conclusion
The Apple foundation models and adapters introduced at WWDC24 underlie Apple Intelligence, the new personal intelligence system that is integrated deeply into iPhone, iPad, and Mac, and enables powerful capabilities across language, images, actions, and personal context. Our models have been created with the purpose of helping users do everyday activities across their Apple products, and developed responsibly at every stage and guided by Apple’s core values. We look forward to sharing more information soon on our broader family of generative models, including language, diffusion, and coding models.

{<h2 id="footnotes">Footnotes</h2>}
[1] We compared against the following model versions: gpt-3.5-turbo-0125, gpt-4-0125-preview, Phi-3-mini-4k-instruct, Mistral-7B-Instruct-v0.2, Mixtral-8x22B-Instruct-v0.1, Gemma-1.1-2B, and Gemma-1.1-7B. The open-source and Apple models are evaluated in bfloat16 precision.

At the 2024 <a href="https://developer.apple.com/wwdc24/" class="more">Worldwide Developers Conference</a>, we introduced Apple Intelligence, a personal intelligence system integrated deeply into iOS 18, iPadOS 18, and macOS Sequoia.
Apple Intelligence is comprised of multiple highly-capable generative models that are specialized for our users’ everyday tasks, and can adapt on the fly for their current activity. The foundation models built into Apple Intelligence have been fine-tuned for user experiences such as writing and refining text, prioritizing and summarizing notifications, creating playful images for conversations with family and friends, and taking in-app actions to simplify interactions across apps.

Introducing Apple’s On-Device and Server Foundation Models

Recent research has explored <a href="https://pubmed.ncbi.nlm.nih.gov/32706685/" target="_blank" aria-label="clinical monitoring, - Opens in a new window" class="icon icon-after icon-external" rel="noopener nofollow">clinical monitoring,</a> <a href="https://arxiv.org/abs/1802.02511" target="_blank" aria-label="cardiovascular events, - Opens in a new window" class="icon icon-after icon-external" rel="noopener nofollow">cardiovascular events,</a> and even clinical lab values from <a href="https://pubmed.ncbi.nlm.nih.gov/34031607/" target="_blank" aria-label="wearables data. - Opens in a new window" class="icon icon-after icon-external" rel="noopener nofollow">wearables data.</a> As adoption increases, wearables data may become crucial in public health applications like <a href="https://www.thelancet.com/journals/landig/article/PIIS2589-7500(19)30222-5/fulltext" target="_blank" aria-label="disease monitoring - Opens in a new window" class="icon icon-after icon-external" rel="noopener nofollow">disease monitoring</a> and the <a href="https://pubmed.ncbi.nlm.nih.gov/34711377/" target="_blank" aria-label="design of epidemiological studies. - Opens in a new window" class="icon icon-after icon-external" rel="noopener nofollow">design of epidemiological studies.</a>

Personalizing Health and Fitness with Hybrid Modeling

<div class="callout">

## Schedule

Stop by the Apple booth in the Arch Building, Exhibit Hall Level 4, booth #1905, from 10:30am - 6:30pm PST June 19 and 20; 10:00am - 3:00pm PST on June 21.

### Monday, June&nbsp;17
<ul class="links-stacked">
<li>ORAL & POSTER PRESENTATION</li>
 <li><a href=https://sites.google.com/view/elvm/home>Workshop on EFFICIENT LARGE VISION MODELS 2024</a></li>
 <li>8:00am PST - 12:35pm PST, Summit 420-422</li>
 <li><a href=https://machinelearning.apple.com/research/sam-clip>SAM-CLIP: Merging Vision Foundation Models towards Semantic and Spatial Understanding</a></li>
<li>Haoxiang Wang (University of Illinois Urbana-Champaign), Pavan Kumar Anasosalu Vasu, Fartash Faghri, Raviteja Vemulapalli, Mehrdad Farajtabar, Sachin Mehta, Mohammad Rastegari, Oncel Tuzel, Hadi Pour Ansari</li>
</ul>

<ul class="links-stacked">
<li>ORAL & POSTER PRESENTATION</li>
 <li><a href=https://image-matching-workshop.github.io>Workshop on Image Matching: Local Features & Beyond 2024</a></li>
 <li>1:00pm PST - 5:45pm PST, Summit 323</li>
 <li><a href=https://machinelearning.apple.com/research/affine-based-attention>Affine-based Deformable Attention and Selective Fusion for Semi-dense Matching</a></li>
<li>Hongkai Chen, Zixin Luo, Ray Tian, Aron Wang, Lei (VE) Zhou, Xuyang Bai, Mingmin Zhen, Tian Fang, Yanghai Tsin, David McKinnon, Long Quan (The Hong Kong University of Science of Technology)</li>
</ul>

### Tuesday, June&nbsp;18
<ul class="links-stacked">
<li>WORKSHOP</li>
<li><a href=https://www.latinxinai.org/cvpr-2024>LatinX in CV (LXCV) at CVPR 2024</a></li>
<li>8:30am PST - 6:00pm PST, Arch 203</li>
<li>Marcel Santos, Conor O'Brien and Angus Choi are representing Apple at the Latin X workshop events.</li>
</ul>

<ul class="links-stacked">
<li>WORKSHOP & MENTORING</li>
<li><a href=https://sites.google.com/view/wicv-cvpr-2024/>Women in Computer Vision (WiCV) at CVPR 2024</a></li>
<li>8:30am PST - 6:00pm PST, Arch 201</li>
<li>Lisa Ta, Tanya Glozman, Diane Zhu, and Syenny Syenny are representing Apple at the WiCV workshop events.</li>
</ul>


### Wednesday, June&nbsp;19
<ul class="links-stacked">
<li>POSTER</li>
<li><a href=https://machinelearning.apple.com/research/hugs>HUGS: Human Gaussian Splatting</a></li>
<li>10:30am PST - 12:00pm PST, #32, Poster Session 1 & Exhibit Hall (Arch 4A-E)</li>
<li>Muhammed Kocabas (Max Planck Institute for Intelligent Systems), Rick Chang, James Gabriel, Oncel Tuzel, Anurag Ranjan</li>
</ul>

<ul class="links-stacked">
<li>POSTER</li>
<li><a href=https://machinelearning.apple.com/research/human-motion-understanding>HumMUSS: Human Motion Understanding using State Space Models</a></li>
<li>10:30am PST - 12:00pm PST, #207, Poster Session 1 & Exhibit Hall (Arch 4A-E)</li>
<li>Arnab Mondal (McGill University), Stefano Alletto, Denis Tome’</li>
</ul>

<ul class="links-stacked">
<li>POSTER</li>
<li><a href=https://machinelearning.apple.com/research/direct-2-5>Direct2.5: Diverse Text-to-3D Generation via Multi-view 2.5D Diffusion</a></li>
<li>5:15pm PST - 6:45pm PST, #382, Poster Session 2 & Exhibit Hall (Arch 4A-E)</li>
<li>Yuanxun Lu (Nanjing University), Jingyang Zhang, Shiwei Li, Tian Fang, David McKinnon, Yanghai Tsin, Long Quan (The Hong Kong University of Science and Technology), Xun Cao (Nanjing University), Yao Yao (Nanjing University)</li>
</ul>

<ul class="links-stacked">
<li>POSTER</li>
<li><a href=https://machinelearning.apple.com/research/kpconvx>KPConvX: Modernizing Kernel Point Convolution with Kernel Attention</a></li>
<li>5:15pm PST - 6:45pm PST, #66, Poster Session 2 & Exhibition Hall (Arch 4A-E)</li>
<li>Hugues Thomas, Hubert Tsai, Tim Barfoot (University of Toronto), Jian (AIML) Zhang</li>
</ul>

<ul class="links-stacked">
<li>POSTER</li>
<li><a href=https://machinelearning.apple.com/research/efficient-diffusion-models>Efficient Diffusion Models without Attention</a></li>
<li>5:15pm PST - 6:45pm PST, #334, Poster Session 2 & Exhibit Hall (Arch 4A-E)</li>
<li>Jing Nathan Yan (Cornell University), Jiatao Gu, Alexander M. Rush (Cornell University)</li>
</ul>

### Thursday, June&nbsp;20
<ul class="links-stacked">
<li>POSTER</li>
<li><a href=https://machinelearning.apple.com/research/mobileclip>MobileCLIP: Fast Image-Text Models through Multi-Modal Reinforced Training</a></li>
<li>5:15pm PST - 6:45pm PST, #130, Poster Session 4 & Exhibit Hall (Arch 4A-E)</li>
<li>Pavan Kumar Anasosalu Vasu, Hadi Pour Ansari, Fartash Faghri, Raviteja Vemulapalli, Oncel Tuzel</li>
</ul>

### Friday, June&nbsp;21
<ul class="links-stacked">
<li>POSTER</li>
<li><a href=https://machinelearning.apple.com/research/facial-motion-synthesis>Probabilistic Speech-Driven 3D Facial Motion Synthesis: New Benchmarks, Methods, and Applications</a></li>
<li>5:15pm PST - 6:45pm PST, #302, Poster Session 6 & Exhibit Hall (Arch 4A-E)</li>
<li>Karren Yang, Anurag Ranjan, Rick Chang, Raviteja Vemulapalli, Oncel Tuzel</li>
</ul>



</div>

## Accepted Papers

[Affine-based Deformable Attention and Selective Fusion for Semi-dense Matching](https://machinelearning.apple.com/research/affine-based-attention)
*Hongkai Chen, Zixin Luo, Ray Tian, Aron Wang, Lei (VE) Zhou, Xuyang Bai, Mingmin Zhen, Tian Fang, Yanghai Tsin, David McKinnon, Long Quan (The Hong Kong University of Science of Technology)*

[Direct2.5: Diverse Text-to-3D Generation via Multi-view 2.5D Diffusion](https://machinelearning.apple.com/research/direct-2-5)
*Yuanxun Lu (Nanjing University), Jingyang Zhang, Shiwei Li, Tian Fang, David McKinnon, Yanghai Tsin, Long Quan (The Hong Kong University of Science and Technology), Xun Cao (Nanjing University), Yao Yao (Nanjing University)*

[KPConvX: Modernizing Kernel Point Convolution with Kernel Attention](https://machinelearning.apple.com/research/kpconvx)
*Hugues Thomas, Hubert Tsai, Tim Barfoot (University of Toronto), Jian (AIML) Zhang*

[SAM-CLIP: Merging Vision Foundation Models towards Semantic and Spatial Understanding](https://machinelearning.apple.com/research/sam-clip)
*Haoxiang Wang (University of Illinois Urbana-Champaign), Pavan Kumar Anasosalu Vasu, Fartash Faghri, Raviteja Vemulapalli, Mehrdad Farajtabar, Sachin Mehta, Mohammad Rastegari, Oncel Tuzel, Hadi Pour Ansari*

[MobileCLIP: Fast Image-Text Models through Multi-Modal Reinforced Training](https://machinelearning.apple.com/research/mobileclip)
*Pavan Kumar Anasosalu Vasu, Hadi Pour Ansari, Fartash Faghri, Raviteja Vemulapalli, Oncel Tuzel*

[Probabilistic Speech-Driven 3D Facial Motion Synthesis: New Benchmarks, Methods, and Applications](https://machinelearning.apple.com/research/facial-motion-synthesis)
*Karren Yang, Anurag Ranjan, Rick Chang, Raviteja Vemulapalli, Oncel Tuzel*

[Efficient Diffusion Models without Attention](https://machinelearning.apple.com/research/efficient-diffusion-models)
*Jing Nathan Yan (Cornell University), Jiatao Gu, Alexander M. Rush (Cornell University)*

[HUGS: Human Gaussian Splatting](https://machinelearning.apple.com/research/hugs)
*Muhammed Kocabas (Max Planck Institute for Intelligent Systems), Rick Chang, James Gabriel, Oncel Tuzel, Anurag Ranjan*

[HumMUSS: Human Motion Understanding using State Space Models](https://machinelearning.apple.com/research/human-motion-understanding)
*Arnab Mondal (McGill University), Stefano Alletto, Denis Tome’*
 


## Demos

**MobileCLIP: Real-Time Image-Text Models**

Wednesday, June 19 - Friday June 21, during exhibition hours

Demo shows zero-shot scene classification running real-time on an iPhone. Since these models align image and text modalities, they can perform zero-shot image classification or image-text/text-image retrieval at blazing speeds. The app showcases the research work, “MobileCLIP: Fast Image-Text Models through Multi-Modal Reinforced Training” being presented at the same venue. The app is built by David Koski, Megan Maher Welsh with contributions from Hugues Thomas, Mouli Sivapurapu, Jian Zhang.

**Flow Composer for Apple ML**

Wednesday, June 19 - Friday June 21, during exhibition hours

Demo shows usage of Apple ML features on Mac Book Pro and iPad, which leverages several technologies such as Vision, CoreML, Core Graphics.


## Acknowledgements

 Alex Schwing and Philipp Kraehenbuehl are Senior Area Chairs for CVPR 2024.

Alex Toshev, Oncel Tuzel, Mehrdad Farajtabar, Hadi Pour Ansari and Fartash Faghri are Area Chairs for CVPR 2024.

Fartash Faghri, Jason Ren, Jianrui Cai, Jiajia Luo, Jierui Lin, Liangchen Song, Or Dinari, Pavan Kumar Anasosalu Vasu, 
Peter Fu, Raviteja Vemulapalli, Haotian Zhang, Hong-You Chen, Wen Shi, Yongzhi Su, Yuyan Li, Trevine Oorloff, Yongxi Lu and Jeff Lai are reviewers for CVPR 2024.

Anshul Shah is a co-organizer for the workshop [Learning from Procedural Videos and Language: What is Next?](https://sites.google.com/view/lpvl-cvpr2024)

Jeff Bigham is a co-organizer for the [VizWiz Grand Challenge Workshop](https://vizwiz.org/workshops/2024-vizwiz-grand-challenge-workshop/)

Pau Rodriguez Lopez is a co-organizer for the [Workshop on Continual Learning in Computer Vision](https://sites.google.com/view/clvision2024/overview)

Jeff Lai has a PhD dissertation selected for Doctoral Consortium.

Apple is sponsoring the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), which is taking place in person from June 17 to 21 in Seattle, Washington. CVPR is the annual computer vision event comprising the main conference and several co-located workshops and short courses. Below is the schedule of our sponsored workshops and events at CVPR 2024.

IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 2024

<div class="callout">

## Schedule

Below is the schedule of Apple sponsored workshops and events at CHI 2024. Stop by the Apple booth in Kamehameha 1 Hall, Booth 100, from 5:30pm - 7:30pm HST on May 13; 10:00am - 6:00pm HST on May 14; 10:00am - 5:30pm HST on May 15; 10:00am - 12:00pm HST on May 16.

### Tuesday, May&nbsp;14
<ul class="links-stacked">
<li>LUNCHEON</li>
<li>CHI 2024 Appreciation Luncheon</a></li>
<li>12:20 PM - 2:00 PM HST, Room 304A</li>
<li>Alex Lim and Erdrin Azemi will be representing Apple at the CHI 2024 Appreciation Luncheon.</li>
</ul>

<ul class="links-stacked">
<li>WORKSHOP</li>
<li>Journey to Apple - Career Panel with Human Factors Engineers</a></li>
<li>12:30 PM - 1:05 PM HST, Room TBD</li>
<li>Spencer Jefferies, Ava Rezvani Lannin, Stacie Lafko and Alex Lim are speakers.</li>
</ul>

<ul class="links-stacked">
<li>ORAL</li>
<li><a href=https://machinelearning.apple.com/research/model-compression-in-practice>Model Compression in Practice: Lessons Learned from Practitioners Creating On-device Machine Learning Experiences</a></li>
<li>4:00 PM - 4:15 PM HST, 316A</li>
<li>Fred Hohman, Mary Beth Kery, Donghao Ren, Dominik Moritz</li>
</ul>

<ul class="links-stacked">
<li>ORAL</li>
<li><a href=https://machinelearning.apple.com/research/talaria>Talaria: Interactively Optimizing Machine Learning Models for Efficient Inference</a></li>
<li>4:15 PM - 4:30 PM HST, 316A</li>
<li>Fred Hohman, Chaoqun Wang, Jinmook Lee, Jochen Görtler, Dominik Moritz, Jeffrey Bigham, Zhile Ren, Cecile Foret, Qi Shan, Xiaoyi Zhang</li>
</ul>

<ul class="links-stacked">
<li>ORAL</li>
<li><a href=https://machinelearning.apple.com/research/axnav>AXNav: Replaying Accessibility Tests from Natural Language</a></li>
<li>4:30 PM - 4:45 PM HST, 314</li>
<li>Maryam Taeb (Florida State University), Amanda Swearngin, Eldon Schoop, Regina Cheng, Yue Jiang, Jeff Nichols</li>
</ul>


</div>

## Accepted Papers

[AXNav: Replaying Accessibility Tests from Natural Language](https://machinelearning.apple.com/research/axnav)

*Maryam Taeb (Florida State University), Amanda Swearngin, Eldon Schoop, Regina Cheng, Yue Jiang, Jeff Nicholsl*
 
[Model Compression in Practice: Lessons Learned from Practitioners Creating On-device Machine Learning Experiences](https://machinelearning.apple.com/research/model-compression-in-practice)

*Fred Hohman, Mary Beth Kery, Donghao Ren, Dominik Moritz*

[Talaria: Interactively Optimizing Machine Learning Models for Efficient Inference](https://machinelearning.apple.com/research/talaria)

*Fred Hohman, Chaoqun Wang, Jinmook Lee, Jochen Görtler, Dominik Moritz, Jeffrey Bigham, Zhile Ren, Cecile Foret, Qi Shan, Xiaoyi Zhang*



## Acknowledgements

Amanda Swearngin, Titus Barik and Eldon Schoop are Associate Chairs of a program or organizing committee for CHI 2024.

Xiaoying Pu, Moses Namara, Amanda Swearngin, Xiaoyi Zhang, Regina Cheng, Mary Beth Kery, Eldon Schoop, Jeremy Warner and Agni Kumar are reviewers for CHI 2024.

Apple is sponsoring the ACM Human-Computer Interaction Conference (CHI), which is taking place in person from May 11 to May 16, 2024 in Honolulu, Hawai'i.

ACM Human-Computer Interaction conference (CHI) 2024

Apple machine learning teams are engaged in state of the art research in machine learning and artificial intelligence. Learn about the latest advancements.

Overview

Voice activity detection (VAD) is a critical component in various applications such as speech recognition, speaker identification, and hands-free communication systems. With the increasing demand for personalized and context-aware technologies, the need for effective personalized VAD systems has become paramount. In this paper, we present a comparative analysis of Personalized Voice Activity Detection (PVAD) systems to assess their real-world effectiveness. We introduce a comprehensive approach to assess PVAD systems, incorporating various performance metrics such as frame-level and utterance-level error Rates and Onset Detection latency, alongside user-level analysis. Through extensive experimentation and evaluation, we provide  a thorough understanding of strengths and limitations of various PVAD variants. This paper advances the understanding of PVAD technology by offering insights into its efficacy and viability in practical applications using a comprehensive set of metrics.

Interspeech

Comparative Analysis of Personalized Voice Activity Detection Systems: Assessing Real-World Effectiveness

This paper was accepted at the Industry Track at NAACL 2024.

With increasingly more powerful compute capabilities and resources in today’s devices, traditionally compute-intensive automatic speech recognition (ASR) has been moving from the cloud to devices to better protect user privacy. However, it is still challenging to implement on-device ASR on resource-constrained devices, such as smartphones, smart wearables, and other small home automation devices. In this paper, we propose a series of model architecture adaptions, neural network graph transformations, and numerical optimizations to fit an advanced Conformer based end-to-end streaming ASR system on resource-constrained devices without accuracy degradation. We achieve over 5.26 times faster than realtime (0.19 RTF) speech recognition on small wearables while minimizing energy consumption and achieving state-of-the-art accuracy. The proposed methods are widely applicable to other transformer-based server-free AI applications. In addition, we provide a complete theory on optimal pre-normalizers that numerically stabilize layer normalization in any Lp-norm using any floating point precision.

Machine Learning
Research at Apple

Recent research

Comparative Analysis of Personalized Voice Activity Detection Systems: Assessing Real-World Effectiveness

Conformer-Based Speech Recognition on Extreme Edge-Computing Devices

Research highlights

Introducing Apple’s On-Device and Server Foundation Models

Personalizing Health and Fitness with Hybrid Modeling

All events

IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 2024

ACM Human-Computer Interaction conference (CHI) 2024

Discover opportunities in Machine Learning.

Machine LearningResearch at Apple

Recent research

Comparative Analysis of Personalized Voice Activity Detection Systems: Assessing Real-World Effectiveness

Conformer-Based Speech Recognition on Extreme Edge-Computing Devices

Research highlights

Introducing Apple’s On-Device and Server Foundation Models

Personalizing Health and Fitness with Hybrid Modeling

All events

IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 2024

ACM Human-Computer Interaction conference (CHI) 2024

Discover opportunities in Machine Learning.

Machine Learning
Research at Apple