Showing 1–49 of 49 results for author: Ling, J

PoseTalk: Text-and-Audio-based Pose Control and Motion Refinement for One-Shot Talking Head Generation

Authors: Jun Ling, Yiwen Wang, Han Xue, Rong Xie, Li Song

Abstract: While previous audio-driven talking head generation (THG) methods generate head poses from driving audio, the generated poses or lips cannot match the audio well or are not editable. In this study, we propose \textbf{PoseTalk}, a THG system that can freely generate lip-synchronized talking head videos with free head poses conditioned on text prompts and audio. The core insight of our method is usi… ▽ More While previous audio-driven talking head generation (THG) methods generate head poses from driving audio, the generated poses or lips cannot match the audio well or are not editable. In this study, we propose \textbf{PoseTalk}, a THG system that can freely generate lip-synchronized talking head videos with free head poses conditioned on text prompts and audio. The core insight of our method is using head pose to connect visual, linguistic, and audio signals. First, we propose to generate poses from both audio and text prompts, where the audio offers short-term variations and rhythm correspondence of the head movements and the text prompts describe the long-term semantics of head motions. To achieve this goal, we devise a Pose Latent Diffusion (PLD) model to generate motion latent from text prompts and audio cues in a pose latent space. Second, we observe a loss-imbalance problem: the loss for the lip region contributes less than 4\% of the total reconstruction loss caused by both pose and lip, making optimization lean towards head movements rather than lip shapes. To address this issue, we propose a refinement-based learning strategy to synthesize natural talking videos using two cascaded networks, i.e., CoarseNet, and RefineNet. The CoarseNet estimates coarse motions to produce animated images in novel poses and the RefineNet focuses on learning finer lip motions by progressively estimating lip motions from low-to-high resolutions, yielding improved lip-synchronization performance. Experiments demonstrate our pose prediction strategy achieves better pose diversity and realness compared to text-only or audio-only, and our video generator model outperforms state-of-the-art methods in synthesizing talking videos with natural head motions. Project: https://junleen.github.io/projects/posetalk. △ Less

Submitted 4 September, 2024; originally announced September 2024.

Comments: 7+5 pages, 15 figures

2DGH: 2D Gaussian-Hermite Splatting for High-quality Rendering and Better Geometry Reconstruction

Authors: Ruihan Yu, Tianyu Huang, Jingwang Ling, Feng Xu

Abstract: 2D Gaussian Splatting has recently emerged as a significant method in 3D reconstruction, enabling novel view synthesis and geometry reconstruction simultaneously. While the well-known Gaussian kernel is broadly used, its lack of anisotropy and deformation ability leads to dim and vague edges at object silhouettes, limiting the reconstruction quality of current Gaussian splatting methods. To enhanc… ▽ More 2D Gaussian Splatting has recently emerged as a significant method in 3D reconstruction, enabling novel view synthesis and geometry reconstruction simultaneously. While the well-known Gaussian kernel is broadly used, its lack of anisotropy and deformation ability leads to dim and vague edges at object silhouettes, limiting the reconstruction quality of current Gaussian splatting methods. To enhance the representation power, we draw inspiration from quantum physics and propose to use the Gaussian-Hermite kernel as the new primitive in Gaussian splatting. The new kernel takes a unified mathematical form and extends the Gaussian function, which serves as the zero-rank term in the updated formulation. Our experiments demonstrate the extraordinary performance of Gaussian-Hermite kernel in both geometry reconstruction and novel-view synthesis tasks. The proposed kernel outperforms traditional Gaussian Splatting kernels, showcasing its potential for high-quality 3D reconstruction and rendering. △ Less

Submitted 29 August, 2024; originally announced August 2024.

PRTGaussian: Efficient Relighting Using 3D Gaussians with Precomputed Radiance Transfer

Authors: Libo Zhang, Yuxuan Han, Wenbin Lin, Jingwang Ling, Feng Xu

Abstract: We present PRTGaussian, a realtime relightable novel-view synthesis method made possible by combining 3D Gaussians and Precomputed Radiance Transfer (PRT). By fitting relightable Gaussians to multi-view OLAT data, our method enables real-time, free-viewpoint relighting. By estimating the radiance transfer based on high-order spherical harmonics, we achieve a balance between capturing detailed reli… ▽ More We present PRTGaussian, a realtime relightable novel-view synthesis method made possible by combining 3D Gaussians and Precomputed Radiance Transfer (PRT). By fitting relightable Gaussians to multi-view OLAT data, our method enables real-time, free-viewpoint relighting. By estimating the radiance transfer based on high-order spherical harmonics, we achieve a balance between capturing detailed relighting effects and maintaining computational efficiency. We utilize a two-stage process: in the first stage, we reconstruct a coarse geometry of the object from multi-view images. In the second stage, we initialize 3D Gaussians with the obtained point cloud, then simultaneously refine the coarse geometry and learn the light transport for each Gaussian. Extensive experiments on synthetic datasets show that our approach can achieve fast and high-quality relighting for general objects. Code and data are available at https://github.com/zhanglbthu/PRTGaussian. △ Less

Submitted 10 August, 2024; originally announced August 2024.

Joint Edge Optimization Deep Unfolding Network for Accelerated MRI Reconstruction

Authors: Yue Cai, Yu Luo, Jie Ling, Shun Yao

Abstract: Magnetic Resonance Imaging (MRI) is a widely used imaging technique, however it has the limitation of long scanning time. Though previous model-based and learning-based MRI reconstruction methods have shown promising performance, most of them have not fully utilized the edge prior of MR images, and there is still much room for improvement. In this paper, we build a joint edge optimization model th… ▽ More Magnetic Resonance Imaging (MRI) is a widely used imaging technique, however it has the limitation of long scanning time. Though previous model-based and learning-based MRI reconstruction methods have shown promising performance, most of them have not fully utilized the edge prior of MR images, and there is still much room for improvement. In this paper, we build a joint edge optimization model that not only incorporates individual regularizers specific to both the MR image and the edges, but also enforces a co-regularizer to effectively establish a stronger correlation between them. Specifically, the edge information is defined through a non-edge probability map to guide the image reconstruction during the optimization process. Meanwhile, the regularizers pertaining to images and edges are incorporated into a deep unfolding network to automatically learn their respective inherent a-priori information.Numerical experiments, consisting of multi-coil and single-coil MRI data with different sampling schemes at a variety of sampling factors, demonstrate that the proposed method outperforms other compared methods. △ Less

Submitted 9 May, 2024; originally announced May 2024.

Bridging Worlds: Achieving Language Interoperability between Julia and Python in Scientific Computing

Authors: Ianna Osborne, Jim Pivarski, Jerry Ling

Abstract: In the realm of scientific computing, both Julia and Python have established themselves as powerful tools. Within the context of High Energy Physics (HEP) data analysis, Python has been traditionally favored, yet there exists a compelling case for migrating legacy software to Julia. This article focuses on language interoperability, specifically exploring how Awkward Array data structures can brid… ▽ More In the realm of scientific computing, both Julia and Python have established themselves as powerful tools. Within the context of High Energy Physics (HEP) data analysis, Python has been traditionally favored, yet there exists a compelling case for migrating legacy software to Julia. This article focuses on language interoperability, specifically exploring how Awkward Array data structures can bridge the gap between Julia and Python. The talk offers insights into key considerations such as memory management, data buffer copies, and dependency handling. It delves into the performance enhancements achieved by invoking Julia from Python and vice versa, particularly for intensive array-oriented calculations involving large-scale, though not excessively dimensional, arrays of HEP data. The advantages and challenges inherent in achieving interoperability between Julia and Python in the domain of scientific computing are discussed. △ Less

Submitted 28 April, 2024; originally announced April 2024.

Comments: 8 pages, 1 figure, ACAT2024 workshop

Exploring Convergence in Relation using Association Rules Mining: A Case Study in Collaborative Knowledge Production

Authors: Jiahe Ling, Corey B. Jackson

Abstract: This study delves into the pivotal role played by non-experts in knowledge production on open collaboration platforms, with a particular focus on the intricate process of tag development that culminates in the proposal of new glitch classes. Leveraging the power of Association Rule Mining (ARM), this research endeavors to unravel the underlying dynamics of collaboration among citizen scientists. B… ▽ More This study delves into the pivotal role played by non-experts in knowledge production on open collaboration platforms, with a particular focus on the intricate process of tag development that culminates in the proposal of new glitch classes. Leveraging the power of Association Rule Mining (ARM), this research endeavors to unravel the underlying dynamics of collaboration among citizen scientists. By meticulously quantifying tag associations and scrutinizing their temporal dynamics, the study provides a comprehensive and nuanced understanding of how non-experts collaborate to generate valuable scientific insights. Furthermore, this investigation extends its purview to examine the phenomenon of ideological convergence within online citizen science knowledge production. To accomplish this, a novel measurement algorithm, based on the Mann-Kendall Trend Test, is introduced. This innovative approach sheds illuminating light on the dynamics of collaborative knowledge production, revealing both the vast opportunities and daunting challenges inherent in leveraging non-expert contributions for scientific research endeavors. Notably, the study uncovers a robust pattern of convergence in ideology, employing both the newly proposed convergence testing method and the traditional approach based on the stationarity of time series data. This groundbreaking discovery holds significant implications for understanding the dynamics of online citizen science communities and underscores the crucial role played by non-experts in shaping the scientific landscape of the digital age. Ultimately, this study contributes significantly to our understanding of online citizen science communities, highlighting their potential to harness collective intelligence for tackling complex scientific tasks and enriching our comprehension of collaborative knowledge production processes in the digital age. △ Less

Submitted 13 May, 2024; v1 submitted 23 April, 2024; originally announced April 2024.

Physics-Guided Neural Networks for Intraventricular Vector Flow Mapping

Authors: Hang Jung Ling, Salomé Bru, Julia Puig, Florian Vixège, Simon Mendez, Franck Nicoud, Pierre-Yves Courand, Olivier Bernard, Damien Garcia

Abstract: Intraventricular vector flow mapping (iVFM) seeks to enhance and quantify color Doppler in cardiac imaging. In this study, we propose novel alternatives to the traditional iVFM optimization scheme by utilizing physics-informed neural networks (PINNs) and a physics-guided nnU-Net-based supervised approach. When evaluated on simulated color Doppler images derived from a patient-specific computationa… ▽ More Intraventricular vector flow mapping (iVFM) seeks to enhance and quantify color Doppler in cardiac imaging. In this study, we propose novel alternatives to the traditional iVFM optimization scheme by utilizing physics-informed neural networks (PINNs) and a physics-guided nnU-Net-based supervised approach. When evaluated on simulated color Doppler images derived from a patient-specific computational fluid dynamics model and in vivo Doppler acquisitions, both approaches demonstrate comparable reconstruction performance to the original iVFM algorithm. The efficiency of PINNs is boosted through dual-stage optimization and pre-optimized weights. On the other hand, the nnU-Net method excels in generalizability and real-time capabilities. Notably, nnU-Net shows superior robustness on sparse and truncated Doppler data while maintaining independence from explicit boundary conditions. Overall, our results highlight the effectiveness of these methods in reconstructing intraventricular vector blood flow. The study also suggests potential applications of PINNs in ultrafast color Doppler imaging and the incorporation of fluid dynamics equations to derive biomarkers for cardiovascular diseases based on blood flow. △ Less

Submitted 27 June, 2024; v1 submitted 19 March, 2024; originally announced March 2024.

Comments: 12 pages, accepted for publication in IEEE TUFFC; camera ready corrections, corrected acknowledgments

FlowPG: Action-constrained Policy Gradient with Normalizing Flows

Authors: Janaka Chathuranga Brahmanage, Jiajing Ling, Akshat Kumar

Abstract: Action-constrained reinforcement learning (ACRL) is a popular approach for solving safety-critical and resource-allocation related decision making problems. A major challenge in ACRL is to ensure agent taking a valid action satisfying constraints in each RL step. Commonly used approach of using a projection layer on top of the policy network requires solving an optimization program which can resul… ▽ More Action-constrained reinforcement learning (ACRL) is a popular approach for solving safety-critical and resource-allocation related decision making problems. A major challenge in ACRL is to ensure agent taking a valid action satisfying constraints in each RL step. Commonly used approach of using a projection layer on top of the policy network requires solving an optimization program which can result in longer training time, slow convergence, and zero gradient problem. To address this, first we use a normalizing flow model to learn an invertible, differentiable mapping between the feasible action space and the support of a simple distribution on a latent variable, such as Gaussian. Second, learning the flow model requires sampling from the feasible action space, which is also challenging. We develop multiple methods, based on Hamiltonian Monte-Carlo and probabilistic sentential decision diagrams for such action sampling for convex and non-convex constraints. Third, we integrate the learned normalizing flow with the DDPG algorithm. By design, a well-trained normalizing flow will transform policy output into a valid action without requiring an optimization solver. Empirically, our approach results in significantly fewer constraint violations (upto an order-of-magnitude for several instances) and is multiple times faster on a variety of continuous control tasks. △ Less

Submitted 7 February, 2024; originally announced February 2024.

Journal ref: Thirty-seventh Conference on Neural Information Processing Systems. 2023

NeRF as a Non-Distant Environment Emitter in Physics-based Inverse Rendering

Authors: Jingwang Ling, Ruihan Yu, Feng Xu, Chun Du, Shuang Zhao

Abstract: Physics-based inverse rendering enables joint optimization of shape, material, and lighting based on captured 2D images. To ensure accurate reconstruction, using a light model that closely resembles the captured environment is essential. Although the widely adopted distant environmental lighting model is adequate in many cases, we demonstrate that its inability to capture spatially varying illumin… ▽ More Physics-based inverse rendering enables joint optimization of shape, material, and lighting based on captured 2D images. To ensure accurate reconstruction, using a light model that closely resembles the captured environment is essential. Although the widely adopted distant environmental lighting model is adequate in many cases, we demonstrate that its inability to capture spatially varying illumination can lead to inaccurate reconstructions in many real-world inverse rendering scenarios. To address this limitation, we incorporate NeRF as a non-distant environment emitter into the inverse rendering pipeline. Additionally, we introduce an emitter importance sampling technique for NeRF to reduce the rendering variance. Through comparisons on both real and synthetic datasets, our results demonstrate that our NeRF-based emitter offers a more precise representation of scene lighting, thereby improving the accuracy of inverse rendering. △ Less

Submitted 1 May, 2024; v1 submitted 7 February, 2024; originally announced February 2024.

Comments: SIGGRAPH 2024. Project page and video: https://nerfemitterpbir.github.io/

High-Quality Mesh Blendshape Generation from Face Videos via Neural Inverse Rendering

Authors: Xin Ming, Jiawei Li, Jingwang Ling, Libo Zhang, Feng Xu

Abstract: Readily editable mesh blendshapes have been widely used in animation pipelines, while recent advancements in neural geometry and appearance representations have enabled high-quality inverse rendering. Building upon these observations, we introduce a novel technique that reconstructs mesh-based blendshape rigs from single or sparse multi-view videos, leveraging state-of-the-art neural inverse rende… ▽ More Readily editable mesh blendshapes have been widely used in animation pipelines, while recent advancements in neural geometry and appearance representations have enabled high-quality inverse rendering. Building upon these observations, we introduce a novel technique that reconstructs mesh-based blendshape rigs from single or sparse multi-view videos, leveraging state-of-the-art neural inverse rendering. We begin by constructing a deformation representation that parameterizes vertex displacements into differential coordinates with tetrahedral connections, allowing for high-quality vertex deformation on high-resolution meshes. By constructing a set of semantic regulations in this representation, we achieve joint optimization of blendshapes and expression coefficients. Furthermore, to enable a user-friendly multi-view setup with unsynchronized cameras, we propose a neural regressor to model time-varying motion parameters. This approach implicitly considers the time difference across multiple cameras, enhancing the accuracy of motion modeling. Experiments demonstrate that, with the flexible input of single or sparse multi-view videos, we reconstruct personalized high-fidelity blendshapes. These blendshapes are both geometrically and semantically accurate, and they are compatible with industrial animation pipelines. Code and data are available at https://github.com/grignarder/high-quality-blendshape-generation. △ Less

Submitted 19 August, 2024; v1 submitted 16 January, 2024; originally announced January 2024.

360-Degree Panorama Generation from Few Unregistered NFoV Images

Authors: Jionghao Wang, Ziyu Chen, Jun Ling, Rong Xie, Li Song

Abstract: 360$^\circ$ panoramas are extensively utilized as environmental light sources in computer graphics. However, capturing a 360$^\circ$ $\times$ 180$^\circ$ panorama poses challenges due to the necessity of specialized and costly equipment, and additional human resources. Prior studies develop various learning-based generative methods to synthesize panoramas from a single Narrow Field-of-View (NFoV)… ▽ More 360$^\circ$ panoramas are extensively utilized as environmental light sources in computer graphics. However, capturing a 360$^\circ$ $\times$ 180$^\circ$ panorama poses challenges due to the necessity of specialized and costly equipment, and additional human resources. Prior studies develop various learning-based generative methods to synthesize panoramas from a single Narrow Field-of-View (NFoV) image, but they are limited in alterable input patterns, generation quality, and controllability. To address these issues, we propose a novel pipeline called PanoDiff, which efficiently generates complete 360$^\circ$ panoramas using one or more unregistered NFoV images captured from arbitrary angles. Our approach has two primary components to overcome the limitations. Firstly, a two-stage angle prediction module to handle various numbers of NFoV inputs. Secondly, a novel latent diffusion-based panorama generation model uses incomplete panorama and text prompts as control signals and utilizes several geometric augmentation schemes to ensure geometric properties in generated panoramas. Experiments show that PanoDiff achieves state-of-the-art panoramic generation quality and high controllability, making it suitable for applications such as content editing. △ Less

Submitted 28 August, 2023; originally announced August 2023.

Comments: Accepted to ACM Multimedia 2023 (MM' 23). Code is available: https://github.com/shanemankiw/Panodiff

VAST: Vivify Your Talking Avatar via Zero-Shot Expressive Facial Style Transfer

Authors: Liyang Chen, Zhiyong Wu, Runnan Li, Weihong Bao, Jun Ling, Xu Tan, Sheng Zhao

Abstract: Current talking face generation methods mainly focus on speech-lip synchronization. However, insufficient investigation on the facial talking style leads to a lifeless and monotonous avatar. Most previous works fail to imitate expressive styles from arbitrary video prompts and ensure the authenticity of the generated video. This paper proposes an unsupervised variational style transfer model (VAST… ▽ More Current talking face generation methods mainly focus on speech-lip synchronization. However, insufficient investigation on the facial talking style leads to a lifeless and monotonous avatar. Most previous works fail to imitate expressive styles from arbitrary video prompts and ensure the authenticity of the generated video. This paper proposes an unsupervised variational style transfer model (VAST) to vivify the neutral photo-realistic avatars. Our model consists of three key components: a style encoder that extracts facial style representations from the given video prompts; a hybrid facial expression decoder to model accurate speech-related movements; a variational style enhancer that enhances the style space to be highly expressive and meaningful. With our essential designs on facial style learning, our model is able to flexibly capture the expressive facial style from arbitrary video prompts and transfer it onto a personalized image renderer in a zero-shot manner. Experimental results demonstrate the proposed approach contributes to a more vivid talking avatar with higher authenticity and richer expressiveness. △ Less

Submitted 11 August, 2023; v1 submitted 9 August, 2023; originally announced August 2023.

Comments: Accepted by ICCV2023 Workshop

Context-Aware Talking-Head Video Editing

Authors: Songlin Yang, Wei Wang, Jun Ling, Bo Peng, Xu Tan, Jing Dong

Abstract: Talking-head video editing aims to efficiently insert, delete, and substitute the word of a pre-recorded video through a text transcript editor. The key challenge for this task is obtaining an editing model that generates new talking-head video clips which simultaneously have accurate lip synchronization and motion smoothness. Previous approaches, including 3DMM-based (3D Morphable Model) methods… ▽ More Talking-head video editing aims to efficiently insert, delete, and substitute the word of a pre-recorded video through a text transcript editor. The key challenge for this task is obtaining an editing model that generates new talking-head video clips which simultaneously have accurate lip synchronization and motion smoothness. Previous approaches, including 3DMM-based (3D Morphable Model) methods and NeRF-based (Neural Radiance Field) methods, are sub-optimal in that they either require minutes of source videos and days of training time or lack the disentangled control of verbal (e.g., lip motion) and non-verbal (e.g., head pose and expression) representations for video clip insertion. In this work, we fully utilize the video context to design a novel framework for talking-head video editing, which achieves efficiency, disentangled motion control, and sequential smoothness. Specifically, we decompose this framework to motion prediction and motion-conditioned rendering: (1) We first design an animation prediction module that efficiently obtains smooth and lip-sync motion sequences conditioned on the driven speech. This module adopts a non-autoregressive network to obtain context prior and improve the prediction efficiency, and it learns a speech-animation mapping prior with better generalization to novel speech from a multi-identity video dataset. (2) We then introduce a neural rendering module to synthesize the photo-realistic and full-head video frames given the predicted motion sequence. This module adopts a pre-trained head topology and uses only few frames for efficient fine-tuning to obtain a person-specific rendering model. Extensive experiments demonstrate that our method efficiently achieves smoother editing results with higher image quality and lip accuracy using less data than previous methods. △ Less

Submitted 20 September, 2023; v1 submitted 1 August, 2023; originally announced August 2023.

Comments: The version of this paper needs to be further improved

Learning Dense UV Completion for Human Mesh Recovery

Authors: Yanjun Wang, Qingping Sun, Wenjia Wang, Jun Ling, Zhongang Cai, Rong Xie, Li Song

Abstract: Human mesh reconstruction from a single image is challenging in the presence of occlusion, which can be caused by self, objects, or other humans. Existing methods either fail to separate human features accurately or lack proper supervision for feature completion. In this paper, we propose Dense Inpainting Human Mesh Recovery (DIMR), a two-stage method that leverages dense correspondence maps to ha… ▽ More Human mesh reconstruction from a single image is challenging in the presence of occlusion, which can be caused by self, objects, or other humans. Existing methods either fail to separate human features accurately or lack proper supervision for feature completion. In this paper, we propose Dense Inpainting Human Mesh Recovery (DIMR), a two-stage method that leverages dense correspondence maps to handle occlusion. Our method utilizes a dense correspondence map to separate visible human features and completes human features on a structured UV map dense human with an attention-based feature completion module. We also design a feature inpainting training procedure that guides the network to learn from unoccluded features. We evaluate our method on several datasets and demonstrate its superior performance under heavily occluded scenarios compared to other methods. Extensive experiments show that our method obviously outperforms prior SOTA methods on heavily occluded images and achieves comparable results on the standard benchmarks (3DPW). △ Less

Submitted 10 August, 2023; v1 submitted 20 July, 2023; originally announced July 2023.

Phase Unwrapping of Color Doppler Echocardiography using Deep Learning

Authors: Hang Jung Ling, Olivier Bernard, Nicolas Ducros, Damien Garcia

Abstract: Color Doppler echocardiography is a widely used non-invasive imaging modality that provides real-time information about the intracardiac blood flow. In an apical long-axis view of the left ventricle, color Doppler is subject to phase wrapping, or aliasing, especially during cardiac filling and ejection. When setting up quantitative methods based on color Doppler, it is necessary to correct this wr… ▽ More Color Doppler echocardiography is a widely used non-invasive imaging modality that provides real-time information about the intracardiac blood flow. In an apical long-axis view of the left ventricle, color Doppler is subject to phase wrapping, or aliasing, especially during cardiac filling and ejection. When setting up quantitative methods based on color Doppler, it is necessary to correct this wrapping artifact. We developed an unfolded primal-dual network to unwrap (dealias) color Doppler echocardiographic images and compared its effectiveness against two state-of-the-art segmentation approaches based on nnU-Net and transformer models. We trained and evaluated the performance of each method on an in-house dataset and found that the nnU-Net-based method provided the best dealiased results, followed by the primal-dual approach and the transformer-based technique. Noteworthy, the primal-dual network, which had significantly fewer trainable parameters, performed competitively with respect to the other two methods, demonstrating the high potential of deep unfolding methods. Our results suggest that deep learning-based methods can effectively remove aliasing artifacts in color Doppler echocardiographic images, outperforming DeAN, a state-of-the-art semi-automatic technique. Overall, our results show that deep learning-based methods have the potential to effectively preprocess color Doppler images for downstream quantitative analysis. △ Less

Submitted 5 July, 2023; v1 submitted 23 June, 2023; originally announced June 2023.

Comments: 11 pages, accepted for publication in IEEE TUFFC, modified graphical abstract

Potential of the Julia programming language for high energy physics computing

Authors: J. Eschle, T. Gal, M. Giordano, P. Gras, B. Hegner, L. Heinrich, U. Hernandez Acosta, S. Kluth, J. Ling, P. Mato, M. Mikhasenko, A. Moreno Briceño, J. Pivarski, K. Samaras-Tsakiris, O. Schulz, G. . A. Stewart, J. Strube, V. Vassilev

Abstract: Research in high energy physics (HEP) requires huge amounts of computing and storage, putting strong constraints on the code speed and resource usage. To meet these requirements, a compiled high-performance language is typically used; while for physicists, who focus on the application when developing the code, better research productivity pleads for a high-level programming language. A popular app… ▽ More Research in high energy physics (HEP) requires huge amounts of computing and storage, putting strong constraints on the code speed and resource usage. To meet these requirements, a compiled high-performance language is typically used; while for physicists, who focus on the application when developing the code, better research productivity pleads for a high-level programming language. A popular approach consists of combining Python, used for the high-level interface, and C++, used for the computing intensive part of the code. A more convenient and efficient approach would be to use a language that provides both high-level programming and high-performance. The Julia programming language, developed at MIT especially to allow the use of a single language in research activities, has followed this path. In this paper the applicability of using the Julia language for HEP research is explored, covering the different aspects that are important for HEP code development: runtime performance, handling of large projects, interface with legacy code, distributed computing, training, and ease of programming. The study shows that the HEP community would benefit from a large scale adoption of this programming language. The HEP-specific foundation libraries that would need to be consolidated are identified △ Less

Submitted 6 October, 2023; v1 submitted 6 June, 2023; originally announced June 2023.

Comments: 32 pages, 5 figures, 4 tables

ACM Class: J.2

Journal ref: Computing. Comput Softw Big Sci 7, 10 (2023)

Extraction of volumetric indices from echocardiography: which deep learning solution for clinical use?

Authors: Hang Jung Ling, Nathan Painchaud, Pierre-Yves Courand, Pierre-Marc Jodoin, Damien Garcia, Olivier Bernard

Abstract: Deep learning-based methods have spearheaded the automatic analysis of echocardiographic images, taking advantage of the publication of multiple open access datasets annotated by experts (CAMUS being one of the largest public databases). However, these models are still considered unreliable by clinicians due to unresolved issues concerning i) the temporal consistency of their predictions, and ii)… ▽ More Deep learning-based methods have spearheaded the automatic analysis of echocardiographic images, taking advantage of the publication of multiple open access datasets annotated by experts (CAMUS being one of the largest public databases). However, these models are still considered unreliable by clinicians due to unresolved issues concerning i) the temporal consistency of their predictions, and ii) their ability to generalize across datasets. In this context, we propose a comprehensive comparison between the current best performing methods in medical/echocardiographic image segmentation, with a particular focus on temporal consistency and cross-dataset aspects. We introduce a new private dataset, named CARDINAL, of apical two-chamber and apical four-chamber sequences, with reference segmentation over the full cardiac cycle. We show that the proposed 3D nnU-Net outperforms alternative 2D and recurrent segmentation methods. We also report that the best models trained on CARDINAL, when tested on CAMUS without any fine-tuning, still manage to perform competitively with respect to prior methods. Overall, the experimental results suggest that with sufficient training data, 3D nnU-Net could become the first automated tool to finally meet the standards of an everyday clinical device. △ Less

Submitted 8 May, 2023; v1 submitted 3 May, 2023; originally announced May 2023.

Comments: 10 pages, accepted for FIMH 2023; camera ready corrections, corrected acknowledgments

Interference-Aware Deployment for Maximizing User Satisfaction in Multi-UAV Wireless Networks

Authors: Chuan-Chi Lai, Ang-Hsun Tsai, Chia-Wei Ting, Ko-Han Lin, Jing-Chi Ling, Chia-En Tsai

Abstract: In this letter, we study the deployment of Unmanned Aerial Vehicle mounted Base Stations (UAV-BSs) in multi-UAV cellular networks. We model the multi-UAV deployment problem as a user satisfaction maximization problem, that is, maximizing the proportion of served ground users (GUs) that meet a given minimum data rate requirement. We propose an interference-aware deployment (IAD) algorithm for servi… ▽ More In this letter, we study the deployment of Unmanned Aerial Vehicle mounted Base Stations (UAV-BSs) in multi-UAV cellular networks. We model the multi-UAV deployment problem as a user satisfaction maximization problem, that is, maximizing the proportion of served ground users (GUs) that meet a given minimum data rate requirement. We propose an interference-aware deployment (IAD) algorithm for serving arbitrarily distributed outdoor GUs. The proposed algorithm can alleviate the problem of overlapping coverage between adjacent UAV-BSs to minimize inter-cell interference. Therefore, reducing co-channel interference between UAV-BSs will improve user satisfaction and ensure that most GUs can achieve the minimum data rate requirement. Simulation results show that our proposed IAD outperforms comparative methods by more than 10% in user satisfaction in high-density environments. △ Less

Submitted 6 April, 2023; originally announced April 2023.

Comments: 5 pages, 3 figures, to appear in IEEE Wireless Communications Letters

Memories are One-to-Many Mapping Alleviators in Talking Face Generation

Authors: Anni Tang, Tianyu He, Xu Tan, Jun Ling, Li Song

Abstract: Talking face generation aims at generating photo-realistic video portraits of a target person driven by input audio. Due to its nature of one-to-many mapping from the input audio to the output video (e.g., one speech content may have multiple feasible visual appearances), learning a deterministic mapping like previous works brings ambiguity during training, and thus causes inferior visual results.… ▽ More Talking face generation aims at generating photo-realistic video portraits of a target person driven by input audio. Due to its nature of one-to-many mapping from the input audio to the output video (e.g., one speech content may have multiple feasible visual appearances), learning a deterministic mapping like previous works brings ambiguity during training, and thus causes inferior visual results. Although this one-to-many mapping could be alleviated in part by a two-stage framework (i.e., an audio-to-expression model followed by a neural-rendering model), it is still insufficient since the prediction is produced without enough information (e.g., emotions, wrinkles, etc.). In this paper, we propose MemFace to complement the missing information with an implicit memory and an explicit memory that follow the sense of the two stages respectively. More specifically, the implicit memory is employed in the audio-to-expression model to capture high-level semantics in the audio-expression shared space, while the explicit memory is employed in the neural-rendering model to help synthesize pixel-level details. Our experimental results show that our proposed MemFace surpasses all the state-of-the-art results across multiple scenarios consistently and significantly. △ Less

Submitted 5 March, 2024; v1 submitted 9 December, 2022; originally announced December 2022.

Comments: Project page: see https://memoryface.github.io

Journal ref: IEEE Transactions on Pattern Analysis and Machine Intelligence (2024)

ShadowNeuS: Neural SDF Reconstruction by Shadow Ray Supervision

Authors: Jingwang Ling, Zhibo Wang, Feng Xu

Abstract: By supervising camera rays between a scene and multi-view image planes, NeRF reconstructs a neural scene representation for the task of novel view synthesis. On the other hand, shadow rays between the light source and the scene have yet to be considered. Therefore, we propose a novel shadow ray supervision scheme that optimizes both the samples along the ray and the ray location. By supervising sh… ▽ More By supervising camera rays between a scene and multi-view image planes, NeRF reconstructs a neural scene representation for the task of novel view synthesis. On the other hand, shadow rays between the light source and the scene have yet to be considered. Therefore, we propose a novel shadow ray supervision scheme that optimizes both the samples along the ray and the ray location. By supervising shadow rays, we successfully reconstruct a neural SDF of the scene from single-view images under multiple lighting conditions. Given single-view binary shadows, we train a neural network to reconstruct a complete scene not limited by the camera's line of sight. By further modeling the correlation between the image colors and the shadow rays, our technique can also be effectively extended to RGB inputs. We compare our method with previous works on challenging tasks of shape reconstruction from single-view binary shadow or RGB images and observe significant improvements. The code and data are available at https://github.com/gerwang/ShadowNeuS. △ Less

Submitted 23 March, 2023; v1 submitted 25 November, 2022; originally announced November 2022.

Comments: CVPR 2023. Project page: https://gerwang.github.io/shadowneus/

StableFace: Analyzing and Improving Motion Stability for Talking Face Generation

Authors: Jun Ling, Xu Tan, Liyang Chen, Runnan Li, Yuchao Zhang, Sheng Zhao, Li Song

Abstract: While previous speech-driven talking face generation methods have made significant progress in improving the visual quality and lip-sync quality of the synthesized videos, they pay less attention to lip motion jitters which greatly undermine the realness of talking face videos. What causes motion jitters, and how to mitigate the problem? In this paper, we conduct systematic analyses on the motion… ▽ More While previous speech-driven talking face generation methods have made significant progress in improving the visual quality and lip-sync quality of the synthesized videos, they pay less attention to lip motion jitters which greatly undermine the realness of talking face videos. What causes motion jitters, and how to mitigate the problem? In this paper, we conduct systematic analyses on the motion jittering problem based on a state-of-the-art pipeline that uses 3D face representations to bridge the input audio and output video, and improve the motion stability with a series of effective designs. We find that several issues can lead to jitters in synthesized talking face video: 1) jitters from the input 3D face representations; 2) training-inference mismatch; 3) lack of dependency modeling among video frames. Accordingly, we propose three effective solutions to address this issue: 1) we propose a gaussian-based adaptive smoothing module to smooth the 3D face representations to eliminate jitters in the input; 2) we add augmented erosions on the input data of the neural renderer in training to simulate the distortion in inference to reduce mismatch; 3) we develop an audio-fused transformer generator to model dependency among video frames. Besides, considering there is no off-the-shelf metric for measuring motion jitters in talking face video, we devise an objective metric (Motion Stability Index, MSI), to quantitatively measure the motion jitters by calculating the reciprocal of variance acceleration. Extensive experimental results show the superiority of our method on motion-stable face video generation, with better quality than previous systems. △ Less

Submitted 29 August, 2022; originally announced August 2022.

Comments: 12 pages,

Structure-aware Editable Morphable Model for 3D Facial Detail Animation and Manipulation

Authors: Jingwang Ling, Zhibo Wang, Ming Lu, Quan Wang, Chen Qian, Feng Xu

Abstract: Morphable models are essential for the statistical modeling of 3D faces. Previous works on morphable models mostly focus on large-scale facial geometry but ignore facial details. This paper augments morphable models in representing facial details by learning a Structure-aware Editable Morphable Model (SEMM). SEMM introduces a detail structure representation based on the distance field of wrinkle l… ▽ More Morphable models are essential for the statistical modeling of 3D faces. Previous works on morphable models mostly focus on large-scale facial geometry but ignore facial details. This paper augments morphable models in representing facial details by learning a Structure-aware Editable Morphable Model (SEMM). SEMM introduces a detail structure representation based on the distance field of wrinkle lines, jointly modeled with detail displacements to establish better correspondences and enable intuitive manipulation of wrinkle structure. Besides, SEMM introduces two transformation modules to translate expression blendshape weights and age values into changes in latent space, allowing effective semantic detail editing while maintaining identity. Extensive experiments demonstrate that the proposed model compactly represents facial details, outperforms previous methods in expression animation qualitatively and quantitatively, and achieves effective age editing and wrinkle line editing of facial details. Code and model are available at https://github.com/gerwang/facial-detail-manipulation. △ Less

Submitted 18 July, 2022; originally announced July 2022.

Comments: ECCV 2022

KEMP: Keyframe-Based Hierarchical End-to-End Deep Model for Long-Term Trajectory Prediction

Authors: Qiujing Lu, Weiqiao Han, Jeffrey Ling, Minfa Wang, Haoyu Chen, Balakrishnan Varadarajan, Paul Covington

Abstract: Predicting future trajectories of road agents is a critical task for autonomous driving. Recent goal-based trajectory prediction methods, such as DenseTNT and PECNet, have shown good performance on prediction tasks on public datasets. However, they usually require complicated goal-selection algorithms and optimization. In this work, we propose KEMP, a hierarchical end-to-end deep learning framewor… ▽ More Predicting future trajectories of road agents is a critical task for autonomous driving. Recent goal-based trajectory prediction methods, such as DenseTNT and PECNet, have shown good performance on prediction tasks on public datasets. However, they usually require complicated goal-selection algorithms and optimization. In this work, we propose KEMP, a hierarchical end-to-end deep learning framework for trajectory prediction. At the core of our framework is keyframe-based trajectory prediction, where keyframes are representative states that trace out the general direction of the trajectory. KEMP first predicts keyframes conditioned on the road context, and then fills in intermediate states conditioned on the keyframes and the road context. Under our general framework, goal-conditioned methods are special cases in which the number of keyframes equal to one. Unlike goal-conditioned methods, our keyframe predictor is learned automatically and does not require hand-crafted goal-selection algorithms. We evaluate our model on public benchmarks and our model ranked 1st on Waymo Open Motion Dataset Leaderboard (as of September 1, 2021). △ Less

Submitted 9 May, 2022; originally announced May 2022.

Comments: Accepted at the 39th IEEE Conference on Robotics and Automation (ICRA), 2022

Transformer-S2A: Robust and Efficient Speech-to-Animation

Authors: Liyang Chen, Zhiyong Wu, Jun Ling, Runnan Li, Xu Tan, Sheng Zhao

Abstract: We propose a novel robust and efficient Speech-to-Animation (S2A) approach for synchronized facial animation generation in human-computer interaction. Compared with conventional approaches, the proposed approach utilizes phonetic posteriorgrams (PPGs) of spoken phonemes as input to ensure the cross-language and cross-speaker ability, and introduces corresponding prosody features (i.e. pitch and en… ▽ More We propose a novel robust and efficient Speech-to-Animation (S2A) approach for synchronized facial animation generation in human-computer interaction. Compared with conventional approaches, the proposed approach utilizes phonetic posteriorgrams (PPGs) of spoken phonemes as input to ensure the cross-language and cross-speaker ability, and introduces corresponding prosody features (i.e. pitch and energy) to further enhance the expression of generated animation. Mixture-of-experts (MOE)-based Transformer is employed to better model contextual information while provide significant optimization on computation efficiency. Experiments demonstrate the effectiveness of the proposed approach on both objective and subjective evaluation with 17x inference speedup compared with the state-of-the-art approach. △ Less

Submitted 6 April, 2022; v1 submitted 18 November, 2021; originally announced November 2021.

Comments: Accepted by ICASSP 2022

Scene Transformer: A unified architecture for predicting multiple agent trajectories

Authors: Jiquan Ngiam, Benjamin Caine, Vijay Vasudevan, Zhengdong Zhang, Hao-Tien Lewis Chiang, Jeffrey Ling, Rebecca Roelofs, Alex Bewley, Chenxi Liu, Ashish Venugopal, David Weiss, Ben Sapp, Zhifeng Chen, Jonathon Shlens

Abstract: Predicting the motion of multiple agents is necessary for planning in dynamic environments. This task is challenging for autonomous driving since agents (e.g. vehicles and pedestrians) and their associated behaviors may be diverse and influence one another. Most prior work have focused on predicting independent futures for each agent based on all past motion, and planning against these independent… ▽ More Predicting the motion of multiple agents is necessary for planning in dynamic environments. This task is challenging for autonomous driving since agents (e.g. vehicles and pedestrians) and their associated behaviors may be diverse and influence one another. Most prior work have focused on predicting independent futures for each agent based on all past motion, and planning against these independent predictions. However, planning against independent predictions can make it challenging to represent the future interaction possibilities between different agents, leading to sub-optimal planning. In this work, we formulate a model for predicting the behavior of all agents jointly, producing consistent futures that account for interactions between agents. Inspired by recent language modeling approaches, we use a masking strategy as the query to our model, enabling one to invoke a single model to predict agent behavior in many ways, such as potentially conditioned on the goal or full future trajectory of the autonomous vehicle or the behavior of other agents in the environment. Our model architecture employs attention to combine features across road elements, agent interactions, and time steps. We evaluate our approach on autonomous driving datasets for both marginal and joint motion prediction, and achieve state of the art performance across two popular datasets. Through combining a scene-centric approach, agent permutation equivariant model, and a sequence masking strategy, we show that our model can unify a variety of motion prediction tasks from joint motion predictions to conditioned prediction. △ Less

Submitted 4 March, 2022; v1 submitted 15 June, 2021; originally announced June 2021.

Comments: ICLR 2022

Region-aware Adaptive Instance Normalization for Image Harmonization

Authors: Jun Ling, Han Xue, Li Song, Rong Xie, Xiao Gu

Abstract: Image composition plays a common but important role in photo editing. To acquire photo-realistic composite images, one must adjust the appearance and visual style of the foreground to be compatible with the background. Existing deep learning methods for harmonizing composite images directly learn an image mapping network from the composite to the real one, without explicit exploration on visual st… ▽ More Image composition plays a common but important role in photo editing. To acquire photo-realistic composite images, one must adjust the appearance and visual style of the foreground to be compatible with the background. Existing deep learning methods for harmonizing composite images directly learn an image mapping network from the composite to the real one, without explicit exploration on visual style consistency between the background and the foreground images. To ensure the visual style consistency between the foreground and the background, in this paper, we treat image harmonization as a style transfer problem. In particular, we propose a simple yet effective Region-aware Adaptive Instance Normalization (RAIN) module, which explicitly formulates the visual style from the background and adaptively applies them to the foreground. With our settings, our RAIN module can be used as a drop-in module for existing image harmonization networks and is able to bring significant improvements. Extensive experiments on the existing image harmonization benchmark datasets show the superior capability of the proposed method. Code is available at {https://github.com/junleen/RainNet}. △ Less

Submitted 5 June, 2021; originally announced June 2021.

Comments: Accepted to IEEE CVPR 2021

Combining Propositional Logic Based Decision Diagrams with Decision Making in Urban Systems

Authors: Jiajing Ling, Kushagra Chandak, Akshat Kumar

Abstract: Solving multiagent problems can be an uphill task due to uncertainty in the environment, partial observability, and scalability of the problem at hand. Especially in an urban setting, there are more challenges since we also need to maintain safety for all users while minimizing congestion of the agents as well as their travel times. To this end, we tackle the problem of multiagent pathfinding unde… ▽ More Solving multiagent problems can be an uphill task due to uncertainty in the environment, partial observability, and scalability of the problem at hand. Especially in an urban setting, there are more challenges since we also need to maintain safety for all users while minimizing congestion of the agents as well as their travel times. To this end, we tackle the problem of multiagent pathfinding under uncertainty and partial observability where the agents are tasked to move from their starting points to ending points while also satisfying some constraints, e.g., low congestion, and model it as a multiagent reinforcement learning problem. We compile the domain constraints using propositional logic and integrate them with the RL algorithms to enable fast simulation for RL. △ Less

Submitted 10 November, 2020; v1 submitted 9 November, 2020; originally announced November 2020.

Fast LDPC GPU Decoder for Cloud RAN

Authors: Jonathan Ling, Paul Cautereels

Abstract: The GPU as a digital signal processing accelerator for cloud RAN is investigated. A new design for a 5G NR low density parity check code decoder running on a GPU is presented. The algorithm is flexibly adaptable to GPU architecture to achieve high resource utilization as well as low latency. It improves over an existing layered design that processes additional codewords in parallel to increase uti… ▽ More The GPU as a digital signal processing accelerator for cloud RAN is investigated. A new design for a 5G NR low density parity check code decoder running on a GPU is presented. The algorithm is flexibly adaptable to GPU architecture to achieve high resource utilization as well as low latency. It improves over an existing layered design that processes additional codewords in parallel to increase utilization. In comparison to a decoder implemented on a FPGA (757K gate), the new GPU (24 core) decoder has 3X higher throughput. The GPU decoder exhibits 3 to 5X lower decoding power efficiency, as typical of a general-purpose processor. Thus, GPUs may find application as cloud accelerators where rapid deployment and flexibility are prioritized over decoding power efficiency. △ Less

Submitted 11 September, 2020; originally announced September 2020.

Comments: 4 pages, 3 figures

Toward Fine-grained Facial Expression Manipulation

Authors: Jun Ling, Han Xue, Li Song, Shuhui Yang, Rong Xie, Xiao Gu

Abstract: Facial expression manipulation aims at editing facial expression with a given condition. Previous methods edit an input image under the guidance of a discrete emotion label or absolute condition (e.g., facial action units) to possess the desired expression. However, these methods either suffer from changing condition-irrelevant regions or are inefficient for fine-grained editing. In this study, we… ▽ More Facial expression manipulation aims at editing facial expression with a given condition. Previous methods edit an input image under the guidance of a discrete emotion label or absolute condition (e.g., facial action units) to possess the desired expression. However, these methods either suffer from changing condition-irrelevant regions or are inefficient for fine-grained editing. In this study, we take these two objectives into consideration and propose a novel method. First, we replace continuous absolute condition with relative condition, specifically, relative action units. With relative action units, the generator learns to only transform regions of interest which are specified by non-zero-valued relative AUs. Second, our generator is built on U-Net but strengthened by Multi-Scale Feature Fusion (MSF) mechanism for high-quality expression editing purposes. Extensive experiments on both quantitative and qualitative evaluation demonstrate the improvements of our proposed approach compared to the state-of-the-art expression editing methods. Code is available at \url{https://github.com/junleen/Expression-manipulator}. △ Less

Submitted 4 December, 2020; v1 submitted 7 April, 2020; originally announced April 2020.

Machine Learning based prediction of noncentrosymmetric crystal materials

Authors: Yuqi Song, Joseph Lindsay, Yong Zhao, Alireza Nasiri, Steph-Yves Louis, Jie Ling, Ming Hu, Jianjun Hu

Abstract: Noncentrosymmetric materials play a critical role in many important applications such as laser technology, communication systems,quantum computing, cybersecurity, and etc. However, the experimental discovery of new noncentrosymmetric materials is extremely difficult. Here we present a machine learning model that could predict whether the composition of a potential crystalline structure would be ce… ▽ More Noncentrosymmetric materials play a critical role in many important applications such as laser technology, communication systems,quantum computing, cybersecurity, and etc. However, the experimental discovery of new noncentrosymmetric materials is extremely difficult. Here we present a machine learning model that could predict whether the composition of a potential crystalline structure would be centrosymmetric or not. By evaluating a diverse set of composition features calculated using matminer featurizer package coupled with different machine learning algorithms, we find that Random Forest Classifiers give the best performance for noncentrosymmetric material prediction, reaching an accuracy of 84.8% when evaluated with 10 fold cross-validation on the dataset with 82,506 samples extracted from Materials Project. A random forest model trained with materials with only 3 elements gives even higher accuracy of 86.9%. We apply our ML model to screen potential noncentrosymmetric materials from 2,000,000 hypothetical materials generated by our inverse design engine and report the top 20 candidate noncentrosymmetric materials with 2 to 4 elements and top 20 borate candidates △ Less

Submitted 11 April, 2020; v1 submitted 25 February, 2020; originally announced February 2020.

Comments: 13 pages

Turbulent scalar flux in inclined jets in crossflow: counter gradient transport and deep learning modelling

Authors: Pedro M. Milani, Julia Ling, John K. Eaton

Abstract: A cylindrical and inclined jet in crossflow is studied under two distinct velocity ratios, $r=1$ and $r=2$, using highly resolved large eddy simulations (LES). First, an investigation of turbulent scalar mixing sheds light onto the previously observed but unexplained phenomenon of negative turbulent diffusivity. We identify two distinct types of counter gradient transport, prevalent in different r… ▽ More A cylindrical and inclined jet in crossflow is studied under two distinct velocity ratios, $r=1$ and $r=2$, using highly resolved large eddy simulations (LES). First, an investigation of turbulent scalar mixing sheds light onto the previously observed but unexplained phenomenon of negative turbulent diffusivity. We identify two distinct types of counter gradient transport, prevalent in different regions: the first, throughout the windward shear layer, is caused by cross-gradient transport; the second, close to the wall right after injection, is caused by non-local effects. Then, we propose a deep learning approach for modelling the turbulent scalar flux by adapting the tensor basis neural network previously developed to model Reynolds stresses (Ling et al. 2016a). This approach uses a deep neural network with embedded coordinate frame invariance to predict a tensorial turbulent diffusivity that is not explicitly available in the high fidelity data used for training. After ensuring analytically that the matrix diffusivity leads to a stable solution for the advection diffusion equation, we apply this approach in the inclined jets in crossflow under study. The results show significant improvement compared to a simple model, particularly where cross-gradient effects play an important role in turbulent mixing. The model proposed herein is not limited to jets in crossflow; it can be used in any turbulent flow where the Reynolds averaged transport of a scalar is considered. △ Less

Submitted 27 September, 2020; v1 submitted 13 January, 2020; originally announced January 2020.

Comments: 30 pages, 20 figures. Final version of the manuscript, accepted for publication in the Journal of Fluid Mechanics

Learning Cross-Context Entity Representations from Text

Authors: Jeffrey Ling, Nicholas FitzGerald, Zifei Shan, Livio Baldini Soares, Thibault Févry, David Weiss, Tom Kwiatkowski

Abstract: Language modeling tasks, in which words, or word-pieces, are predicted on the basis of a local context, have been very effective for learning word embeddings and context dependent representations of phrases. Motivated by the observation that efforts to code world knowledge into machine readable knowledge bases or human readable encyclopedias tend to be entity-centric, we investigate the use of a f… ▽ More Language modeling tasks, in which words, or word-pieces, are predicted on the basis of a local context, have been very effective for learning word embeddings and context dependent representations of phrases. Motivated by the observation that efforts to code world knowledge into machine readable knowledge bases or human readable encyclopedias tend to be entity-centric, we investigate the use of a fill-in-the-blank task to learn context independent representations of entities from the text contexts in which those entities were mentioned. We show that large scale training of neural models allows us to learn high quality entity representations, and we demonstrate successful results on four domains: (1) existing entity-level typing benchmarks, including a 64% error reduction over previous work on TypeNet (Murty et al., 2018); (2) a novel few-shot category reconstruction task; (3) existing entity linking benchmarks, where we match the state-of-the-art on CoNLL-Aida without linking-specific features and obtain a score of 89.8% on TAC-KBP 2010 without using any alias table, external knowledge base or in domain training data and (4) answering trivia questions, which uniquely identify entities. Our global entity representations encode fine-grained type categories, such as Scottish footballers, and can answer trivia questions such as: Who was the last inmate of Spandau jail in Berlin? △ Less

Submitted 11 January, 2020; originally announced January 2020.

Machine-learned metrics for predicting the likelihood of success in materials discovery

Authors: Yoolhee Kim, Edward Kim, Erin Antono, Bryce Meredig, Julia Ling

Abstract: Materials discovery is often compared to the challenge of finding a needle in a haystack. While much work has focused on accurately predicting the properties of candidate materials with machine learning (ML), which amounts to evaluating whether a given candidate is a piece of straw or a needle, less attention has been paid to a critical question: Are we searching in the right haystack? We refer to… ▽ More Materials discovery is often compared to the challenge of finding a needle in a haystack. While much work has focused on accurately predicting the properties of candidate materials with machine learning (ML), which amounts to evaluating whether a given candidate is a piece of straw or a needle, less attention has been paid to a critical question: Are we searching in the right haystack? We refer to the haystack as the design space for a particular materials discovery problem (i.e. the set of possible candidate materials to synthesize), and thus frame this question as one of design space selection. In this paper, we introduce two metrics, the Predicted Fraction of Improved Candidates (PFIC), and the Cumulative Maximum Likelihood of Improvement (CMLI), which we demonstrate can identify discovery-rich and discovery-poor design spaces, respectively. Using CMLI and PFIC together to identify optimal design spaces can significantly accelerate ML-driven materials discovery. △ Less

Submitted 27 November, 2019; v1 submitted 25 November, 2019; originally announced November 2019.

Comments: 13 pages, 10 figures, 2 tables

Assessing the Frontier: Active Learning, Model Accuracy, and Multi-objective Materials Discovery and Optimization

Authors: Zachary del Rosario, Matthias Rupp, Yoolhee Kim, Erin Antono, Julia Ling

Abstract: Discovering novel materials can be greatly accelerated by iterative machine learning-informed proposal of candidates---active learning. However, standard \emph{global-scope error} metrics for model quality are not predictive of discovery performance, and can be misleading. We introduce the notion of \emph{Pareto shell-scope error} to help judge the suitability of a model for proposing material can… ▽ More Discovering novel materials can be greatly accelerated by iterative machine learning-informed proposal of candidates---active learning. However, standard \emph{global-scope error} metrics for model quality are not predictive of discovery performance, and can be misleading. We introduce the notion of \emph{Pareto shell-scope error} to help judge the suitability of a model for proposing material candidates. Further, through synthetic cases and a thermoelectric dataset, we probe the relation between acquisition function fidelity and active learning performance. Results suggest novel diagnostic tools, as well as new insights for acquisition function design. △ Less

Submitted 27 January, 2020; v1 submitted 6 November, 2019; originally announced November 2019.

Comments: 17 pages, 10 figures

Performance of High-Mobility MIMO Communications with Doppler Diversity

Authors: Xiaoyun Hou, Jie Ling, Dongming Wang

Abstract: A rapid change of channels in high-speed mobile communications will lead to difficulties in channel estimation and tracking but can also provide Doppler diversity. In this paper, the performance of a multiple-input multiple-output system with pilot-assisted repetition coding and spatial multiplexing is studied. With minimum mean square error (MMSE) channel estimation, an equivalent channel model a… ▽ More A rapid change of channels in high-speed mobile communications will lead to difficulties in channel estimation and tracking but can also provide Doppler diversity. In this paper, the performance of a multiple-input multiple-output system with pilot-assisted repetition coding and spatial multiplexing is studied. With minimum mean square error (MMSE) channel estimation, an equivalent channel model and the corresponding system model are presented. Based on random matrix theory, asymptotic expressions of the normalized achievable sum rate of the linear receivers, such as the maximal ratio combining (MRC) receiver, MMSE receiver and MRC-like receiver, are derived. In addition, according to the symbol error rate of the MRC-like receiver, the maximum normalized Doppler diversity order and the minimum coding gain loss can be achieved when the repetition number and signal-to-noise ratio tend to infinity, and the corresponding conditions are derived. Based on the theoretical results, the impacts of different system configurations and channel parameters on the system performance are demonstrated. △ Less

Submitted 18 February, 2020; v1 submitted 14 October, 2019; originally announced October 2019.

Comments: 13 pages, 13 figures. This is the revised and accepted version with DOI added

Generalization of machine-learned turbulent heat flux models applied to film cooling flows

Authors: Pedro M. Milani, Julia Ling, John K. Eaton

Abstract: The design of film cooling systems relies heavily on Reynolds-Averaged Navier-Stokes (RANS) simulations, which solve for mean quantities and model all turbulent scales. Most turbulent heat flux models, which are based on isotropic diffusion with a fixed turbulent Prandtl number ($Pr_t$), fail to accurately predict heat transfer in film cooling flows. In the present work, machine learning models ar… ▽ More The design of film cooling systems relies heavily on Reynolds-Averaged Navier-Stokes (RANS) simulations, which solve for mean quantities and model all turbulent scales. Most turbulent heat flux models, which are based on isotropic diffusion with a fixed turbulent Prandtl number ($Pr_t$), fail to accurately predict heat transfer in film cooling flows. In the present work, machine learning models are trained to predict a non-uniform $Pr_t$ field, using various datasets as training sets. The ability of these models to generalize beyond the flows on which they were trained is explored. Furthermore, visualization techniques are employed to compare distinct datasets and to help explain the cross-validation results. △ Less

Submitted 7 October, 2019; originally announced October 2019.

Comments: Presented at ASME Turbo Expo 2019, accepted to the Journal of Turbomachinery

Algorithms for Manipulating Sequential Allocation

Authors: Mingyu Xiao, Jiaxing Ling

Abstract: Sequential allocation is a simple and widely studied mechanism to allocate indivisible items in turns to agents according to a pre-specified picking sequence of agents. At each turn, the current agent in the picking sequence picks its most preferred item among all items having not been allocated yet. This problem is well-known to be not strategyproof, i.e., an agent may get more utility by reporti… ▽ More Sequential allocation is a simple and widely studied mechanism to allocate indivisible items in turns to agents according to a pre-specified picking sequence of agents. At each turn, the current agent in the picking sequence picks its most preferred item among all items having not been allocated yet. This problem is well-known to be not strategyproof, i.e., an agent may get more utility by reporting an untruthful preference ranking of items. It arises the problem: how to find the best response of an agent? It is known that this problem is polynomially solvable for only two agents and NP-complete for arbitrary number of agents. The computational complexity of this problem with three agents was left as an open problem. In this paper, we give a novel algorithm that solves the problem in polynomial time for each fixed number of agents. We also show that an agent can always get at least half of its optimal utility by simply using its truthful preference as the response. △ Less

Submitted 15 September, 2019; originally announced September 2019.

Fusion of Detected Objects in Text for Visual Question Answering

Authors: Chris Alberti, Jeffrey Ling, Michael Collins, David Reitter

Abstract: To advance models of multimodal context, we introduce a simple yet powerful neural architecture for data that combines vision and natural language. The "Bounding Boxes in Text Transformer" (B2T2) also leverages referential information binding words to portions of the image in a single unified architecture. B2T2 is highly effective on the Visual Commonsense Reasoning benchmark (https://visualcommon… ▽ More To advance models of multimodal context, we introduce a simple yet powerful neural architecture for data that combines vision and natural language. The "Bounding Boxes in Text Transformer" (B2T2) also leverages referential information binding words to portions of the image in a single unified architecture. B2T2 is highly effective on the Visual Commonsense Reasoning benchmark (https://visualcommonsense.com), achieving a new state-of-the-art with a 25% relative reduction in error rate compared to published baselines and obtaining the best performance to date on the public leaderboard (as of May 22, 2019). A detailed ablation analysis shows that the early integration of the visual features into the text analysis is key to the effectiveness of the new architecture. A reference implementation of our models is provided (https://github.com/google-research/language/tree/master/language/question_answering/b2t2). △ Less

Submitted 3 November, 2019; v1 submitted 14 August, 2019; originally announced August 2019.

Matching the Blanks: Distributional Similarity for Relation Learning

Authors: Livio Baldini Soares, Nicholas FitzGerald, Jeffrey Ling, Tom Kwiatkowski

Abstract: General purpose relation extractors, which can model arbitrary relations, are a core aspiration in information extraction. Efforts have been made to build general purpose extractors that represent relations with their surface forms, or which jointly embed surface forms with relations from an existing knowledge graph. However, both of these approaches are limited in their ability to generalize. In… ▽ More General purpose relation extractors, which can model arbitrary relations, are a core aspiration in information extraction. Efforts have been made to build general purpose extractors that represent relations with their surface forms, or which jointly embed surface forms with relations from an existing knowledge graph. However, both of these approaches are limited in their ability to generalize. In this paper, we build on extensions of Harris' distributional hypothesis to relations, as well as recent advances in learning text representations (specifically, BERT), to build task agnostic relation representations solely from entity-linked text. We show that these representations significantly outperform previous work on exemplar based relation extraction (FewRel) even without using any of that task's training data. We also show that models initialized with our task agnostic representations, and then tuned on supervised relation extraction datasets, significantly outperform the previous methods on SemEval 2010 Task 8, KBP37, and TACRED. △ Less

Submitted 7 June, 2019; originally announced June 2019.

Comments: To appear at ACL 2019

A Generic Multi-Projection-Center Model and Calibration Method for Light Field Cameras

Authors: Qi Zhang, Chunping Zhang, Jinbo Ling, Qing Wang, Jingyi Yu

Abstract: Light field cameras can capture both spatial and angular information of light rays, enabling 3D reconstruction by a single exposure. The geometry of 3D reconstruction is affected by intrinsic parameters of a light field camera significantly. In the paper, we propose a multi-projection-center (MPC) model with 6 intrinsic parameters to characterize light field cameras based on traditional two-parall… ▽ More Light field cameras can capture both spatial and angular information of light rays, enabling 3D reconstruction by a single exposure. The geometry of 3D reconstruction is affected by intrinsic parameters of a light field camera significantly. In the paper, we propose a multi-projection-center (MPC) model with 6 intrinsic parameters to characterize light field cameras based on traditional two-parallel-plane (TPP) representation. The MPC model can generally parameterize light field in different imaging formations, including conventional and focused light field cameras. By the constraints of 4D ray and 3D geometry, a 3D projective transformation is deduced to describe the relationship between geometric structure and the MPC coordinates. Based on the MPC model and projective transformation, we propose a calibration algorithm to verify our light field camera model. Our calibration method includes a close-form solution and a non-linear optimization by minimizing re-projection errors. Experimental results on both simulated and real scene data have verified the performance of our algorithm. △ Less

Submitted 7 August, 2018; originally announced August 2018.

Comments: accepted by T-PAMI

Overcoming data scarcity with transfer learning

Authors: Maxwell L. Hutchinson, Erin Antono, Brenna M. Gibbons, Sean Paradiso, Julia Ling, Bryce Meredig

Abstract: Despite increasing focus on data publication and discovery in materials science and related fields, the global view of materials data is highly sparse. This sparsity encourages training models on the union of multiple datasets, but simple unions can prove problematic as (ostensibly) equivalent properties may be measured or computed differently depending on the data source. These hidden contextual… ▽ More Despite increasing focus on data publication and discovery in materials science and related fields, the global view of materials data is highly sparse. This sparsity encourages training models on the union of multiple datasets, but simple unions can prove problematic as (ostensibly) equivalent properties may be measured or computed differently depending on the data source. These hidden contextual differences introduce irreducible errors into analyses, fundamentally limiting their accuracy. Transfer learning, where information from one dataset is used to inform a model on another, can be an effective tool for bridging sparse data while preserving the contextual differences in the underlying measurements. Here, we describe and compare three techniques for transfer learning: multi-task, difference, and explicit latent variable architectures. We show that difference architectures are most accurate in the multi-fidelity case of mixed DFT and experimental band gaps, while multi-task most improves classification performance of color with band gaps. For activation energies of steps in NO reduction, the explicit latent variable method is not only the most accurate, but also enjoys cancellation of errors in functions that depend on multiple tasks. These results motivate the publication of high quality materials datasets that encode transferable information, independent of industrial or academic interest in the particular labels, and encourage further development and application of transfer learning methods to materials informatics problems. △ Less

Submitted 2 November, 2017; originally announced November 2017.

Building Data-driven Models with Microstructural Images: Generalization and Interpretability

Authors: Julia Ling, Maxwell Hutchinson, Erin Antono, Brian DeCost, Elizabeth A. Holm, Bryce Meredig

Abstract: As data-driven methods rise in popularity in materials science applications, a key question is how these machine learning models can be used to understand microstructure. Given the importance of process-structure-property relations throughout materials science, it seems logical that models that can leverage microstructural data would be more capable of predicting property information. While there… ▽ More As data-driven methods rise in popularity in materials science applications, a key question is how these machine learning models can be used to understand microstructure. Given the importance of process-structure-property relations throughout materials science, it seems logical that models that can leverage microstructural data would be more capable of predicting property information. While there have been some recent attempts to use convolutional neural networks to understand microstructural images, these early studies have focused only on which featurizations yield the highest machine learning model accuracy for a single data set. This paper explores the use of convolutional neural networks for classifying microstructure with a more holistic set of objectives in mind: generalization between data sets, number of features required, and interpretability. △ Less

Submitted 1 November, 2017; originally announced November 2017.

Compressed Sensing Algorithms for OFDM Channel Estimation

Authors: Jonathan Ling, Dmitry Chizhik, A. Tulino, Inaki Esnaola

Abstract: Radio channels are typically sparse in the delay domain, and ideal for compressed sensing. A new compressed sensing algorithm called eX-OMP is developed that yields performance similar to that of the optimal MMSE estimator. The new algorithm relies on a small amount additional data. Both eX-OMP and the MMSE estimator adaptively balance channel tracking and noise reduction. They perform better than… ▽ More Radio channels are typically sparse in the delay domain, and ideal for compressed sensing. A new compressed sensing algorithm called eX-OMP is developed that yields performance similar to that of the optimal MMSE estimator. The new algorithm relies on a small amount additional data. Both eX-OMP and the MMSE estimator adaptively balance channel tracking and noise reduction. They perform better than simple estimators such as the linear-interpolator which fix this trade-off a priori. Some wideband measurements are examined, and the channels are found to be represented by a few delays. △ Less

Submitted 22 December, 2016; originally announced December 2016.

Comments: Patent no: US8929390 Methods And Apparatuses For Channel Estimation In Wireless Networks

Practical LTE and Wi-Fi Coexistence Techniques beyond LBT

Authors: Jonathan Ling, David Lopez-Perez, Mohammad R. Khawer

Abstract: Coexistence with Wi-Fi is the key issue for unlicensed band LTE. The main coexistence mechanism is Listen-Before-Talk, whereby radio frequency energy is sensed over a short period of time and compared to a threshold. Given the default energy thresholds, the energy sensing range is actually much less than the cell range. Both technologies can experience collisions due to transmission being below en… ▽ More Coexistence with Wi-Fi is the key issue for unlicensed band LTE. The main coexistence mechanism is Listen-Before-Talk, whereby radio frequency energy is sensed over a short period of time and compared to a threshold. Given the default energy thresholds, the energy sensing range is actually much less than the cell range. Both technologies can experience collisions due to transmission being below energy detection threshold. Currently Wi-Fi is agnostic of LTE presence in the unlicensed spectrum. To improve coexistence a communications channel via relaying is proposed to be used by the unlicensed band LTE, to announce its presence on an unlicensed channel. Legacy Wi-Fi APs may be programmed to interpret and respond by firmware upgrade at the AP to enhance its channel selection algorithm. Higher performance for both networks is demonstrated via more effective radio frequency channel selection and adaptive energy detection thresholding. △ Less

Submitted 19 December, 2016; originally announced December 2016.

Comments: Submitted to IEEE Com Mag

Journal ref: in IEEE Communications Magazine, vol. 55, no. 10, pp. 127-133, Oct. 2017

Image-to-Markup Generation with Coarse-to-Fine Attention

Authors: Yuntian Deng, Anssi Kanervisto, Jeffrey Ling, Alexander M. Rush

Abstract: We present a neural encoder-decoder model to convert images into presentational markup based on a scalable coarse-to-fine attention mechanism. Our method is evaluated in the context of image-to-LaTeX generation, and we introduce a new dataset of real-world rendered mathematical expressions paired with LaTeX markup. We show that unlike neural OCR techniques using CTC-based models, attention-based a… ▽ More We present a neural encoder-decoder model to convert images into presentational markup based on a scalable coarse-to-fine attention mechanism. Our method is evaluated in the context of image-to-LaTeX generation, and we introduce a new dataset of real-world rendered mathematical expressions paired with LaTeX markup. We show that unlike neural OCR techniques using CTC-based models, attention-based approaches can tackle this non-standard OCR task. Our approach outperforms classical mathematical OCR systems by a large margin on in-domain rendered data, and, with pretraining, also performs well on out-of-domain handwritten data. To reduce the inference complexity associated with the attention-based approaches, we introduce a new coarse-to-fine attention layer that selects a support region before applying attention. △ Less

Submitted 13 June, 2017; v1 submitted 16 September, 2016; originally announced September 2016.

Comments: Accepted by ICML 2017

LWIP and Wi-Fi Boost Link Management

Authors: David Lopez-Perez, Jonathan Ling, Bong Ho Kim, Vasudevan Subramanian, Satish Kanugovi, Ming Ding

Abstract: 3GPP LWIP Release 13 technology and its prestandard version Wi-Fi Boost have recently emerged as an efficient LTE and Wi-Fi integration at the IP layer, allowing uplink on LTE and downlink on Wi-Fi. This solves all the contention problems of Wi-Fi and allows an optimum usage of the unlicensed band for downlink. In this paper, we present a new feature of Wi-Fi Boost, its radio link management, whic… ▽ More 3GPP LWIP Release 13 technology and its prestandard version Wi-Fi Boost have recently emerged as an efficient LTE and Wi-Fi integration at the IP layer, allowing uplink on LTE and downlink on Wi-Fi. This solves all the contention problems of Wi-Fi and allows an optimum usage of the unlicensed band for downlink. In this paper, we present a new feature of Wi-Fi Boost, its radio link management, which allows to steer the downlink traffic between both LTE and Wi-Fi upon congestion detection in an intelligent manner. This customised congestion detection algorithm is based on IP probing, and can work with any Wi-Fi access point. Simulation results in a typical enterprise scenario show that LWIP R13 and Wi-Fi Boost can enhance network performance up to 5x and 6x over LTE-only, and 4x and 5x over Wi-Fi only networks, respectively, and that the the proposed radio link management can further improve Wi-Fi Boost performance over LWIP R13 up to 19 %. Based on the promising results, this paper suggests to enhance LWIP R13 user feedback in future LTE releases. △ Less

Submitted 27 July, 2016; originally announced July 2016.

Comments: in IEEE Globecom, Washington DC, 2016

Wi-Fi Scaling and Performance in Dense Frequency Planned Networks

Authors: Jonathan Ling, Howard Huang, Reinaldo Valenzuela

Abstract: The capacity scaling of a Wi-Fi network with fixed transmit power is investigated. The 802.11 MAC appears to impose a hard limit on the area capacity by enforcing single transmissions in overlapped cell coverage areas. However due to the difficulty reading the packet header when transmissions either overlap or are faded, stations in different cells may lose synchronization with each other. A large… ▽ More The capacity scaling of a Wi-Fi network with fixed transmit power is investigated. The 802.11 MAC appears to impose a hard limit on the area capacity by enforcing single transmissions in overlapped cell coverage areas. However due to the difficulty reading the packet header when transmissions either overlap or are faded, stations in different cells may lose synchronization with each other. A large frequency planned network with full propagation impairments of fast and slow fading is simulated in NS-3 with an upgraded receiver model supporting packet capture. Downlink Wi-Fi performance is compared to an equivalent LTE network on the same cell centers. Results show that Wi-Fi downlink throughput grows with densification limited by decreasing SINR and partially operational virtual carrier sense. LTE area capacity grows faster as it is only constrained by SINR. △ Less

Submitted 19 December, 2016; v1 submitted 21 April, 2016; originally announced April 2016.

Comments: Submitted to IEEE ICC 2017

Online Algorithms Modeled After Mousehunt

Authors: Jeffrey Ling, Kai Xiao, Dai Yang

Abstract: In this paper we study a variety of novel online algorithm problems inspired by the game Mousehunt. We consider a number of basic models that approximate the game, and we provide solutions to these models using Markov Decision Processes, deterministic online algorithms, and randomized online algorithms. We analyze these solutions' performance by deriving results on their competitive ratios. In this paper we study a variety of novel online algorithm problems inspired by the game Mousehunt. We consider a number of basic models that approximate the game, and we provide solutions to these models using Markov Decision Processes, deterministic online algorithms, and randomized online algorithms. We analyze these solutions' performance by deriving results on their competitive ratios. △ Less

Submitted 7 January, 2015; originally announced January 2015.

Comments: 14 pages, 2 figures

Enhanced capacity & coverage by Wi-Fi LTE Integration

Authors: Jonathan Ling, Satish Kanugovi, Subramanian Vasudevan, A Krishna Pramod

Abstract: Wi-Fi provides cost-effective data capacity at hotspots in conjunction with broadband cellular networks. The hotspots are required to capture a large number of users and provide high data rates. Data rates, over the Wi-Fi interface, are influenced by the media access protocol, which loses throughput due to delays and unintended collisions when a large number of users are active. The hotspot range… ▽ More Wi-Fi provides cost-effective data capacity at hotspots in conjunction with broadband cellular networks. The hotspots are required to capture a large number of users and provide high data rates. Data rates, over the Wi-Fi interface, are influenced by the media access protocol, which loses throughput due to delays and unintended collisions when a large number of users are active. The hotspot range which determines the number of users, that can associate, is limited by the lower power of the client rather than the access point. By diverting the traffic destined to the access point via another access network, both range and efficiency can be improved. This uplink redirection or diversion is achieved by simultaneous use of the Wi-Fi and LTE radio interfaces. Three options - loose, tight, and hybrid integration are presented towards providing enhanced capacity and coverage. △ Less

Submitted 12 June, 2014; originally announced June 2014.

Comments: Submitted as a candidate article for IEEE Communications Magazine, Topic: The Future of Wi-Fi (November 2014)