-
Retrofitting Temporal Graph Neural Networks with Transformer
Authors:
Qiang Huang,
Xiao Yan,
Xin Wang,
Susie Xi Rao,
Zhichao Han,
Fangcheng Fu,
Wentao Zhang,
Jiawei Jiang
Abstract:
Temporal graph neural networks (TGNNs) outperform regular GNNs by incorporating time information into graph-based operations. However, TGNNs adopt specialized models (e.g., TGN, TGAT, and APAN ) and require tailored training frameworks (e.g., TGL and ETC). In this paper, we propose TF-TGN, which uses Transformer decoder as the backbone model for TGNN to enjoy Transformer's codebase for efficient t…
▽ More
Temporal graph neural networks (TGNNs) outperform regular GNNs by incorporating time information into graph-based operations. However, TGNNs adopt specialized models (e.g., TGN, TGAT, and APAN ) and require tailored training frameworks (e.g., TGL and ETC). In this paper, we propose TF-TGN, which uses Transformer decoder as the backbone model for TGNN to enjoy Transformer's codebase for efficient training. In particular, Transformer achieves tremendous success for language modeling, and thus the community developed high-performance kernels (e.g., flash-attention and memory-efficient attention) and efficient distributed training schemes (e.g., PyTorch FSDP, DeepSpeed, and Megatron-LM). We observe that TGNN resembles language modeling, i.e., the message aggregation operation between chronologically occurring nodes and their temporal neighbors in TGNNs can be structured as sequence modeling. Beside this similarity, we also incorporate a series of algorithm designs including suffix infilling, temporal graph attention with self-loop, and causal masking self-attention to make TF-TGN work. During training, existing systems are slow in transforming the graph topology and conducting graph sampling. As such, we propose methods to parallelize the CSR format conversion and graph sampling. We also adapt Transformer codebase to train TF-TGN efficiently with multiple GPUs. We experiment with 9 graphs and compare with 2 state-of-the-art TGNN training frameworks. The results show that TF-TGN can accelerate training by over 2.20 while providing comparable or even superior accuracy to existing SOTA TGNNs. TF-TGN is available at https://github.com/qianghuangwhu/TF-TGN.
△ Less
Submitted 10 September, 2024; v1 submitted 9 September, 2024;
originally announced September 2024.
-
Efficient Multi-Task Large Model Training via Data Heterogeneity-aware Model Management
Authors:
Yujie Wang,
Shenhan Zhu,
Fangcheng Fu,
Xupeng Miao,
Jie Zhang,
Juan Zhu,
Fan Hong,
Yong Li,
Bin Cui
Abstract:
Recent foundation models are capable of handling multiple machine learning (ML) tasks and multiple data modalities with the unified base model structure and several specialized model components. However, the development of such multi-task (MT) multi-modal (MM) models poses significant model management challenges to existing training systems. Due to the sophisticated model architecture and the hete…
▽ More
Recent foundation models are capable of handling multiple machine learning (ML) tasks and multiple data modalities with the unified base model structure and several specialized model components. However, the development of such multi-task (MT) multi-modal (MM) models poses significant model management challenges to existing training systems. Due to the sophisticated model architecture and the heterogeneous workloads of different ML tasks and data modalities, training these models usually requires massive GPU resources and suffers from sub-optimal system efficiency.
In this paper, we investigate how to achieve high-performance training of large-scale MT MM models through data heterogeneity-aware model management optimization. The key idea is to decompose the model execution into stages and address the joint optimization problem sequentially, including both heterogeneity-aware workload parallelization and dependency-driven execution scheduling. Based on this, we build a prototype system and evaluate it on various large MT MM models. Experiments demonstrate the superior performance and efficiency of our system, with speedup ratio up to 71% compared to state-of-the-art training systems.
△ Less
Submitted 5 September, 2024;
originally announced September 2024.
-
Transforming Time-Varying to Static Channels: The Power of Fluid Antenna Mobility
Authors:
Weidong Li,
Haifan Yin,
Fanpo Fu,
Yandi Cao,
Merouane Debbah
Abstract:
This paper addresses the mobility problem with the assistance of fluid antenna (FA) on the user equipment (UE) side. We propose a matrix pencil-based moving port (MPMP) prediction method, which may transform the time-varying channel to a static channel by timely sliding the liquid. Different from the existing channel prediction method, we design a moving port selection method, which is the first a…
▽ More
This paper addresses the mobility problem with the assistance of fluid antenna (FA) on the user equipment (UE) side. We propose a matrix pencil-based moving port (MPMP) prediction method, which may transform the time-varying channel to a static channel by timely sliding the liquid. Different from the existing channel prediction method, we design a moving port selection method, which is the first attempt to transform the channel prediction to the port prediction by exploiting the movability of FA. Theoretical analysis shows that for the line-ofsight (LoS) channel, the prediction error of our proposed MPMP method may converge to zero, as the number of BS antennas and the port density of the FA are large enough. For a multi-path channel, we also derive the upper and lower bounds of the prediction error when the number of paths is large enough. When the UEs move at a speed of 60 or 120 km/h, simulation results show that, with the assistance of FA, our proposed MPMP method performs better than the existing channel prediction method.
△ Less
Submitted 9 August, 2024; v1 submitted 8 August, 2024;
originally announced August 2024.
-
PQCache: Product Quantization-based KVCache for Long Context LLM Inference
Authors:
Hailin Zhang,
Xiaodong Ji,
Yilin Chen,
Fangcheng Fu,
Xupeng Miao,
Xiaonan Nie,
Weipeng Chen,
Bin Cui
Abstract:
As the field of Large Language Models (LLMs) continues to evolve, the context length in inference is steadily growing. Key-Value Cache (KVCache), a crucial component in LLM inference, has now become the primary memory bottleneck due to limited GPU memory. Current methods selectively determine suitable keys and values for self-attention computation in LLMs to address the issue. However, they either…
▽ More
As the field of Large Language Models (LLMs) continues to evolve, the context length in inference is steadily growing. Key-Value Cache (KVCache), a crucial component in LLM inference, has now become the primary memory bottleneck due to limited GPU memory. Current methods selectively determine suitable keys and values for self-attention computation in LLMs to address the issue. However, they either fall short in maintaining model quality or result in high serving latency. Drawing inspiration from advanced embedding retrieval techniques used in the database community, we consider the storage and searching of KVCache as a typical embedding retrieval problem. We propose PQCache, which employs Product Quantization (PQ) to manage KVCache, maintaining model quality while ensuring low serving latency. During the prefilling phase, we apply PQ to tokens' keys for each LLM layer and head. During the autoregressive decoding phase, for each newly generated token, we first identify important tokens through Maximum Inner-Product Search (MIPS) using PQ codes and centroids, then fetch the corresponding key-value pairs for self-attention computation. Through meticulous design of overlapping and caching, we minimize any additional computation and communication overhead during both phases. Extensive experiments show that PQCache achieves both effectiveness and efficiency. It maintains model quality even when only 1/5 of the tokens are involved in attention, while attaining acceptable system latency.
△ Less
Submitted 1 July, 2024;
originally announced July 2024.
-
SigDLA: A Deep Learning Accelerator Extension for Signal Processing
Authors:
Fangfa Fu,
Wenyu Zhang,
Zesong Jiang,
Zhiyu Zhu,
Guoyu Li,
Bing Yang,
Cheng Liu,
Liyi Xiao,
Jinxiang Wang,
Huawei Li,
Xiaowei Li
Abstract:
Deep learning and signal processing are closely correlated in many IoT scenarios such as anomaly detection to empower intelligence of things. Many IoT processors utilize digital signal processors (DSPs) for signal processing and build deep learning frameworks on this basis. While deep learning is usually much more computing-intensive than signal processing, the computing efficiency of deep learnin…
▽ More
Deep learning and signal processing are closely correlated in many IoT scenarios such as anomaly detection to empower intelligence of things. Many IoT processors utilize digital signal processors (DSPs) for signal processing and build deep learning frameworks on this basis. While deep learning is usually much more computing-intensive than signal processing, the computing efficiency of deep learning on DSPs is limited due to the lack of native hardware support. In this case, we present a contrary strategy and propose to enable signal processing on top of a classical deep learning accelerator (DLA). With the observation that irregular data patterns such as butterfly operations in FFT are the major barrier that hinders the deployment of signal processing on DLAs, we propose a programmable data shuffling fabric and have it inserted between the input buffer and computing array of DLAs such that the irregular data is reorganized and the processing is converted to be regular. With the online data shuffling, the proposed architecture, SigDLA, can adapt to various signal processing tasks without affecting the deep learning processing. Moreover, we build a reconfigurable computing array to suit the various data width requirements of both signal processing and deep learning. According to our experiments, SigDLA achieves an average performance speedup of 4.4$\times$, 1.4$\times$, and 1.52$\times$, and average energy reduction of 4.82$\times$, 3.27$\times$, and 2.15$\times$ compared to an embedded ARM processor with customized DSP instructions, a DSP processor, and an independent DSP-DLA architecture respectively with 17% more chip area over the original DLAs.
△ Less
Submitted 17 July, 2024;
originally announced July 2024.
-
Efficiently Training 7B LLM with 1 Million Sequence Length on 8 GPUs
Authors:
Pinxue Zhao,
Hailin Zhang,
Fangcheng Fu,
Xiaonan Nie,
Qibin Liu,
Fang Yang,
Yuanbo Peng,
Dian Jiao,
Shuaipeng Li,
Jinbao Xue,
Yangyu Tao,
Bin Cui
Abstract:
Nowadays, Large Language Models (LLMs) have been trained using extended context lengths to foster more creative applications. However, long context training poses great challenges considering the constraint of GPU memory. It not only leads to substantial activation memory consumption during training, but also incurs considerable memory fragmentation. To facilitate long context training, existing f…
▽ More
Nowadays, Large Language Models (LLMs) have been trained using extended context lengths to foster more creative applications. However, long context training poses great challenges considering the constraint of GPU memory. It not only leads to substantial activation memory consumption during training, but also incurs considerable memory fragmentation. To facilitate long context training, existing frameworks have adopted strategies such as recomputation and various forms of parallelisms. Nevertheless, these techniques rely on redundant computation or extensive communication, resulting in low Model FLOPS Utilization (MFU). In this paper, we propose MEMO, a novel LLM training framework designed for fine-grained activation memory management. Given the quadratic scaling of computation and linear scaling of memory with sequence lengths when using FlashAttention, we offload memory-consuming activations to CPU memory after each layer's forward pass and fetch them during the backward pass. To maximize the swapping of activations without hindering computation, and to avoid exhausting limited CPU memory, we implement a token-wise activation recomputation and swapping mechanism. Furthermore, we tackle the memory fragmentation issue by employing a bi-level Mixed Integer Programming (MIP) approach, optimizing the reuse of memory across transformer layers. Empirical results demonstrate that MEMO achieves an average of 2.42x and 2.26x MFU compared to Megatron-LM and DeepSpeed, respectively. This improvement is attributed to MEMO's ability to minimize memory fragmentation, reduce recomputation and intensive communication, and circumvent the delays associated with the memory reorganization process due to fragmentation. By leveraging fine-grained activation memory management, MEMO facilitates efficient training of 7B LLM with 1 million sequence length on just 8 A800 GPUs, achieving an MFU of 52.30%.
△ Less
Submitted 16 July, 2024;
originally announced July 2024.
-
Chromosomal Structural Abnormality Diagnosis by Homologous Similarity
Authors:
Juren Li,
Fanzhe Fu,
Ran Wei,
Yifei Sun,
Zeyu Lai,
Ning Song,
Xin Chen,
Yang Yang
Abstract:
Pathogenic chromosome abnormalities are very common among the general population. While numerical chromosome abnormalities can be quickly and precisely detected, structural chromosome abnormalities are far more complex and typically require considerable efforts by human experts for identification. This paper focuses on investigating the modeling of chromosome features and the identification of chr…
▽ More
Pathogenic chromosome abnormalities are very common among the general population. While numerical chromosome abnormalities can be quickly and precisely detected, structural chromosome abnormalities are far more complex and typically require considerable efforts by human experts for identification. This paper focuses on investigating the modeling of chromosome features and the identification of chromosomes with structural abnormalities. Most existing data-driven methods concentrate on a single chromosome and consider each chromosome independently, overlooking the crucial aspect of homologous chromosomes. In normal cases, homologous chromosomes share identical structures, with the exception that one of them is abnormal. Therefore, we propose an adaptive method to align homologous chromosomes and diagnose structural abnormalities through homologous similarity. Inspired by the process of human expert diagnosis, we incorporate information from multiple pairs of homologous chromosomes simultaneously, aiming to reduce noise disturbance and improve prediction performance. Extensive experiments on real-world datasets validate the effectiveness of our model compared to baselines.
△ Less
Submitted 11 July, 2024;
originally announced July 2024.
-
Unbending strategies shepherd cooperation and suppress extortion in spatial populations
Authors:
Zijie Chen,
Yuxin Geng,
Xingru Chen,
Feng Fu
Abstract:
Evolutionary game dynamics on networks typically consider the competition among simple strategies such as cooperation and defection in the Prisoner's Dilemma and summarize the effect of population structure as network reciprocity. However, it remains largely unknown regarding the evolutionary dynamics involving multiple powerful strategies typically considered in repeated games, such as the zero-d…
▽ More
Evolutionary game dynamics on networks typically consider the competition among simple strategies such as cooperation and defection in the Prisoner's Dilemma and summarize the effect of population structure as network reciprocity. However, it remains largely unknown regarding the evolutionary dynamics involving multiple powerful strategies typically considered in repeated games, such as the zero-determinant (ZD) strategies that are able to enforce a linear payoff relationship between them and their co-players. Here, we consider the evolutionary dynamics of always cooperate (AllC), extortionate ZD (extortioners), and unbending players in lattice populations based on the commonly used death-birth updating. Out of the class of unbending strategies, we consider a particular candidate, PSO Gambler, a machine-learning-optimized memory-one strategy, which can foster reciprocal cooperation and fairness among extortionate players. We derive analytical results under weak selection and rare mutations, including pairwise fixation probabilities and long-term frequencies of strategies. In the absence of the third unbending type, extortioners can achieve a half-half split in equilibrium with unconditional cooperators for sufficiently large extortion factors. However, the presence of unbending players fundamentally changes the dynamics and tilts the system to favor unbending cooperation. Most surprisingly, extortioners cannot dominate at all regardless of how large their extortion factor is, and the long-term frequency of unbending players is maintained almost as a constant. Our analytical method is applicable to studying the evolutionary dynamics of multiple strategies in structured populations. Our work provides insights into the interplay between network reciprocity and direct reciprocity, revealing the role of unbending strategies in enforcing fairness and suppressing extortion.
△ Less
Submitted 29 May, 2024;
originally announced May 2024.
-
Evolutionary game dynamics with environmental feedback in a network with two communities
Authors:
Katherine Betz,
Feng Fu,
Naoki Masuda
Abstract:
Recent developments of eco-evolutionary models have shown that evolving feedbacks between behavioral strategies and the environment of game interactions, leading to changes in the underlying payoff matrix, can impact the underlying population dynamics in various manners. We propose and analyze an eco-evolutionary game dynamics model on a network with two communities such that players interact with…
▽ More
Recent developments of eco-evolutionary models have shown that evolving feedbacks between behavioral strategies and the environment of game interactions, leading to changes in the underlying payoff matrix, can impact the underlying population dynamics in various manners. We propose and analyze an eco-evolutionary game dynamics model on a network with two communities such that players interact with other players in the same community and those in the opposite community at different rates. In our model, we consider two-person matrix games with pairwise interactions occurring on individual edges and assume that the environmental state depends on edges rather than on nodes or being globally shared in the population. We analytically determine the equilibria and their stability under a symmetric population structure assumption, and we also numerically study the replicator dynamics of the general model. The model shows rich dynamical behavior, such as multiple transcritical bifurcations, multistability, and anti-synchronous oscillations. Our work offers insights into understanding how the presence of community structure impacts the eco-evolutionary dynamics within and between niches.
△ Less
Submitted 7 June, 2024; v1 submitted 25 April, 2024;
originally announced April 2024.
-
Sentiment-oriented Transformer-based Variational Autoencoder Network for Live Video Commenting
Authors:
Fengyi Fu,
Shancheng Fang,
Weidong Chen,
Zhendong Mao
Abstract:
Automatic live video commenting is with increasing attention due to its significance in narration generation, topic explanation, etc. However, the diverse sentiment consideration of the generated comments is missing from the current methods. Sentimental factors are critical in interactive commenting, and lack of research so far. Thus, in this paper, we propose a Sentiment-oriented Transformer-base…
▽ More
Automatic live video commenting is with increasing attention due to its significance in narration generation, topic explanation, etc. However, the diverse sentiment consideration of the generated comments is missing from the current methods. Sentimental factors are critical in interactive commenting, and lack of research so far. Thus, in this paper, we propose a Sentiment-oriented Transformer-based Variational Autoencoder (So-TVAE) network which consists of a sentiment-oriented diversity encoder module and a batch attention module, to achieve diverse video commenting with multiple sentiments and multiple semantics. Specifically, our sentiment-oriented diversity encoder elegantly combines VAE and random mask mechanism to achieve semantic diversity under sentiment guidance, which is then fused with cross-modal features to generate live video comments. Furthermore, a batch attention module is also proposed in this paper to alleviate the problem of missing sentimental samples, caused by the data imbalance, which is common in live videos as the popularity of videos varies. Extensive experiments on Livebot and VideoIC datasets demonstrate that the proposed So-TVAE outperforms the state-of-the-art methods in terms of the quality and diversity of generated comments. Related code is available at https://github.com/fufy1024/So-TVAE.
△ Less
Submitted 19 April, 2024;
originally announced April 2024.
-
Association schemes arising from non-weakly regular bent functions
Authors:
Yadi Wei,
Jiaxin Wang,
Fang-Wei Fu
Abstract:
Association schemes play an important role in algebraic combinatorics and have important applications in coding theory, graph theory and design theory. The methods to construct association schemes by using bent functions have been extensively studied. Recently, in [13], {Ö}zbudak and Pelen constructed infinite families of symmetric association schemes of classes $5$ and $6$ by using ternary non-we…
▽ More
Association schemes play an important role in algebraic combinatorics and have important applications in coding theory, graph theory and design theory. The methods to construct association schemes by using bent functions have been extensively studied. Recently, in [13], {Ö}zbudak and Pelen constructed infinite families of symmetric association schemes of classes $5$ and $6$ by using ternary non-weakly regular bent functions.They also stated that constructing $2p$-class association schemes from $p$-ary non-weakly regular bent functions is an interesting problem, where $p>3$ is an odd prime. In this paper, using non-weakly regular bent functions, we construct infinite families of symmetric association schemes of classes $2p$, $(2p+1)$ and $\frac{3p+1}{2}$ for any odd prime $p$. Fusing those association schemes, we also obtain $t$-class symmetric association schemes, where $t=4,5,6,7$. In addition, we give the sufficient and necessary conditions for the partitions $P$, $D$, $T$, $U$ and $V$ (defined in this paper) to induce symmetric association schemes.
△ Less
Submitted 13 June, 2024; v1 submitted 8 April, 2024;
originally announced April 2024.
-
Self-Orthogonal Codes from Vectorial Dual-Bent Functions
Authors:
Jiaxin Wang,
Yadi Wei,
Fang-Wei Fu,
Juan Li
Abstract:
Self-orthogonal codes are a significant class of linear codes in coding theory and have attracted a lot of attention. In \cite{HLL2023Te,LH2023Se}, $p$-ary self-orthogonal codes were constructed by using $p$-ary weakly regular bent functions, where $p$ is an odd prime. In \cite{WH2023Se}, two classes of non-degenerate quadratic forms were used to construct $q$-ary self-orthogonal codes, where $q$…
▽ More
Self-orthogonal codes are a significant class of linear codes in coding theory and have attracted a lot of attention. In \cite{HLL2023Te,LH2023Se}, $p$-ary self-orthogonal codes were constructed by using $p$-ary weakly regular bent functions, where $p$ is an odd prime. In \cite{WH2023Se}, two classes of non-degenerate quadratic forms were used to construct $q$-ary self-orthogonal codes, where $q$ is a power of a prime. In this paper, we construct new families of $q$-ary self-orthogonal codes using vectorial dual-bent functions. Some classes of at least almost optimal linear codes are obtained from the dual codes of the constructed self-orthogonal codes. In some cases, we completely determine the weight distributions of the constructed self-orthogonal codes. From the view of vectorial dual-bent functions, we illustrate that the works on constructing self-orthogonal codes from $p$-ary weakly regular bent functions \cite{HLL2023Te,LH2023Se} and non-degenerate quadratic forms with $q$ being odd \cite{WH2023Se} can be obtained by our results. We partially answer an open problem on determining the weight distribution of a class of self-orthogonal codes given in \cite{LH2023Se}. As applications, we construct new infinite families of at least almost optimal $q$-ary linear complementary dual codes (for short, LCD codes) and quantum codes.
△ Less
Submitted 19 March, 2024;
originally announced March 2024.
-
Mathematics of multi-agent learning systems at the interface of game theory and artificial intelligence
Authors:
Long Wang,
Feng Fu,
Xingru Chen
Abstract:
Evolutionary Game Theory (EGT) and Artificial Intelligence (AI) are two fields that, at first glance, might seem distinct, but they have notable connections and intersections. The former focuses on the evolution of behaviors (or strategies) in a population, where individuals interact with others and update their strategies based on imitation (or social learning). The more successful a strategy is,…
▽ More
Evolutionary Game Theory (EGT) and Artificial Intelligence (AI) are two fields that, at first glance, might seem distinct, but they have notable connections and intersections. The former focuses on the evolution of behaviors (or strategies) in a population, where individuals interact with others and update their strategies based on imitation (or social learning). The more successful a strategy is, the more prevalent it becomes over time. The latter, meanwhile, is centered on machine learning algorithms and (deep) neural networks. It is often from a single-agent perspective but increasingly involves multi-agent environments, in which intelligent agents adjust their strategies based on feedback and experience, somewhat akin to the evolutionary process yet distinct in their self-learning capacities. In light of the key components necessary to address real-world problems, including (i) learning and adaptation, (ii) cooperation and competition, (iii) robustness and stability, and altogether (iv) population dynamics of individual agents whose strategies evolve, the cross-fertilization of ideas between both fields will contribute to the advancement of mathematics of multi-agent learning systems, in particular, to the nascent domain of ``collective cooperative intelligence'' bridging evolutionary dynamics and multi-agent reinforcement learning.
△ Less
Submitted 9 March, 2024;
originally announced March 2024.
-
SiLVR: Scalable Lidar-Visual Reconstruction with Neural Radiance Fields for Robotic Inspection
Authors:
Yifu Tao,
Yash Bhalgat,
Lanke Frank Tarimo Fu,
Matias Mattamala,
Nived Chebrolu,
Maurice Fallon
Abstract:
We present a neural-field-based large-scale reconstruction system that fuses lidar and vision data to generate high-quality reconstructions that are geometrically accurate and capture photo-realistic textures. This system adapts the state-of-the-art neural radiance field (NeRF) representation to also incorporate lidar data which adds strong geometric constraints on the depth and surface normals. W…
▽ More
We present a neural-field-based large-scale reconstruction system that fuses lidar and vision data to generate high-quality reconstructions that are geometrically accurate and capture photo-realistic textures. This system adapts the state-of-the-art neural radiance field (NeRF) representation to also incorporate lidar data which adds strong geometric constraints on the depth and surface normals. We exploit the trajectory from a real-time lidar SLAM system to bootstrap a Structure-from-Motion (SfM) procedure to both significantly reduce the computation time and to provide metric scale which is crucial for lidar depth loss. We use submapping to scale the system to large-scale environments captured over long trajectories. We demonstrate the reconstruction system with data from a multi-camera, lidar sensor suite onboard a legged robot, hand-held while scanning building scenes for 600 metres, and onboard an aerial robot surveying a multi-storey mock disaster site-building. Website: https://ori-drs.github.io/projects/silvr/
△ Less
Submitted 11 March, 2024;
originally announced March 2024.
-
Reinforcement Learning Jazz Improvisation: When Music Meets Game Theory
Authors:
Vedant Tapiavala,
Joshua Piesner,
Sourjyamoy Barman,
Feng Fu
Abstract:
Live performances of music are always charming, with the unpredictability of improvisation due to the dynamic between musicians and interactions with the audience. Jazz improvisation is a particularly noteworthy example for further investigation from a theoretical perspective. Here, we introduce a novel mathematical game theory model for jazz improvisation, providing a framework for studying music…
▽ More
Live performances of music are always charming, with the unpredictability of improvisation due to the dynamic between musicians and interactions with the audience. Jazz improvisation is a particularly noteworthy example for further investigation from a theoretical perspective. Here, we introduce a novel mathematical game theory model for jazz improvisation, providing a framework for studying music theory and improvisational methodologies. We use computational modeling, mainly reinforcement learning, to explore diverse stochastic improvisational strategies and their paired performance on improvisation. We find that the most effective strategy pair is a strategy that reacts to the most recent payoff (Stepwise Changes) with a reinforcement learning strategy limited to notes in the given chord (Chord-Following Reinforcement Learning). Conversely, a strategy that reacts to the partner's last note and attempts to harmonize with it (Harmony Prediction) strategy pair yields the lowest non-control payoff and highest standard deviation, indicating that picking notes based on immediate reactions to the partner player can yield inconsistent outcomes. On average, the Chord-Following Reinforcement Learning strategy demonstrates the highest mean payoff, while Harmony Prediction exhibits the lowest. Our work lays the foundation for promising applications beyond jazz: including the use of artificial intelligence (AI) models to extract data from audio clips to refine musical reward systems, and training machine learning (ML) models on existing jazz solos to further refine strategies within the game.
△ Less
Submitted 25 February, 2024;
originally announced March 2024.
-
SCott: Accelerating Diffusion Models with Stochastic Consistency Distillation
Authors:
Hongjian Liu,
Qingsong Xie,
Zhijie Deng,
Chen Chen,
Shixiang Tang,
Fueyang Fu,
Zheng-jun Zha,
Haonan Lu
Abstract:
The iterative sampling procedure employed by diffusion models (DMs) often leads to significant inference latency. To address this, we propose Stochastic Consistency Distillation (SCott) to enable accelerated text-to-image generation, where high-quality generations can be achieved with just 1-2 sampling steps, and further improvements can be obtained by adding additional steps. In contrast to vanil…
▽ More
The iterative sampling procedure employed by diffusion models (DMs) often leads to significant inference latency. To address this, we propose Stochastic Consistency Distillation (SCott) to enable accelerated text-to-image generation, where high-quality generations can be achieved with just 1-2 sampling steps, and further improvements can be obtained by adding additional steps. In contrast to vanilla consistency distillation (CD) which distills the ordinary differential equation solvers-based sampling process of a pretrained teacher model into a student, SCott explores the possibility and validates the efficacy of integrating stochastic differential equation (SDE) solvers into CD to fully unleash the potential of the teacher. SCott is augmented with elaborate strategies to control the noise strength and sampling process of the SDE solver. An adversarial loss is further incorporated to strengthen the sample quality with rare sampling steps. Empirically, on the MSCOCO-2017 5K dataset with a Stable Diffusion-V1.5 teacher, SCott achieves an FID (Frechet Inceptio Distance) of 22.1, surpassing that (23.4) of the 1-step InstaFlow (Liu et al., 2023) and matching that of 4-step UFOGen (Xue et al., 2023b). Moreover, SCott can yield more diverse samples than other consistency models for high-resolution image generation (Luo et al., 2023a), with up to 16% improvement in a qualified metric. The code and checkpoints are coming soon.
△ Less
Submitted 15 April, 2024; v1 submitted 3 March, 2024;
originally announced March 2024.
-
Retrieval-Augmented Generation for AI-Generated Content: A Survey
Authors:
Penghao Zhao,
Hailin Zhang,
Qinhan Yu,
Zhengren Wang,
Yunteng Geng,
Fangcheng Fu,
Ling Yang,
Wentao Zhang,
Jie Jiang,
Bin Cui
Abstract:
Advancements in model algorithms, the growth of foundational models, and access to high-quality datasets have propelled the evolution of Artificial Intelligence Generated Content (AIGC). Despite its notable successes, AIGC still faces hurdles such as updating knowledge, handling long-tail data, mitigating data leakage, and managing high training and inference costs. Retrieval-Augmented Generation…
▽ More
Advancements in model algorithms, the growth of foundational models, and access to high-quality datasets have propelled the evolution of Artificial Intelligence Generated Content (AIGC). Despite its notable successes, AIGC still faces hurdles such as updating knowledge, handling long-tail data, mitigating data leakage, and managing high training and inference costs. Retrieval-Augmented Generation (RAG) has recently emerged as a paradigm to address such challenges. In particular, RAG introduces the information retrieval process, which enhances the generation process by retrieving relevant objects from available data stores, leading to higher accuracy and better robustness. In this paper, we comprehensively review existing efforts that integrate RAG technique into AIGC scenarios. We first classify RAG foundations according to how the retriever augments the generator, distilling the fundamental abstractions of the augmentation methodologies for various retrievers and generators. This unified perspective encompasses all RAG scenarios, illuminating advancements and pivotal technologies that help with potential future progress. We also summarize additional enhancements methods for RAG, facilitating effective engineering and implementation of RAG systems. Then from another view, we survey on practical applications of RAG across different modalities and tasks, offering valuable references for researchers and practitioners. Furthermore, we introduce the benchmarks for RAG, discuss the limitations of current RAG systems, and suggest potential directions for future research. Github: https://github.com/PKU-DAIR/RAG-Survey.
△ Less
Submitted 21 June, 2024; v1 submitted 29 February, 2024;
originally announced February 2024.
-
Are Synthetic Time-series Data Really not as Good as Real Data?
Authors:
Fanzhe Fu,
Junru Chen,
Jing Zhang,
Carl Yang,
Lvbin Ma,
Yang Yang
Abstract:
Time-series data presents limitations stemming from data quality issues, bias and vulnerabilities, and generalization problem. Integrating universal data synthesis methods holds promise in improving generalization. However, current methods cannot guarantee that the generator's output covers all unseen real data. In this paper, we introduce InfoBoost -- a highly versatile cross-domain data synthesi…
▽ More
Time-series data presents limitations stemming from data quality issues, bias and vulnerabilities, and generalization problem. Integrating universal data synthesis methods holds promise in improving generalization. However, current methods cannot guarantee that the generator's output covers all unseen real data. In this paper, we introduce InfoBoost -- a highly versatile cross-domain data synthesizing framework with time series representation learning capability. We have developed a method based on synthetic data that enables model training without the need for real data, surpassing the performance of models trained with real data. Additionally, we have trained a universal feature extractor based on our synthetic data that is applicable to all time-series data. Our approach overcomes interference from multiple sources rhythmic signal, noise interference, and long-period features that exceed sampling window capabilities. Through experiments, our non-deep-learning synthetic data enables models to achieve superior reconstruction performance and universal explicit representation extraction without the need for real data.
△ Less
Submitted 1 February, 2024;
originally announced February 2024.
-
Towards Context-Stable and Visual-Consistent Image Inpainting
Authors:
Yikai Wang,
Chenjie Cao,
Ke Fan Xiangyang Xue Yanwei Fu
Abstract:
Recent progress in inpainting increasingly relies on generative models, leveraging their strong generation capabilities for addressing large irregular masks. However, this enhanced generation often introduces context-instability, leading to arbitrary object generation within masked regions. This paper proposes a balanced solution, emphasizing the importance of unmasked regions in guiding inpaintin…
▽ More
Recent progress in inpainting increasingly relies on generative models, leveraging their strong generation capabilities for addressing large irregular masks. However, this enhanced generation often introduces context-instability, leading to arbitrary object generation within masked regions. This paper proposes a balanced solution, emphasizing the importance of unmasked regions in guiding inpainting while preserving generation capacity. Our approach, Aligned Stable Inpainting with UnKnown Areas Prior (ASUKA), employs a Masked Auto-Encoder (MAE) to produce reconstruction-based prior. Aligned with the powerful Stable Diffusion inpainting model (SD), ASUKA significantly improves context stability. ASUKA further adopts an inpainting-specialized decoder, highly reducing the color inconsistency issue of SD and thus ensuring more visual-consistent inpainting. We validate effectiveness of inpainting algorithms on benchmark dataset Places 2 and a collection of several existing datasets, dubbed MISATO, across diverse domains and masking scenarios. Results on these benchmark datasets confirm ASUKA's efficacy in both context-stability and visual-consistency compared to SD and other inpainting algorithms.
△ Less
Submitted 17 March, 2024; v1 submitted 8 December, 2023;
originally announced December 2023.
-
Learning with Errors over Group Rings Constructed by Semi-direct Product
Authors:
Jiaqi Liu,
Fang-Wei Fu
Abstract:
The Learning with Errors (LWE) problem has been widely utilized as a foundation for numerous cryptographic tools over the years. In this study, we focus on an algebraic variant of the LWE problem called Group ring LWE (GR-LWE). We select group rings (or their direct summands) that underlie specific families of finite groups constructed by taking the semi-direct product of two cyclic groups. Unlike…
▽ More
The Learning with Errors (LWE) problem has been widely utilized as a foundation for numerous cryptographic tools over the years. In this study, we focus on an algebraic variant of the LWE problem called Group ring LWE (GR-LWE). We select group rings (or their direct summands) that underlie specific families of finite groups constructed by taking the semi-direct product of two cyclic groups. Unlike the Ring-LWE problem described in \cite{lyubashevsky2010ideal}, the multiplication operation in the group rings considered here is non-commutative. As an extension of Ring-LWE, it maintains computational hardness and can be potentially applied in many cryptographic scenarios. In this paper, we present two polynomial-time quantum reductions. Firstly, we provide a quantum reduction from the worst-case shortest independent vectors problem (SIVP) in ideal lattices with polynomial approximate factor to the search version of GR-LWE. This reduction requires that the underlying group ring possesses certain mild properties; Secondly, we present another quantum reduction for two types of group rings, where the worst-case SIVP problem is directly reduced to the (average-case) decision GR-LWE problem. The pseudorandomness of GR-LWE samples guaranteed by this reduction can be consequently leveraged to construct semantically secure public-key cryptosystems.
△ Less
Submitted 1 December, 2023; v1 submitted 27 November, 2023;
originally announced November 2023.
-
E-CORE: Emotion Correlation Enhanced Empathetic Dialogue Generation
Authors:
Fengyi Fu,
Lei Zhang,
Quan Wang,
Zhendong Mao
Abstract:
Achieving empathy is a crucial step toward humanized dialogue systems. Current approaches for empathetic dialogue generation mainly perceive an emotional label to generate an empathetic response conditioned on it, which simply treat emotions independently, but ignore the intrinsic emotion correlation in dialogues, resulting in inaccurate emotion perception and unsuitable response generation. In th…
▽ More
Achieving empathy is a crucial step toward humanized dialogue systems. Current approaches for empathetic dialogue generation mainly perceive an emotional label to generate an empathetic response conditioned on it, which simply treat emotions independently, but ignore the intrinsic emotion correlation in dialogues, resulting in inaccurate emotion perception and unsuitable response generation. In this paper, we propose a novel emotion correlation enhanced empathetic dialogue generation framework, which comprehensively realizes emotion correlation learning, utilization, and supervising. Specifically, a multi-resolution emotion graph is devised to capture context-based emotion interactions from different resolutions, further modeling emotion correlation. Then we propose an emotion correlation enhanced decoder, with a novel correlation-aware aggregation and soft/hard strategy, respectively improving the emotion perception and response generation. Experimental results on the benchmark dataset demonstrate the superiority of our model in both empathetic perception and expression.
△ Less
Submitted 25 November, 2023;
originally announced November 2023.
-
Linear codes with few weights from non-weakly regular plateaued functions
Authors:
Yadi Wei,
Jiaxin Wang,
Fang-Wei Fu
Abstract:
Linear codes with few weights have significant applications in secret sharing schemes, authentication codes, association schemes, and strongly regular graphs. There are a number of methods to construct linear codes, one of which is based on functions. Furthermore, two generic constructions of linear codes from functions called the first and the second generic constructions, have aroused the resear…
▽ More
Linear codes with few weights have significant applications in secret sharing schemes, authentication codes, association schemes, and strongly regular graphs. There are a number of methods to construct linear codes, one of which is based on functions. Furthermore, two generic constructions of linear codes from functions called the first and the second generic constructions, have aroused the research interest of scholars. Recently, in \cite{Nian}, Li and Mesnager proposed two open problems: Based on the first and the second generic constructions, respectively, construct linear codes from non-weakly regular plateaued functions and determine their weight distributions.
Motivated by these two open problems, in this paper, firstly, based on the first generic construction, we construct some three-weight and at most five-weight linear codes from non-weakly regular plateaued functions and determine the weight distributions of the constructed codes. Next, based on the second generic construction, we construct some three-weight and at most five-weight linear codes from non-weakly regular plateaued functions belonging to $\mathcal{NWRF}$ (defined in this paper) and determine the weight distributions of the constructed codes. We also give the punctured codes of these codes obtained based on the second generic construction and determine their weight distributions. Meanwhile, we obtain some optimal and almost optimal linear codes. Besides, by the Ashikhmin-Barg condition, we have that the constructed codes are minimal for almost all cases and obtain some secret sharing schemes with nice access structures based on their dual codes.
△ Less
Submitted 30 October, 2023;
originally announced October 2023.
-
Generative and Contrastive Paradigms Are Complementary for Graph Self-Supervised Learning
Authors:
Yuxiang Wang,
Xiao Yan,
Chuang Hu,
Fangcheng Fu,
Wentao Zhang,
Hao Wang,
Shuo Shang,
Jiawei Jiang
Abstract:
For graph self-supervised learning (GSSL), masked autoencoder (MAE) follows the generative paradigm and learns to reconstruct masked graph edges or node features. Contrastive Learning (CL) maximizes the similarity between augmented views of the same graph and is widely used for GSSL. However, MAE and CL are considered separately in existing works for GSSL. We observe that the MAE and CL paradigms…
▽ More
For graph self-supervised learning (GSSL), masked autoencoder (MAE) follows the generative paradigm and learns to reconstruct masked graph edges or node features. Contrastive Learning (CL) maximizes the similarity between augmented views of the same graph and is widely used for GSSL. However, MAE and CL are considered separately in existing works for GSSL. We observe that the MAE and CL paradigms are complementary and propose the graph contrastive masked autoencoder (GCMAE) framework to unify them. Specifically, by focusing on local edges or node features, MAE cannot capture global information of the graph and is sensitive to particular edges and features. On the contrary, CL excels in extracting global information because it considers the relation between graphs. As such, we equip GCMAE with an MAE branch and a CL branch, and the two branches share a common encoder, which allows the MAE branch to exploit the global information extracted by the CL branch. To force GCMAE to capture global graph structures, we train it to reconstruct the entire adjacency matrix instead of only the masked edges as in existing works. Moreover, a discrimination loss is proposed for feature reconstruction, which improves the disparity between node embeddings rather than reducing the reconstruction error to tackle the feature smoothing problem of MAE. We evaluate GCMAE on four popular graph tasks (i.e., node classification, node clustering, link prediction, and graph classification) and compare with 14 state-of-the-art baselines. The results show that GCMAE consistently provides good accuracy across these tasks, and the maximum accuracy improvement is up to 3.2% compared with the best-performing baseline.
△ Less
Submitted 24 October, 2023;
originally announced October 2023.
-
New Lower Bounds for the Minimum Distance of Cyclic Codes and Applications to Locally Repairable Codes
Authors:
Jing Qiu,
Weijun Fang,
Fang-Wei Fu
Abstract:
Cyclic codes are an important class of linear codes. Bounding the minimum distance of cyclic codes is a long-standing research topic in coding theory, and several well-known and basic results have been developed on this topic. Recently, locally repairable codes (LRCs) have attracted much attention due to their repair efficiency in large-scale distributed storage systems. In this paper, by employin…
▽ More
Cyclic codes are an important class of linear codes. Bounding the minimum distance of cyclic codes is a long-standing research topic in coding theory, and several well-known and basic results have been developed on this topic. Recently, locally repairable codes (LRCs) have attracted much attention due to their repair efficiency in large-scale distributed storage systems. In this paper, by employing the singleton procedure technique, we first provide a sufficient condition for bounding the minimum distance of cyclic codes with typical defining sets. Secondly, by considering a specific case, we establish a connection between bounds for the minimum distance of cyclic codes and solutions to a system of inequalities. This connection leads to the derivation of new bounds, including some with general patterns. In particular, we provide three new bounds with general patterns, one of which serves as a generalization of the Betti-Sala bound. Finally, we present a generalized lower bound for a special case and construct several families of $(2, δ)$-LRCs with unbounded length and minimum distance $2δ$. It turns out that these LRCs are distance-optimal, and their parameters are new. To the best of our knowledge, this work represents the first construction of distance-optimal $(r, δ)$-LRCs with unbounded length and minimum distance exceeding $r+δ-1$.
△ Less
Submitted 11 October, 2023;
originally announced October 2023.
-
A Further Study of Vectorial Dual-Bent Functions
Authors:
Jiaxin Wang,
Fang-Wei Fu,
Yadi Wei,
Jing Yang
Abstract:
Vectorial dual-bent functions have recently attracted some researchers' interest as they play a significant role in constructing partial difference sets, association schemes, bent partitions and linear codes. In this paper, we further study vectorial dual-bent functions $F: V_{n}^{(p)}\rightarrow V_{m}^{(p)}$, where $2\leq m \leq \frac{n}{2}$, $V_{n}^{(p)}$ denotes an $n$-dimensional vector space…
▽ More
Vectorial dual-bent functions have recently attracted some researchers' interest as they play a significant role in constructing partial difference sets, association schemes, bent partitions and linear codes. In this paper, we further study vectorial dual-bent functions $F: V_{n}^{(p)}\rightarrow V_{m}^{(p)}$, where $2\leq m \leq \frac{n}{2}$, $V_{n}^{(p)}$ denotes an $n$-dimensional vector space over the prime field $\mathbb{F}_{p}$. We give new characterizations of certain vectorial dual-bent functions (called vectorial dual-bent functions with Condition A) in terms of amorphic association schemes, linear codes and generalized Hadamard matrices, respectively. When $p=2$, we characterize vectorial dual-bent functions with Condition A in terms of bent partitions. Furthermore, we characterize certain bent partitions in terms of amorphic association schemes, linear codes and generalized Hadamard matrices, respectively. For general vectorial dual-bent functions $F: V_{n}^{(p)}\rightarrow V_{m}^{(p)}$ with $F(0)=0, F(x)=F(-x)$ and $2\leq m \leq \frac{n}{2}$, we give a necessary and sufficient condition on constructing association schemes. Based on such a result, more association schemes are constructed from vectorial dual-bent functions.
△ Less
Submitted 23 September, 2023;
originally announced September 2023.
-
Some new constructions of optimal linear codes and alphabet-optimal $(r,δ)$-locally repairable codes
Authors:
Jing Qiu,
Fang-Wei Fu
Abstract:
In distributed storage systems, locally repairable codes (LRCs) are designed to reduce disk I/O and repair costs by enabling recovery of each code symbol from a small number of other symbols. To handle multiple node failures, $(r,δ)$-LRCs are introduced to enable local recovery in the event of up to $δ-1$ failed nodes. Constructing optimal $(r,δ)$-LRCs has been a significant research topic over th…
▽ More
In distributed storage systems, locally repairable codes (LRCs) are designed to reduce disk I/O and repair costs by enabling recovery of each code symbol from a small number of other symbols. To handle multiple node failures, $(r,δ)$-LRCs are introduced to enable local recovery in the event of up to $δ-1$ failed nodes. Constructing optimal $(r,δ)$-LRCs has been a significant research topic over the past decade. In \cite{Luo2022}, Luo \emph{et al.} proposed a construction of linear codes by using unions of some projective subspaces within a projective space. Several new classes of Griesmer codes and distance-optimal codes were constructed, and some of them were proved to be alphabet-optimal $2$-LRCs.
In this paper, we first modify the method of constructing linear codes in \cite{Luo2022} by considering a more general situation of intersecting projective subspaces. This modification enables us to construct good codes with more flexible parameters. Additionally, we present the conditions for the constructed linear codes to qualify as Griesmer codes or achieve distance optimality. Next, we explore the locality of linear codes constructed by eliminating elements from a complete projective space. The novelty of our work lies in establishing the locality as $(2,p-2)$, $(2,p-1)$, or $(2,p)$-locality, in contrast to the previous literature that only considered $2$-locality. Moreover, by combining analysis of code parameters and the C-M like bound for $(r,δ)$-LRCs, we construct some alphabet-optimal $(2,δ)$-LRCs which may be either Griesmer codes or not Griesmer codes. Finally, we investigate the availability and alphabet-optimality of $(r,δ)$-LRCs constructed from our modified framework.
△ Less
Submitted 9 July, 2023;
originally announced July 2023.
-
Improving Automatic Parallel Training via Balanced Memory Workload Optimization
Authors:
Yujie Wang,
Youhe Jiang,
Xupeng Miao,
Fangcheng Fu,
Shenhan Zhu,
Xiaonan Nie,
Yaofeng Tu,
Bin Cui
Abstract:
Transformer models have emerged as the leading approach for achieving state-of-the-art performance across various application domains, serving as the foundation for advanced large-scale deep learning (DL) models. However, efficiently training these models across multiple GPUs remains a complex challenge due to the abundance of parallelism options. Existing DL systems either require manual efforts…
▽ More
Transformer models have emerged as the leading approach for achieving state-of-the-art performance across various application domains, serving as the foundation for advanced large-scale deep learning (DL) models. However, efficiently training these models across multiple GPUs remains a complex challenge due to the abundance of parallelism options. Existing DL systems either require manual efforts to design distributed training plans or limit parallelism combinations to a constrained search space. In this paper, we present Galvatron-BMW, a novel system framework that integrates multiple prevalent parallelism dimensions and automatically identifies the most efficient hybrid parallelism strategy. To effectively navigate this vast search space, we employ a decision tree approach for decomposition and pruning based on intuitive insights. We further utilize a dynamic programming search algorithm to derive the optimal plan. Moreover, to improve resource utilization and enhance system efficiency, we propose a bi-objective optimization workflow that focuses on workload balance. Our evaluations on different Transformer models demonstrate the capabilities of Galvatron-BMW in automating distributed training under varying GPU memory constraints. Across all tested scenarios, Galvatron-BMW consistently achieves superior system throughput, surpassing previous approaches that rely on limited parallelism strategies.
△ Less
Submitted 24 February, 2024; v1 submitted 5 July, 2023;
originally announced July 2023.
-
OVLA: Neural Network Ownership Verification using Latent Watermarks
Authors:
Feisi Fu,
Wenchao Li
Abstract:
Ownership verification for neural networks is important for protecting these models from illegal copying, free-riding, re-distribution and other intellectual property misuse. We present a novel methodology for neural network ownership verification based on the notion of latent watermarks. Existing ownership verification methods either modify or introduce constraints to the neural network parameter…
▽ More
Ownership verification for neural networks is important for protecting these models from illegal copying, free-riding, re-distribution and other intellectual property misuse. We present a novel methodology for neural network ownership verification based on the notion of latent watermarks. Existing ownership verification methods either modify or introduce constraints to the neural network parameters, which are accessible to an attacker in a white-box attack and can be harmful to the network's normal operation, or train the network to respond to specific watermarks in the inputs similar to data poisoning-based backdoor attacks, which are susceptible to backdoor removal techniques. In this paper, we address these problems by decoupling a network's normal operation from its responses to watermarked inputs during ownership verification. The key idea is to train the network such that the watermarks remain dormant unless the owner's secret key is applied to activate it. The secret key is realized as a specific perturbation only known to the owner to the network's parameters. We show that our approach offers strong defense against backdoor detection, backdoor removal and surrogate model attacks.In addition, our method provides protection against ambiguity attacks where the attacker either tries to guess the secret weight key or uses fine-tuning to embed their own watermarks with a different key into a pre-trained neural network. Experimental results demonstrate the advantages and effectiveness of our proposed approach.
△ Less
Submitted 25 June, 2023; v1 submitted 15 June, 2023;
originally announced June 2023.
-
Steering control of payoff-maximizing players in adaptive learning dynamics
Authors:
Xingru Chen,
Feng Fu
Abstract:
Evolutionary game theory provides a mathematical foundation for cross-disciplinary fertilization, especially for integrating ideas from artificial intelligence and game theory. Such integration offers a transparent and rigorous approach to complex decision-making problems in a variety of important contexts, ranging from evolutionary computation to machine behavior. Despite the astronomically huge…
▽ More
Evolutionary game theory provides a mathematical foundation for cross-disciplinary fertilization, especially for integrating ideas from artificial intelligence and game theory. Such integration offers a transparent and rigorous approach to complex decision-making problems in a variety of important contexts, ranging from evolutionary computation to machine behavior. Despite the astronomically huge individual behavioral strategy space for interactions in the iterated Prisoner's Dilemma (IPD) games, the so-called Zero-Determinant (ZD) strategies is a set of rather simple memory-one strategies yet can unilaterally set a linear payoff relationship between themselves and their opponent. Although the witting of ZD strategies gives players an upper hand in the IPD games, we find and characterize unbending strategies that can force ZD players to be fair in their own interest. Moreover, our analysis reveals the ubiquity of unbending properties in common IPD strategies which are previously overlooked. In this work, we demonstrate the important steering role of unbending strategies in fostering fairness and cooperation in pairwise interactions. Our results will help bring a new perspective by means of combining game theory and multi-agent learning systems for optimizing winning strategies that are robust to noises, errors, and deceptions in non-zero-sum games.
△ Less
Submitted 29 May, 2023;
originally announced May 2023.
-
Accelerating Text-to-Image Editing via Cache-Enabled Sparse Diffusion Inference
Authors:
Zihao Yu,
Haoyang Li,
Fangcheng Fu,
Xupeng Miao,
Bin Cui
Abstract:
Due to the recent success of diffusion models, text-to-image generation is becoming increasingly popular and achieves a wide range of applications. Among them, text-to-image editing, or continuous text-to-image generation, attracts lots of attention and can potentially improve the quality of generated images. It's common to see that users may want to slightly edit the generated image by making min…
▽ More
Due to the recent success of diffusion models, text-to-image generation is becoming increasingly popular and achieves a wide range of applications. Among them, text-to-image editing, or continuous text-to-image generation, attracts lots of attention and can potentially improve the quality of generated images. It's common to see that users may want to slightly edit the generated image by making minor modifications to their input textual descriptions for several rounds of diffusion inference. However, such an image editing process suffers from the low inference efficiency of many existing diffusion models even using GPU accelerators. To solve this problem, we introduce Fast Image Semantically Edit (FISEdit), a cached-enabled sparse diffusion model inference engine for efficient text-to-image editing. The key intuition behind our approach is to utilize the semantic mapping between the minor modifications on the input text and the affected regions on the output image. For each text editing step, FISEdit can automatically identify the affected image regions and utilize the cached unchanged regions' feature map to accelerate the inference process. Extensive empirical results show that FISEdit can be $3.4\times$ and $4.4\times$ faster than existing methods on NVIDIA TITAN RTX and A100 GPUs respectively, and even generates more satisfactory images.
△ Less
Submitted 4 January, 2024; v1 submitted 27 May, 2023;
originally announced May 2023.
-
OSDP: Optimal Sharded Data Parallel for Distributed Deep Learning
Authors:
Youhe Jiang,
Fangcheng Fu,
Xupeng Miao,
Xiaonan Nie,
Bin Cui
Abstract:
Large-scale deep learning models contribute to significant performance improvements on varieties of downstream tasks. Current data and model parallelism approaches utilize model replication and partition techniques to support the distributed training of ultra-large models. However, directly deploying these systems often leads to sub-optimal training efficiency due to the complex model architecture…
▽ More
Large-scale deep learning models contribute to significant performance improvements on varieties of downstream tasks. Current data and model parallelism approaches utilize model replication and partition techniques to support the distributed training of ultra-large models. However, directly deploying these systems often leads to sub-optimal training efficiency due to the complex model architectures and the strict device memory constraints. In this paper, we propose Optimal Sharded Data Parallel (OSDP), an automated parallel training system that combines the advantages from both data and model parallelism. Given the model description and the device information, OSDP makes trade-offs between the memory consumption and the hardware utilization, thus automatically generates the distributed computation graph and maximizes the overall system throughput. In addition, OSDP introduces operator splitting to further alleviate peak memory footprints during training with negligible overheads, which enables the trainability of larger models as well as the higher throughput. Extensive experimental results of OSDP on multiple different kinds of large-scale models demonstrate that the proposed strategy outperforms the state-of-the-art in multiple regards.
△ Less
Submitted 17 May, 2023; v1 submitted 17 May, 2023;
originally announced May 2023.
-
Singleton-Optimal LRCs and Perfect LRCs via Cyclic and Constacyclic Codes
Authors:
Weijun Fang,
Fang-Wei Fu,
Bin Chen,
Shu-Tao Xia
Abstract:
Locally repairable codes (LRCs) have emerged as an important coding scheme in distributed storage systems (DSSs) with relatively low repair cost by accessing fewer non-failure nodes. Theoretical bounds and optimal constructions of LRCs have been widely investigated. Optimal LRCs via cyclic and constacyclic codes provide significant benefit of elegant algebraic structure and efficient encoding proc…
▽ More
Locally repairable codes (LRCs) have emerged as an important coding scheme in distributed storage systems (DSSs) with relatively low repair cost by accessing fewer non-failure nodes. Theoretical bounds and optimal constructions of LRCs have been widely investigated. Optimal LRCs via cyclic and constacyclic codes provide significant benefit of elegant algebraic structure and efficient encoding procedure. In this paper, we continue to consider the constructions of optimal LRCs via cyclic and constacyclic codes with long code length. Specifically, we first obtain two classes of $q$-ary cyclic Singleton-optimal $(n, k, d=6;r=2)$-LRCs with length $n=3(q+1)$ when $3 \mid (q-1)$ and $q$ is even, and length $n=\frac{3}{2}(q+1)$ when $3 \mid (q-1)$ and $q \equiv 1(\bmod~4)$, respectively. To the best of our knowledge, this is the first construction of $q$-ary cyclic Singleton-optimal LRCs with length $n>q+1$ and minimum distance $d \geq 5$. On the other hand, an LRC acheiving the Hamming-type bound is called a perfect LRC. By using cyclic and constacyclic codes, we construct two new families of $q$-ary perfect LRCs with length $n=\frac{q^m-1}{q-1}$, minimum distance $d=5$ and locality $r=2$.
△ Less
Submitted 10 March, 2023;
originally announced March 2023.
-
Complex Systems of Secrecy: The Offshore Networks of Oligarchs
Authors:
Ho-Chun Herbert Chang,
Brooke Harrington,
Feng Fu,
Daniel Rockmore
Abstract:
Following the invasion of Ukraine, the US, UK, and EU governments--among others--sanctioned oligarchs close to Putin. This approach has come under scrutiny, as evidence has emerged of the oligarchs' successful evasion of these punishments. To address this problem, we analyze the role of an overlooked but highly influential group: the secretive professional intermediaries who create and administer…
▽ More
Following the invasion of Ukraine, the US, UK, and EU governments--among others--sanctioned oligarchs close to Putin. This approach has come under scrutiny, as evidence has emerged of the oligarchs' successful evasion of these punishments. To address this problem, we analyze the role of an overlooked but highly influential group: the secretive professional intermediaries who create and administer the oligarchs' offshore financial empires. Drawing on the Offshore Leaks Database provided by the International Consortium of Investigative Journalists (ICIJ), we examine the ties linking offshore expert advisors (lawyers, accountants, and other wealth management professionals) to ultra-high-net-worth individuals from four countries: Russia, China, the United States, and Hong Kong. We find that resulting nation-level "oligarch networks" share a scale-free structure characterized by heterogeneity of heavy-tailed degree distributions of wealth managers; however, network topologies diverge across clients from democratic versus autocratic regimes. While generally robust, scale-free networks are fragile when targeted by attacks on highly-connected nodes. Our "knock-out" experiments pinpoint this vulnerability to the small group of wealth managers themselves, suggesting that sanctioning these professional intermediaries may be more effective and efficient in disrupting dark finance flows than sanctions on their wealthy clients. This vulnerability is especially pronounced amongst Russian oligarchs, who concentrate their offshore business in a handful of boutique wealth management firms. The distinctive patterns we identify suggest a new approach to sanctions, focused on expert intermediaries to disrupt the finances and alliances of their wealthy clients. More generally, our research contributes to the larger body of work on complexity science and the structures of secrecy.
△ Less
Submitted 6 March, 2023;
originally announced March 2023.
-
Angel-PTM: A Scalable and Economical Large-scale Pre-training System in Tencent
Authors:
Xiaonan Nie,
Yi Liu,
Fangcheng Fu,
Jinbao Xue,
Dian Jiao,
Xupeng Miao,
Yangyu Tao,
Bin Cui
Abstract:
Recent years have witnessed the unprecedented achievements of large-scale pre-trained models, especially the Transformer models. Many products and services in Tencent Inc., such as WeChat, QQ, and Tencent Advertisement, have been opted in to gain the power of pre-trained models. In this work, we present Angel-PTM, a productive deep learning system designed for pre-training and fine-tuning Transfor…
▽ More
Recent years have witnessed the unprecedented achievements of large-scale pre-trained models, especially the Transformer models. Many products and services in Tencent Inc., such as WeChat, QQ, and Tencent Advertisement, have been opted in to gain the power of pre-trained models. In this work, we present Angel-PTM, a productive deep learning system designed for pre-training and fine-tuning Transformer models. Angel-PTM can train extremely large-scale models with hierarchical memory efficiently. The key designs of Angel-PTM are the fine-grained memory management via the Page abstraction and a unified scheduling method that coordinate the computations, data movements, and communications. Furthermore, Angel-PTM supports extreme model scaling with SSD storage and implements the lock-free updating mechanism to address the SSD I/O bandwidth bottlenecks. Experimental results demonstrate that Angel-PTM outperforms existing systems by up to 114.8% in terms of maximum model scale as well as up to 88.9% in terms of training throughput. Additionally, experiments on GPT3-175B and T5-MoE-1.2T models utilizing hundreds of GPUs verify the strong scalability of Angel-PTM.
△ Less
Submitted 5 March, 2023;
originally announced March 2023.
-
Bent Partitions, Vectorial Dual-Bent Functions and Partial Difference Sets
Authors:
Jiaxin Wang,
Fang-Wei Fu,
Yadi Wei
Abstract:
It is known that partial spreads is a class of bent partitions. In \cite{AM2022Be,MP2021Be}, two classes of bent partitions whose forms are similar to partial spreads were presented. In \cite{AKM2022Ge}, more bent partitions $Γ_{1}, Γ_{2}, Γ_{1}^{\bullet}, Γ_{2}^{\bullet}, Θ_{1}, Θ_{2}$ were presented from (pre)semifields, including the bent partitions given in \cite{AM2022Be,MP2021Be}. In this pa…
▽ More
It is known that partial spreads is a class of bent partitions. In \cite{AM2022Be,MP2021Be}, two classes of bent partitions whose forms are similar to partial spreads were presented. In \cite{AKM2022Ge}, more bent partitions $Γ_{1}, Γ_{2}, Γ_{1}^{\bullet}, Γ_{2}^{\bullet}, Θ_{1}, Θ_{2}$ were presented from (pre)semifields, including the bent partitions given in \cite{AM2022Be,MP2021Be}. In this paper, we investigate the relations between bent partitions and vectorial dual-bent functions. For any prime $p$, we show that one can generate certain bent partitions (called bent partitions satisfying Condition $\mathcal{C}$) from certain vectorial dual-bent functions (called vectorial dual-bent functions satisfying Condition A). In particular, when $p$ is an odd prime, we show that bent partitions satisfying Condition $\mathcal{C}$ one-to-one correspond to vectorial dual-bent functions satisfying Condition A. We give an alternative proof that $Γ_{1}, Γ_{2}, Γ_{1}^{\bullet}, Γ_{2}^{\bullet}, Θ_{1}, Θ_{2}$ are bent partitions. We present a secondary construction of vectorial dual-bent functions, which can be used to generate more bent partitions. We show that any ternary weakly regular bent function $f: V_{n}^{(3)}\rightarrow \mathbb{F}_{3}$ ($n$ even) of $2$-form can generate a bent partition. When such $f$ is weakly regular but not regular, the generated bent partition by $f$ is not coming from a normal bent partition, which answers an open problem proposed in \cite{AM2022Be}. We give a sufficient condition on constructing partial difference sets from bent partitions, and when $p$ is an odd prime, we provide a characterization of bent partitions satisfying Condition $\mathcal{C}$ in terms of partial difference sets.
△ Less
Submitted 2 January, 2023;
originally announced January 2023.
-
Dormant Neural Trojans
Authors:
Feisi Fu,
Panagiota Kiourti,
Wenchao Li
Abstract:
We present a novel methodology for neural network backdoor attacks. Unlike existing training-time attacks where the Trojaned network would respond to the Trojan trigger after training, our approach inserts a Trojan that will remain dormant until it is activated. The activation is realized through a specific perturbation to the network's weight parameters only known to the attacker. Our analysis an…
▽ More
We present a novel methodology for neural network backdoor attacks. Unlike existing training-time attacks where the Trojaned network would respond to the Trojan trigger after training, our approach inserts a Trojan that will remain dormant until it is activated. The activation is realized through a specific perturbation to the network's weight parameters only known to the attacker. Our analysis and the experimental results demonstrate that dormant Trojaned networks can effectively evade detection by state-of-the-art backdoor detection methods.
△ Less
Submitted 2 November, 2022;
originally announced November 2022.
-
Convergence analysis of a quasi-Monte Carlo-based deep learning algorithm for solving partial differential equations
Authors:
Fengjiang Fu,
Xiaoqun Wang
Abstract:
Deep learning methods have achieved great success in solving partial differential equations (PDEs), where the loss is often defined as an integral. The accuracy and efficiency of these algorithms depend greatly on the quadrature method. We propose to apply quasi-Monte Carlo (QMC) methods to the Deep Ritz Method (DRM) for solving the Neumann problems for the Poisson equation and the static Schrödin…
▽ More
Deep learning methods have achieved great success in solving partial differential equations (PDEs), where the loss is often defined as an integral. The accuracy and efficiency of these algorithms depend greatly on the quadrature method. We propose to apply quasi-Monte Carlo (QMC) methods to the Deep Ritz Method (DRM) for solving the Neumann problems for the Poisson equation and the static Schrödinger equation. For error estimation, we decompose the error of using the deep learning algorithm to solve PDEs into the generalization error, the approximation error and the training error. We establish the upper bounds and prove that QMC-based DRM achieves an asymptotically smaller error bound than DRM. Numerical experiments show that the proposed method converges faster in all cases and the variances of the gradient estimators of randomized QMC-based DRM are much smaller than those of DRM, which illustrates the superiority of QMC in deep learning over MC.
△ Less
Submitted 28 October, 2022;
originally announced October 2022.
-
OSDP: Optimal Sharded Data Parallel for Distributed Deep Learning
Authors:
Youhe Jiang,
Fangcheng Fu,
Xupeng Miao,
Xiaonan Nie,
Bin Cui
Abstract:
Large-scale deep learning models contribute to significant performance improvements on varieties of downstream tasks. Current data and model parallelism approaches utilize model replication and partition techniques to support the distributed training of ultra-large models. However, directly deploying these systems often leads to sub-optimal training efficiency due to the complex model architecture…
▽ More
Large-scale deep learning models contribute to significant performance improvements on varieties of downstream tasks. Current data and model parallelism approaches utilize model replication and partition techniques to support the distributed training of ultra-large models. However, directly deploying these systems often leads to sub-optimal training efficiency due to the complex model architectures and the strict device memory constraints. In this paper, we propose Optimal Sharded Data Parallel (OSDP), an automated parallel training system that combines the advantages from both data and model parallelism. Given the model description and the device information, OSDP makes trade-offs between the memory consumption and the hardware utilization, thus automatically generates the distributed computation graph and maximizes the overall system throughput. In addition, OSDP introduces operator splitting to further alleviate peak memory footprints during training with negligible overheads, which enables the trainability of larger models as well as the higher throughput. Extensive experimental results of OSDP on multiple different kinds of large-scale models demonstrate that the proposed strategy outperforms the state-of-the-art in multiple regards. Our code is available at https://github.com/Youhe-Jiang/OptimalShardedDataParallel.
△ Less
Submitted 18 May, 2023; v1 submitted 27 September, 2022;
originally announced September 2022.
-
A multi-chain synchronization protocol that leverage zero knowledge proof to minimize communication trust base
Authors:
Sinka Gao,
Guo Qiang Li,
Hong Fei Fu,
Heng Zhang
Abstract:
Delphinus cross-chain aggregator is a universal firmware which synchronise states between different smart contracts on different block-chains. In the world of block-chains, synchronization challenges are two-folded. Firstly, contracts from different main block-chain can not communicate with each other which makes it hard to establish a trustworthy communication channel for them to share and mainta…
▽ More
Delphinus cross-chain aggregator is a universal firmware which synchronise states between different smart contracts on different block-chains. In the world of block-chains, synchronization challenges are two-folded. Firstly, contracts from different main block-chain can not communicate with each other which makes it hard to establish a trustworthy communication channel for them to share and maintain a universal state between each other. Secondly, transactions on different block-chains can hardly be ordered thus conflicts are common and we need a novel way to avoid and handle these conflicts. Delphinus cross-chain aggregator is a ZKSNARK based multi-block-chain layer on top of which rich cross chain applications can run safely and efficiently.
△ Less
Submitted 7 September, 2022;
originally announced September 2022.
-
Hilti-Oxford Dataset: A Millimetre-Accurate Benchmark for Simultaneous Localization and Mapping
Authors:
Lintong Zhang,
Michael Helmberger,
Lanke Frank Tarimo Fu,
David Wisth,
Marco Camurri,
Davide Scaramuzza,
Maurice Fallon
Abstract:
Simultaneous Localization and Mapping (SLAM) is being deployed in real-world applications, however many state-of-the-art solutions still struggle in many common scenarios. A key necessity in progressing SLAM research is the availability of high-quality datasets and fair and transparent benchmarking. To this end, we have created the Hilti-Oxford Dataset, to push state-of-the-art SLAM systems to the…
▽ More
Simultaneous Localization and Mapping (SLAM) is being deployed in real-world applications, however many state-of-the-art solutions still struggle in many common scenarios. A key necessity in progressing SLAM research is the availability of high-quality datasets and fair and transparent benchmarking. To this end, we have created the Hilti-Oxford Dataset, to push state-of-the-art SLAM systems to their limits. The dataset has a variety of challenges ranging from sparse and regular construction sites to a 17th century neoclassical building with fine details and curved surfaces. To encourage multi-modal SLAM approaches, we designed a data collection platform featuring a lidar, five cameras, and an IMU (Inertial Measurement Unit). With the goal of benchmarking SLAM algorithms for tasks where accuracy and robustness are paramount, we implemented a novel ground truth collection method that enables our dataset to accurately measure SLAM pose errors with millimeter accuracy. To further ensure accuracy, the extrinsics of our platform were verified with a micrometer-accurate scanner, and temporal calibration was managed online using hardware time synchronization. The multi-modality and diversity of our dataset attracted a large field of academic and industrial researchers to enter the second edition of the Hilti SLAM challenge, which concluded in June 2022. The results of the challenge show that while the top three teams could achieve an accuracy of 2cm or better for some sequences, the performance dropped off in more difficult sequences.
△ Less
Submitted 15 May, 2023; v1 submitted 21 August, 2022;
originally announced August 2022.
-
A Tool for Neural Network Global Robustness Certification and Training
Authors:
Zhilu Wang,
Yixuan Wang,
Feisi Fu,
Ruochen Jiao,
Chao Huang,
Wenchao Li,
Qi Zhu
Abstract:
With the increment of interest in leveraging machine learning technology in safety-critical systems, the robustness of neural networks under external disturbance receives more and more concerns. Global robustness is a robustness property defined on the entire input domain. And a certified globally robust network can ensure its robustness on any possible network input. However, the state-of-the-art…
▽ More
With the increment of interest in leveraging machine learning technology in safety-critical systems, the robustness of neural networks under external disturbance receives more and more concerns. Global robustness is a robustness property defined on the entire input domain. And a certified globally robust network can ensure its robustness on any possible network input. However, the state-of-the-art global robustness certification algorithm can only certify networks with at most several thousand neurons. In this paper, we propose the GPU-supported global robustness certification framework GROCET, which is more efficient than the previous optimization-based certification approach. Moreover, GROCET provides differentiable global robustness, which is leveraged in the training of globally robust neural networks.
△ Less
Submitted 15 August, 2022;
originally announced August 2022.
-
Towards Communication-efficient Vertical Federated Learning Training via Cache-enabled Local Updates
Authors:
Fangcheng Fu,
Xupeng Miao,
Jiawei Jiang,
Huanran Xue,
Bin Cui
Abstract:
Vertical federated learning (VFL) is an emerging paradigm that allows different parties (e.g., organizations or enterprises) to collaboratively build machine learning models with privacy protection. In the training phase, VFL only exchanges the intermediate statistics, i.e., forward activations and backward derivatives, across parties to compute model gradients. Nevertheless, due to its geo-distri…
▽ More
Vertical federated learning (VFL) is an emerging paradigm that allows different parties (e.g., organizations or enterprises) to collaboratively build machine learning models with privacy protection. In the training phase, VFL only exchanges the intermediate statistics, i.e., forward activations and backward derivatives, across parties to compute model gradients. Nevertheless, due to its geo-distributed nature, VFL training usually suffers from the low WAN bandwidth.
In this paper, we introduce CELU-VFL, a novel and efficient VFL training framework that exploits the local update technique to reduce the cross-party communication rounds. CELU-VFL caches the stale statistics and reuses them to estimate model gradients without exchanging the ad hoc statistics. Significant techniques are proposed to improve the convergence performance. First, to handle the stochastic variance problem, we propose a uniform sampling strategy to fairly choose the stale statistics for local updates. Second, to harness the errors brought by the staleness, we devise an instance weighting mechanism that measures the reliability of the estimated gradients. Theoretical analysis proves that CELU-VFL achieves a similar sub-linear convergence rate as vanilla VFL training but requires much fewer communication rounds. Empirical results on both public and real-world workloads validate that CELU-VFL can be up to six times faster than the existing works.
△ Less
Submitted 29 July, 2022;
originally announced July 2022.
-
Constructions of linear codes with two or three weights from vectorial dual-bent functions
Authors:
Jiaxin Wang,
Zexia Shi,
Yadi Wei,
Fang-Wei Fu
Abstract:
Linear codes with a few weights are an important class of codes in coding theory and have attracted a lot of attention. In this paper, we present several constructions of $q$-ary linear codes with two or three weights from vectorial dual-bent functions, where $q$ is a power of an odd prime $p$. The weight distributions of the constructed $q$-ary linear codes are completely determined. We illustrat…
▽ More
Linear codes with a few weights are an important class of codes in coding theory and have attracted a lot of attention. In this paper, we present several constructions of $q$-ary linear codes with two or three weights from vectorial dual-bent functions, where $q$ is a power of an odd prime $p$. The weight distributions of the constructed $q$-ary linear codes are completely determined. We illustrate that some known constructions in the literature can be obtained by our constructions. In some special cases, our constructed linear codes can meet the Griesmer bound. Furthermore, based on the constructed $q$-ary linear codes, we obtain secret sharing schemes with interesting access structures.
△ Less
Submitted 24 July, 2022;
originally announced July 2022.
-
Bounds and Constructions of Singleton-Optimal Locally Repairable Codes with Small Localities
Authors:
Weijun Fang,
Bin Chen,
Shu-Tao Xia,
Fang-Wei Fu
Abstract:
Constructions of optimal locally repairable codes (LRCs) achieving Singleton-type bound have been exhaustively investigated in recent years. In this paper, we consider new bounds and constructions of Singleton-optimal LRCs with minmum distance $d=6$, locality $r=3$ and minimum distance $d=7$ and locality $r=2$, respectively. Firstly, we establish equivalent connections between the existence of the…
▽ More
Constructions of optimal locally repairable codes (LRCs) achieving Singleton-type bound have been exhaustively investigated in recent years. In this paper, we consider new bounds and constructions of Singleton-optimal LRCs with minmum distance $d=6$, locality $r=3$ and minimum distance $d=7$ and locality $r=2$, respectively. Firstly, we establish equivalent connections between the existence of these two families of LRCs and the existence of some subsets of lines in the projective space with certain properties. Then, we employ the line-point incidence matrix and Johnson bounds for constant weight codes to derive new improved bounds on the code length, which are tighter than known results. Finally, by using some techniques of finite field and finite geometry, we give some new constructions of Singleton-optimal LRCs, which have larger length than previous ones.
△ Less
Submitted 12 July, 2022;
originally announced July 2022.
-
BlindFL: Vertical Federated Machine Learning without Peeking into Your Data
Authors:
Fangcheng Fu,
Huanran Xue,
Yong Cheng,
Yangyu Tao,
Bin Cui
Abstract:
Due to the rising concerns on privacy protection, how to build machine learning (ML) models over different data sources with security guarantees is gaining more popularity. Vertical federated learning (VFL) describes such a case where ML models are built upon the private data of different participated parties that own disjoint features for the same set of instances, which fits many real-world coll…
▽ More
Due to the rising concerns on privacy protection, how to build machine learning (ML) models over different data sources with security guarantees is gaining more popularity. Vertical federated learning (VFL) describes such a case where ML models are built upon the private data of different participated parties that own disjoint features for the same set of instances, which fits many real-world collaborative tasks. Nevertheless, we find that existing solutions for VFL either support limited kinds of input features or suffer from potential data leakage during the federated execution. To this end, this paper aims to investigate both the functionality and security of ML modes in the VFL scenario.
To be specific, we introduce BlindFL, a novel framework for VFL training and inference. First, to address the functionality of VFL models, we propose the federated source layers to unite the data from different parties. Various kinds of features can be supported efficiently by the federated source layers, including dense, sparse, numerical, and categorical features. Second, we carefully analyze the security during the federated execution and formalize the privacy requirements. Based on the analysis, we devise secure and accurate algorithm protocols, and further prove the security guarantees under the ideal-real simulation paradigm. Extensive experiments show that BlindFL supports diverse datasets and models efficiently whilst achieves robust privacy guarantees.
△ Less
Submitted 16 June, 2022;
originally announced June 2022.
-
Spatial Games of Fake News
Authors:
Matthew I Jones,
Scott D. Pauls,
Feng Fu
Abstract:
To curb the spread of fake news on social media platforms, recent studies have considered an online crowdsourcing fact-checking approach as one possible intervention method to reduce misinformation. However, it remains unclear under what conditions crowdsourcing fact-checking efforts deter the spread of misinformation. To address this issue, we model such distributed fact-checking as `peer policin…
▽ More
To curb the spread of fake news on social media platforms, recent studies have considered an online crowdsourcing fact-checking approach as one possible intervention method to reduce misinformation. However, it remains unclear under what conditions crowdsourcing fact-checking efforts deter the spread of misinformation. To address this issue, we model such distributed fact-checking as `peer policing' that will reduce the perceived payoff to share or disseminate false information (fake news) and also reward the spread of trustworthy information (real news). By simulating our model on synthetic square lattices and small-world networks, we show that the presence of social network structure enables fake news spreaders to be self-organized into echo chambers, thereby providing a boost to the efficacy of fake news and thus its resistance to fact-checking efforts. Additionally, to study our model in a more realistic setting, we utilize a Twitter network dataset and study the effectiveness of deliberately choosing specific individuals to be fact-checkers. We find that targeted fact-checking efforts can be highly effective, seeing the same level of success with as little as a fifth of the number of fact-checkers, but it depends on the structure of the network in question. In the limit of weak selection, we obtain closed-form analytical conditions for critical threshold of crowdsourced fact-checking in terms of the payoff values in our fact-checker/fake news game. Our work has practical implications for developing model-based mitigation strategies for controlling the spread of misinformation that interferes with the political discourse.
△ Less
Submitted 8 June, 2022;
originally announced June 2022.
-
On generalized quasi-cyclic codes over $\mathbb{Z}_4$
Authors:
Jian Gao,
Xiangrui Meng,
Fang-Wei Fu
Abstract:
Based on good algebraic structures and practicabilities, generalized quasi-cyclic (GQC) codes play important role in coding theory. In this paper, we study some results on GQC codes over $\mathbb{Z}_4$ including the normalized generating set, the minimum generating set and the normalized generating set of their dual codes. As an application, new $\mathbb{Z}_4$-linear codes and good nonlinear binar…
▽ More
Based on good algebraic structures and practicabilities, generalized quasi-cyclic (GQC) codes play important role in coding theory. In this paper, we study some results on GQC codes over $\mathbb{Z}_4$ including the normalized generating set, the minimum generating set and the normalized generating set of their dual codes. As an application, new $\mathbb{Z}_4$-linear codes and good nonlinear binary codes are constructed from GQC codes over $\mathbb{Z}_4$.
△ Less
Submitted 22 March, 2022;
originally announced March 2022.
-
Post-quantum Multi-stage Secret Sharing Schemes using Inhomogeneous Linear Recursion and Ajtai's Function
Authors:
Jing Yang,
Fang-Wei Fu
Abstract:
Secret sharing was firstly proposed in 1979 by Shamir and Blakley respectively. To avoid deficiencies of original schemes, researchers presented improvement schemes, among which the multi-secret sharing scheme (MSS) is significant. There are three categories of MSSs, however, we focus on multi-stage secret sharing scheme (MSSS) recovering secrets with any order in this work. By observing inhomogen…
▽ More
Secret sharing was firstly proposed in 1979 by Shamir and Blakley respectively. To avoid deficiencies of original schemes, researchers presented improvement schemes, among which the multi-secret sharing scheme (MSS) is significant. There are three categories of MSSs, however, we focus on multi-stage secret sharing scheme (MSSS) recovering secrets with any order in this work. By observing inhomogeneous linear recursions (ILRs) in the literature, we conclude a general formula and divide ILRs into two types according to different variables in them. Utilizing these two kinds of ILRs, we propose four verifiable MSSSs with Ajtai's function, which is a lattice-based function. Our schemes have the following advantages. Firstly, our schemes can detect cheat of the dealer and participants, and are multi-use. Secondly, we have several ways to restore secrets. Thirdly, we can turn our schemes into other types of MSSs due to the universality of our method. Fourthly, since we utilize a lattice-based function to mask shares, our schemes can resist the attack from the quantum computer with computational security. Finally, although our schemes need more memory consumption than some known schemes, we need much less time consumption, which makes our schemes more suitable facing limited computing power.
△ Less
Submitted 18 February, 2022;
originally announced February 2022.
-
New results on vectorial dual-bent functions and partial difference sets
Authors:
Jiaxin Wang,
Fang-Wei Fu
Abstract:
Bent functions $f: V_{n}\rightarrow \mathbb{F}_{p}$ with certain additional properties play an important role in constructing partial difference sets, where $V_{n}$ denotes an $n$-dimensional vector space over $\mathbb{F}_{p}$, $p$ is an odd prime. In \cite{Cesmelioglu1,Cesmelioglu2}, the so-called vectorial dual-bent functions are considered to construct partial difference sets. In \cite{Cesmelio…
▽ More
Bent functions $f: V_{n}\rightarrow \mathbb{F}_{p}$ with certain additional properties play an important role in constructing partial difference sets, where $V_{n}$ denotes an $n$-dimensional vector space over $\mathbb{F}_{p}$, $p$ is an odd prime. In \cite{Cesmelioglu1,Cesmelioglu2}, the so-called vectorial dual-bent functions are considered to construct partial difference sets. In \cite{Cesmelioglu1}, Çeşmelioǧlu \emph{et al.} showed that for vectorial dual-bent functions $F: V_{n}\rightarrow V_{s}$ with certain additional properties, the preimage set of $0$ for $F$ forms a partial difference set. In \cite{Cesmelioglu2}, Çeşmelioǧlu \emph{et al.} showed that for a class of Maiorana-McFarland vectorial dual-bent functions $F: V_{n}\rightarrow \mathbb{F}_{p^s}$, the preimage set of the squares (non-squares) in $\mathbb{F}_{p^s}^{*}$ for $F$ forms a partial difference set. In this paper, we further study vectorial dual-bent functions and partial difference sets. We prove that for vectorial dual-bent functions $F: V_{n}\rightarrow \mathbb{F}_{p^s}$ with certain additional properties, the preimage set of the squares (non-squares) in $\mathbb{F}_{p^s}^{*}$ for $F$ and the preimage set of any coset of some subgroup of $\mathbb{F}_{p^s}^{*}$ for $F$ form partial difference sets. Furthermore, explicit constructions of partial difference sets are yielded from some (non)-quadratic vectorial dual-bent functions. In this paper, we illustrate that almost all the results of using weakly regular $p$-ary bent functions to construct partial difference sets are special cases of our results.
△ Less
Submitted 28 April, 2022; v1 submitted 8 February, 2022;
originally announced February 2022.
-
Some Results on the Improved Bound and Construction of Optimal $(r,δ)$ LRCs
Authors:
Bin Chen,
Weijun Fang,
Yueqi Chen,
Shu-Tao Xia,
Fang-Wei Fu,
Xiangyu Chen
Abstract:
Locally repairable codes (LRCs) with $(r,δ)$ locality were introduced by Prakash \emph{et al.} into distributed storage systems (DSSs) due to their benefit of locally repairing at least $δ-1$ erasures via other $r$ survival nodes among the same local group. An LRC achieving the $(r,δ)$ Singleton-type bound is called an optimal $(r,δ)$ LRC. Constructions of optimal $(r,δ)$ LRCs with longer code len…
▽ More
Locally repairable codes (LRCs) with $(r,δ)$ locality were introduced by Prakash \emph{et al.} into distributed storage systems (DSSs) due to their benefit of locally repairing at least $δ-1$ erasures via other $r$ survival nodes among the same local group. An LRC achieving the $(r,δ)$ Singleton-type bound is called an optimal $(r,δ)$ LRC. Constructions of optimal $(r,δ)$ LRCs with longer code length and determining the maximal code length have been an important research direction in coding theory in recent years. In this paper, we conduct further research on the improvement of maximum code length of optimal $(r,δ)$ LRCs. For $2δ+1\leq d\leq 2δ+2$, our upper bounds largely improve the ones by Cai \emph{et al.}, which are tight in some special cases. Moreover, we generalize the results of Chen \emph{et al.} and obtain a complete characterization of optimal $(r=2, δ)$-LRCs in the sense of geometrical existence in the finite projective plane $PG(2,q)$. Within this geometrical characterization, we construct a class of optimal $(r,δ)$ LRCs based on the sunflower structure. Both the construction and upper bounds are better than previous ones.
△ Less
Submitted 7 February, 2022;
originally announced February 2022.