-
Integrating Explanations in Learning LTL Specifications from Demonstrations
Authors:
Ashutosh Gupta,
John Komp,
Abhay Singh Rajput,
Krishna Shankaranarayanan,
Ashutosh Trivedi,
Namrita Varshney
Abstract:
This paper investigates whether recent advances in Large Language Models (LLMs) can assist in translating human explanations into a format that can robustly support learning Linear Temporal Logic (LTL) from demonstrations. Both LLMs and optimization-based methods can extract LTL specifications from demonstrations; however, they have distinct limitations. LLMs can quickly generate solutions and inc…
▽ More
This paper investigates whether recent advances in Large Language Models (LLMs) can assist in translating human explanations into a format that can robustly support learning Linear Temporal Logic (LTL) from demonstrations. Both LLMs and optimization-based methods can extract LTL specifications from demonstrations; however, they have distinct limitations. LLMs can quickly generate solutions and incorporate human explanations, but their lack of consistency and reliability hampers their applicability in safety-critical domains. On the other hand, optimization-based methods do provide formal guarantees but cannot process natural language explanations and face scalability challenges. We present a principled approach to combining LLMs and optimization-based methods to faithfully translate human explanations and demonstrations into LTL specifications. We have implemented a tool called Janaka based on our approach. Our experiments demonstrate the effectiveness of combining explanations with demonstrations in learning LTL specifications through several case studies.
△ Less
Submitted 3 April, 2024;
originally announced April 2024.
-
COMET: Generating Commit Messages using Delta Graph Context Representation
Authors:
Abhinav Reddy Mandli,
Saurabhsingh Rajput,
Tushar Sharma
Abstract:
Commit messages explain code changes in a commit and facilitate collaboration among developers. Several commit message generation approaches have been proposed; however, they exhibit limited success in capturing the context of code changes. We propose Comet (Context-Aware Commit Message Generation), a novel approach that captures context of code changes using a graph-based representation and lever…
▽ More
Commit messages explain code changes in a commit and facilitate collaboration among developers. Several commit message generation approaches have been proposed; however, they exhibit limited success in capturing the context of code changes. We propose Comet (Context-Aware Commit Message Generation), a novel approach that captures context of code changes using a graph-based representation and leverages a transformer-based model to generate high-quality commit messages. Our proposed method utilizes delta graph that we developed to effectively represent code differences. We also introduce a customizable quality assurance module to identify optimal messages, mitigating subjectivity in commit messages. Experiments show that Comet outperforms state-of-the-art techniques in terms of bleu-norm and meteor metrics while being comparable in terms of rogue-l. Additionally, we compare the proposed approach with the popular gpt-3.5-turbo model, along with gpt-4-turbo; the most capable GPT model, over zero-shot, one-shot, and multi-shot settings. We found Comet outperforming the GPT models, on five and four metrics respectively and provide competitive results with the two other metrics. The study has implications for researchers, tool developers, and software developers. Software developers may utilize Comet to generate context-aware commit messages. Researchers and tool developers can apply the proposed delta graph technique in similar contexts, like code review summarization.
△ Less
Submitted 2 February, 2024;
originally announced February 2024.
-
SegmentAnyBone: A Universal Model that Segments Any Bone at Any Location on MRI
Authors:
Hanxue Gu,
Roy Colglazier,
Haoyu Dong,
Jikai Zhang,
Yaqian Chen,
Zafer Yildiz,
Yuwen Chen,
Lin Li,
Jichen Yang,
Jay Willhite,
Alex M. Meyer,
Brian Guo,
Yashvi Atul Shah,
Emily Luo,
Shipra Rajput,
Sally Kuehn,
Clark Bulleit,
Kevin A. Wu,
Jisoo Lee,
Brandon Ramirez,
Darui Lu,
Jay M. Levin,
Maciej A. Mazurowski
Abstract:
Magnetic Resonance Imaging (MRI) is pivotal in radiology, offering non-invasive and high-quality insights into the human body. Precise segmentation of MRIs into different organs and tissues would be highly beneficial since it would allow for a higher level of understanding of the image content and enable important measurements, which are essential for accurate diagnosis and effective treatment pla…
▽ More
Magnetic Resonance Imaging (MRI) is pivotal in radiology, offering non-invasive and high-quality insights into the human body. Precise segmentation of MRIs into different organs and tissues would be highly beneficial since it would allow for a higher level of understanding of the image content and enable important measurements, which are essential for accurate diagnosis and effective treatment planning. Specifically, segmenting bones in MRI would allow for more quantitative assessments of musculoskeletal conditions, while such assessments are largely absent in current radiological practice. The difficulty of bone MRI segmentation is illustrated by the fact that limited algorithms are publicly available for use, and those contained in the literature typically address a specific anatomic area. In our study, we propose a versatile, publicly available deep-learning model for bone segmentation in MRI across multiple standard MRI locations. The proposed model can operate in two modes: fully automated segmentation and prompt-based segmentation. Our contributions include (1) collecting and annotating a new MRI dataset across various MRI protocols, encompassing over 300 annotated volumes and 8485 annotated slices across diverse anatomic regions; (2) investigating several standard network architectures and strategies for automated segmentation; (3) introducing SegmentAnyBone, an innovative foundational model-based approach that extends Segment Anything Model (SAM); (4) comparative analysis of our algorithm and previous approaches; and (5) generalization analysis of our algorithm across different anatomical locations and MRI sequences, as well as an external dataset. We publicly release our model at https://github.com/mazurowski-lab/SegmentAnyBone.
△ Less
Submitted 23 January, 2024;
originally announced January 2024.
-
Maestro: Uncovering Low-Rank Structures via Trainable Decomposition
Authors:
Samuel Horvath,
Stefanos Laskaridis,
Shashank Rajput,
Hongyi Wang
Abstract:
Deep Neural Networks (DNNs) have been a large driver for AI breakthroughs in recent years. However, these models have been getting increasingly large as they become more accurate and safe. This means that their training becomes increasingly costly and time-consuming and typically yields a single model to fit all targets. Various techniques have been proposed in the literature to mitigate this, inc…
▽ More
Deep Neural Networks (DNNs) have been a large driver for AI breakthroughs in recent years. However, these models have been getting increasingly large as they become more accurate and safe. This means that their training becomes increasingly costly and time-consuming and typically yields a single model to fit all targets. Various techniques have been proposed in the literature to mitigate this, including pruning, sparsification, or quantization of model weights and updates. While achieving high compression rates, they often incur significant computational overheads at training or lead to non-negligible accuracy penalty. Alternatively, factorization methods have been leveraged for low-rank compression of DNNs. Similarly, such techniques (e.g., SVD) frequently rely on heavy iterative decompositions of layers and are potentially sub-optimal for non-linear models, such as DNNs. We take a further step in designing efficient low-rank models and propose Maestro, a framework for trainable low-rank layers. Instead of iteratively applying a priori decompositions, the low-rank structure is baked into the training process through LoD, a low-rank ordered decomposition. Not only is this the first time importance ordering via sampling is applied on the decomposed DNN structure, but it also allows selecting ranks at a layer granularity. Our theoretical analysis demonstrates that in special cases LoD recovers the SVD decomposition and PCA. Applied to DNNs, Maestro enables the extraction of lower footprint models that preserve performance. Simultaneously, it enables the graceful trade-off between accuracy-latency for deployment to even more constrained devices without retraining.
△ Less
Submitted 14 June, 2024; v1 submitted 28 August, 2023;
originally announced August 2023.
-
Enhancing Energy-Awareness in Deep Learning through Fine-Grained Energy Measurement
Authors:
Saurabhsingh Rajput,
Tim Widmayer,
Ziyuan Shang,
Maria Kechagia,
Federica Sarro,
Tushar Sharma
Abstract:
With the increasing usage, scale, and complexity of Deep Learning (DL) models, their rapidly growing energy consumption has become a critical concern. Promoting green development and energy awareness at different granularities is the need of the hour to limit carbon emissions of DL systems. However, the lack of standard and repeatable tools to accurately measure and optimize energy consumption at…
▽ More
With the increasing usage, scale, and complexity of Deep Learning (DL) models, their rapidly growing energy consumption has become a critical concern. Promoting green development and energy awareness at different granularities is the need of the hour to limit carbon emissions of DL systems. However, the lack of standard and repeatable tools to accurately measure and optimize energy consumption at a fine granularity (e.g., at method level) hinders progress in this area. This paper introduces FECoM (Fine-grained Energy Consumption Meter), a framework for fine-grained DL energy consumption measurement. FECoM enables researchers and developers to profile DL APIs from energy perspective. FECoM addresses the challenges of measuring energy consumption at fine-grained level by using static instrumentation and considering various factors, including computational load and temperature stability. We assess FECoM's capability to measure fine-grained energy consumption for one of the most popular open-source DL frameworks, namely TensorFlow. Using FECoM, we also investigate the impact of parameter size and execution time on energy consumption, enriching our understanding of TensorFlow APIs' energy profiles. Furthermore, we elaborate on the considerations, issues, and challenges that one needs to consider while designing and implementing a fine-grained energy consumption measurement tool. This work will facilitate further advances in DL energy measurement and the development of energy-aware practices for DL systems.
△ Less
Submitted 1 February, 2024; v1 submitted 23 August, 2023;
originally announced August 2023.
-
Hyper-automation-The next peripheral for automation in IT industries
Authors:
Ayush Singh Rajput,
Richa Gupta
Abstract:
The extension of legacy business process automation beyond the bounds of specific processes is known as hyperautomation. Hyperautomation provides automation for nearly any repetitive action performed by business users by combining AI tools with RPA. It automates complex IT business processes that a company's top brains might not be able to complete. This is an end-to-end automation of a standard b…
▽ More
The extension of legacy business process automation beyond the bounds of specific processes is known as hyperautomation. Hyperautomation provides automation for nearly any repetitive action performed by business users by combining AI tools with RPA. It automates complex IT business processes that a company's top brains might not be able to complete. This is an end-to-end automation of a standard business process deployment. It enables automation to perform task digitalization by combining a brain computer interface (BCI) with AI and RPA automation tools. BCI, in conjunction with automation tools, will advance the detection and generation of automation processes to the next level. It allows enterprises to combine business intelligence systems, address complex requirements, and enhance human expertise and automation experience. Hyperautomation and its importance in today's environment are briefly discussed in this paper. The article then goes on to discuss how BCI and sensors might aid Hyperautomation. The specific sectors of solicitations were examined using a variety of flexible technologies associated to this concept, as well as dedicated workflow techniques, which are also diagrammatically illustrated. Hyperautomation is being utilized to improve the efficiency, accuracy, and human enhancement of automated tasks dramatically. It incorporates a number of automated tools in its discovery, implementation, and automation phases. As a result, it's well-suited to integrating cutting-edge technologies and experimenting with new methods of working. Keywords- Hyperautomation, Brain computer Interface (BCI), Technology, Used case, Sensors, Industries.
△ Less
Submitted 14 May, 2023;
originally announced May 2023.
-
From Assistive Technologies to Metaverse: Technologies in Inclusive Higher Education for Students with Specific Learning Difficulties
Authors:
Gokul Yenduri,
Rajesh Kaluri,
Dharmendra Singh Rajput,
Kuruva Lakshmanna,
Thippa Reddy Gadekallu,
Mufti Mahmud,
David J. Brown
Abstract:
The development of new technologies and their expanding use in a wide range of educational environments are driving the transformation of higher education. Assistive technologies are a subset of cutting-edge technology that can help students learn more effectively and make education accessible to everyone. Assistive technology can enhance, maintain, or improve the capacities of students with learn…
▽ More
The development of new technologies and their expanding use in a wide range of educational environments are driving the transformation of higher education. Assistive technologies are a subset of cutting-edge technology that can help students learn more effectively and make education accessible to everyone. Assistive technology can enhance, maintain, or improve the capacities of students with learning difficulties. Students with learning difficulties will be greatly benefited from the use of assistive technologies. If these technologies are used effectively, students with learning difficulties can compete with their peers and complete their academic tasks. We aim to conduct this review to better understand the role of assistive technologies in providing inclusive higher education for students with learning difficulties. The review begins with the introduction of learning difficulties and their causes; inclusive education and the need for assistive technologies; the reasoning for conducting this review; and a summary of related reviews on assistive technologies for students with learning difficulties in inclusive higher education. Then, we discuss the preliminaries for the learning difficulties type and assistive technology. Later, we discuss the effects of assistive technology on inclusive higher education for students with learning difficulties. Additionally, we discuss related projects and support tools available in inclusive higher education for students with learning difficulties. We also explore the challenges and possible solutions related to using assistive technology in higher education to provide inclusive education for students with learning difficulties. We conclude the review with a discussion of potential promising future directions.
△ Less
Submitted 5 May, 2023;
originally announced May 2023.
-
Recommender Systems with Generative Retrieval
Authors:
Shashank Rajput,
Nikhil Mehta,
Anima Singh,
Raghunandan H. Keshavan,
Trung Vu,
Lukasz Heldt,
Lichan Hong,
Yi Tay,
Vinh Q. Tran,
Jonah Samost,
Maciej Kula,
Ed H. Chi,
Maheswaran Sathiamoorthy
Abstract:
Modern recommender systems perform large-scale retrieval by first embedding queries and item candidates in the same unified space, followed by approximate nearest neighbor search to select top candidates given a query embedding. In this paper, we propose a novel generative retrieval approach, where the retrieval model autoregressively decodes the identifiers of the target candidates. To that end,…
▽ More
Modern recommender systems perform large-scale retrieval by first embedding queries and item candidates in the same unified space, followed by approximate nearest neighbor search to select top candidates given a query embedding. In this paper, we propose a novel generative retrieval approach, where the retrieval model autoregressively decodes the identifiers of the target candidates. To that end, we create semantically meaningful tuple of codewords to serve as a Semantic ID for each item. Given Semantic IDs for items in a user session, a Transformer-based sequence-to-sequence model is trained to predict the Semantic ID of the next item that the user will interact with. To the best of our knowledge, this is the first Semantic ID-based generative model for recommendation tasks. We show that recommender systems trained with the proposed paradigm significantly outperform the current SOTA models on various datasets. In addition, we show that incorporating Semantic IDs into the sequence-to-sequence model enhances its ability to generalize, as evidenced by the improved retrieval performance observed for items with no prior interaction history.
△ Less
Submitted 3 November, 2023; v1 submitted 8 May, 2023;
originally announced May 2023.
-
The Expressive Power of Tuning Only the Normalization Layers
Authors:
Angeliki Giannou,
Shashank Rajput,
Dimitris Papailiopoulos
Abstract:
Feature normalization transforms such as Batch and Layer-Normalization have become indispensable ingredients of state-of-the-art deep neural networks. Recent studies on fine-tuning large pretrained models indicate that just tuning the parameters of these affine transforms can achieve high accuracy for downstream tasks. These findings open the questions about the expressive power of tuning the norm…
▽ More
Feature normalization transforms such as Batch and Layer-Normalization have become indispensable ingredients of state-of-the-art deep neural networks. Recent studies on fine-tuning large pretrained models indicate that just tuning the parameters of these affine transforms can achieve high accuracy for downstream tasks. These findings open the questions about the expressive power of tuning the normalization layers of frozen networks. In this work, we take the first step towards this question and show that for random ReLU networks, fine-tuning only its normalization layers can reconstruct any target network that is $O(\sqrt{\text{width}})$ times smaller. We show that this holds even for randomly sparsified networks, under sufficient overparameterization, in agreement with prior empirical work.
△ Less
Submitted 4 July, 2023; v1 submitted 15 February, 2023;
originally announced February 2023.
-
Looped Transformers as Programmable Computers
Authors:
Angeliki Giannou,
Shashank Rajput,
Jy-yong Sohn,
Kangwook Lee,
Jason D. Lee,
Dimitris Papailiopoulos
Abstract:
We present a framework for using transformer networks as universal computers by programming them with specific weights and placing them in a loop. Our input sequence acts as a punchcard, consisting of instructions and memory for data read/writes. We demonstrate that a constant number of encoder layers can emulate basic computing blocks, including embedding edit operations, non-linear functions, fu…
▽ More
We present a framework for using transformer networks as universal computers by programming them with specific weights and placing them in a loop. Our input sequence acts as a punchcard, consisting of instructions and memory for data read/writes. We demonstrate that a constant number of encoder layers can emulate basic computing blocks, including embedding edit operations, non-linear functions, function calls, program counters, and conditional branches. Using these building blocks, we emulate a small instruction-set computer. This allows us to map iterative algorithms to programs that can be executed by a looped, 13-layer transformer. We show how this transformer, instructed by its input, can emulate a basic calculator, a basic linear algebra library, and in-context learning algorithms that employ backpropagation. Our work highlights the versatility of the attention mechanism, and demonstrates that even shallow transformers can execute full-fledged, general-purpose programs.
△ Less
Submitted 30 January, 2023;
originally announced January 2023.
-
Gui at MixMT 2022 : English-Hinglish: An MT approach for translation of code mixed data
Authors:
Akshat Gahoi,
Jayant Duneja,
Anshul Padhi,
Shivam Mangale,
Saransh Rajput,
Tanvi Kamble,
Dipti Misra Sharma,
Vasudeva Varma
Abstract:
Code-mixed machine translation has become an important task in multilingual communities and extending the task of machine translation to code mixed data has become a common task for these languages. In the shared tasks of WMT 2022, we try to tackle the same for both English + Hindi to Hinglish and Hinglish to English. The first task dealt with both Roman and Devanagari script as we had monolingual…
▽ More
Code-mixed machine translation has become an important task in multilingual communities and extending the task of machine translation to code mixed data has become a common task for these languages. In the shared tasks of WMT 2022, we try to tackle the same for both English + Hindi to Hinglish and Hinglish to English. The first task dealt with both Roman and Devanagari script as we had monolingual data in both English and Hindi whereas the second task only had data in Roman script. To our knowledge, we achieved one of the top ROUGE-L and WER scores for the first task of Monolingual to Code-Mixed machine translation. In this paper, we discuss the use of mBART with some special pre-processing and post-processing (transliteration from Devanagari to Roman) for the first task in detail and the experiments that we performed for the second task of translating code-mixed Hinglish to monolingual English.
△ Less
Submitted 21 October, 2022;
originally announced October 2022.
-
LIFT: Language-Interfaced Fine-Tuning for Non-Language Machine Learning Tasks
Authors:
Tuan Dinh,
Yuchen Zeng,
Ruisu Zhang,
Ziqian Lin,
Michael Gira,
Shashank Rajput,
Jy-yong Sohn,
Dimitris Papailiopoulos,
Kangwook Lee
Abstract:
Fine-tuning pretrained language models (LMs) without making any architectural changes has become a norm for learning various language downstream tasks. However, for non-language downstream tasks, a common practice is to employ task-specific designs for input, output layers, and loss functions. For instance, it is possible to fine-tune an LM into an MNIST classifier by replacing the word embedding…
▽ More
Fine-tuning pretrained language models (LMs) without making any architectural changes has become a norm for learning various language downstream tasks. However, for non-language downstream tasks, a common practice is to employ task-specific designs for input, output layers, and loss functions. For instance, it is possible to fine-tune an LM into an MNIST classifier by replacing the word embedding layer with an image patch embedding layer, the word token output layer with a 10-way output layer, and the word prediction loss with a 10-way classification loss, respectively. A natural question arises: Can LM fine-tuning solve non-language downstream tasks without changing the model architecture or loss function? To answer this, we propose Language-Interfaced Fine-Tuning (LIFT) and study its efficacy and limitations by conducting an extensive empirical study on a suite of non-language classification and regression tasks. LIFT does not make any changes to the model architecture or loss function, and it solely relies on the natural language interface, enabling "no-code machine learning with LMs." We find that LIFT performs comparably well across a wide range of low-dimensional classification and regression tasks, matching the performances of the best baselines in many cases, especially for the classification tasks. We also report experimental results on the fundamental properties of LIFT, including inductive bias, robustness, and sample complexity. We also analyze the effect of pretraining on LIFT and a few properties/techniques specific to LIFT, e.g., context-aware learning via appropriate prompting, calibrated predictions, data generation, and two-stage fine-tuning. Our code is available at https://github.com/UW-Madison-Lee-Lab/LanguageInterfacedFineTuning.
△ Less
Submitted 30 October, 2022; v1 submitted 13 June, 2022;
originally announced June 2022.
-
Utilizing Language-Image Pretraining for Efficient and Robust Bilingual Word Alignment
Authors:
Tuan Dinh,
Jy-yong Sohn,
Shashank Rajput,
Timothy Ossowski,
Yifei Ming,
Junjie Hu,
Dimitris Papailiopoulos,
Kangwook Lee
Abstract:
Word translation without parallel corpora has become feasible, rivaling the performance of supervised methods. Recent findings have shown that the accuracy and robustness of unsupervised word translation (UWT) can be improved by making use of visual observations, which are universal representations across languages. In this work, we investigate the potential of using not only visual observations b…
▽ More
Word translation without parallel corpora has become feasible, rivaling the performance of supervised methods. Recent findings have shown that the accuracy and robustness of unsupervised word translation (UWT) can be improved by making use of visual observations, which are universal representations across languages. In this work, we investigate the potential of using not only visual observations but also pretrained language-image models for enabling a more efficient and robust UWT. Specifically, we develop a novel UWT method dubbed Word Alignment using Language-Image Pretraining (WALIP), which leverages visual observations via the shared embedding space of images and texts provided by CLIP models (Radford et al., 2021). WALIP has a two-step procedure. First, we retrieve word pairs with high confidences of similarity, computed using our proposed image-based fingerprints, which define the initial pivot for the word alignment. Second, we apply our robust Procrustes algorithm to estimate the linear mapping between two embedding spaces, which iteratively corrects and refines the estimated alignment. Our extensive experiments show that WALIP improves upon the state-of-the-art performance of bilingual word alignment for a few language pairs across different word embeddings and displays great robustness to the dissimilarity of language pairs or training corpora for two word embeddings.
△ Less
Submitted 7 November, 2022; v1 submitted 23 May, 2022;
originally announced May 2022.
-
Minibatch vs Local SGD with Shuffling: Tight Convergence Bounds and Beyond
Authors:
Chulhee Yun,
Shashank Rajput,
Suvrit Sra
Abstract:
In distributed learning, local SGD (also known as federated averaging) and its simple baseline minibatch SGD are widely studied optimization methods. Most existing analyses of these methods assume independent and unbiased gradient estimates obtained via with-replacement sampling. In contrast, we study shuffling-based variants: minibatch and local Random Reshuffling, which draw stochastic gradients…
▽ More
In distributed learning, local SGD (also known as federated averaging) and its simple baseline minibatch SGD are widely studied optimization methods. Most existing analyses of these methods assume independent and unbiased gradient estimates obtained via with-replacement sampling. In contrast, we study shuffling-based variants: minibatch and local Random Reshuffling, which draw stochastic gradients without replacement and are thus closer to practice. For smooth functions satisfying the Polyak-Łojasiewicz condition, we obtain convergence bounds (in the large epoch regime) which show that these shuffling-based variants converge faster than their with-replacement counterparts. Moreover, we prove matching lower bounds showing that our convergence analysis is tight. Finally, we propose an algorithmic modification called synchronized shuffling that leads to convergence rates faster than our lower bounds in near-homogeneous settings.
△ Less
Submitted 23 March, 2022; v1 submitted 19 October, 2021;
originally announced October 2021.
-
Finding Everything within Random Binary Networks
Authors:
Kartik Sreenivasan,
Shashank Rajput,
Jy-yong Sohn,
Dimitris Papailiopoulos
Abstract:
A recent work by Ramanujan et al. (2020) provides significant empirical evidence that sufficiently overparameterized, random neural networks contain untrained subnetworks that achieve state-of-the-art accuracy on several predictive tasks. A follow-up line of theoretical work provides justification of these findings by proving that slightly overparameterized neural networks, with commonly used cont…
▽ More
A recent work by Ramanujan et al. (2020) provides significant empirical evidence that sufficiently overparameterized, random neural networks contain untrained subnetworks that achieve state-of-the-art accuracy on several predictive tasks. A follow-up line of theoretical work provides justification of these findings by proving that slightly overparameterized neural networks, with commonly used continuous-valued random initializations can indeed be pruned to approximate any target network. In this work, we show that the amplitude of those random weights does not even matter. We prove that any target network can be approximated up to arbitrary accuracy by simply pruning a random network of binary $\{\pm1\}$ weights that is only a polylogarithmic factor wider and deeper than the target network.
△ Less
Submitted 22 October, 2021; v1 submitted 17 October, 2021;
originally announced October 2021.
-
An Exponential Improvement on the Memorization Capacity of Deep Threshold Networks
Authors:
Shashank Rajput,
Kartik Sreenivasan,
Dimitris Papailiopoulos,
Amin Karbasi
Abstract:
It is well known that modern deep neural networks are powerful enough to memorize datasets even when the labels have been randomized. Recently, Vershynin (2020) settled a long standing question by Baum (1988), proving that \emph{deep threshold} networks can memorize $n$ points in $d$ dimensions using $\widetilde{\mathcal{O}}(e^{1/δ^2}+\sqrt{n})$ neurons and…
▽ More
It is well known that modern deep neural networks are powerful enough to memorize datasets even when the labels have been randomized. Recently, Vershynin (2020) settled a long standing question by Baum (1988), proving that \emph{deep threshold} networks can memorize $n$ points in $d$ dimensions using $\widetilde{\mathcal{O}}(e^{1/δ^2}+\sqrt{n})$ neurons and $\widetilde{\mathcal{O}}(e^{1/δ^2}(d+\sqrt{n})+n)$ weights, where $δ$ is the minimum distance between the points. In this work, we improve the dependence on $δ$ from exponential to almost linear, proving that $\widetilde{\mathcal{O}}(\frac{1}δ+\sqrt{n})$ neurons and $\widetilde{\mathcal{O}}(\frac{d}δ+n)$ weights are sufficient. Our construction uses Gaussian random weights only in the first layer, while all the subsequent layers use binary or integer weights. We also prove new lower bounds by connecting memorization in neural networks to the purely geometric problem of separating $n$ points on a sphere using hyperplanes.
△ Less
Submitted 14 June, 2021;
originally announced June 2021.
-
IndoFashion : Apparel Classification for Indian Ethnic Clothes
Authors:
Pranjal Singh Rajput,
Shivangi Aneja
Abstract:
Cloth categorization is an important research problem that is used by e-commerce websites for displaying correct products to the end-users. Indian clothes have a large number of clothing categories both for men and women. The traditional Indian clothes like "Saree" and "Dhoti" are worn very differently from western clothes like t-shirts and jeans. Moreover, the style and patterns of ethnic clothes…
▽ More
Cloth categorization is an important research problem that is used by e-commerce websites for displaying correct products to the end-users. Indian clothes have a large number of clothing categories both for men and women. The traditional Indian clothes like "Saree" and "Dhoti" are worn very differently from western clothes like t-shirts and jeans. Moreover, the style and patterns of ethnic clothes have a very different distribution from western outfits. Thus the models trained on standard cloth datasets fail miserably on ethnic outfits. To address these challenges, we introduce the first large-scale ethnic dataset of over 106k images with 15 different categories for fine-grained classification of Indian ethnic clothes. We gathered a diverse dataset from a large number of Indian e-commerce websites. We then evaluate several baselines for the cloth classification task on our dataset. In the end, we obtain 88.43% classification accuracy. We hope that our dataset would foster research in the development of several algorithms such as cloth classification, landmark detection, especially for ethnic clothes.
△ Less
Submitted 6 April, 2021;
originally announced April 2021.
-
Heuristics2Annotate: Efficient Annotation of Large-Scale Marathon Dataset For Bounding Box Regression
Authors:
Pranjal Singh Rajput,
Yeshwanth Napolean,
Jan van Gemert
Abstract:
Annotating a large-scale in-the-wild person re-identification dataset especially of marathon runners is a challenging task. The variations in the scenarios such as camera viewpoints, resolution, occlusion, and illumination make the problem non-trivial. Manually annotating bounding boxes in such large-scale datasets is cost-inefficient. Additionally, due to crowdedness and occlusion in the videos,…
▽ More
Annotating a large-scale in-the-wild person re-identification dataset especially of marathon runners is a challenging task. The variations in the scenarios such as camera viewpoints, resolution, occlusion, and illumination make the problem non-trivial. Manually annotating bounding boxes in such large-scale datasets is cost-inefficient. Additionally, due to crowdedness and occlusion in the videos, aligning the identity of runners across multiple disjoint cameras is a challenge. We collected a novel large-scale in-the-wild video dataset of marathon runners. The dataset consists of hours of recording of thousands of runners captured using 42 hand-held smartphone cameras and covering real-world scenarios. Due to the presence of crowdedness and occlusion in the videos, the annotation of runners becomes a challenging task. We propose a new scheme for tackling the challenges in the annotation of such large dataset. Our technique reduces the overall cost of annotation in terms of time as well as budget. We demonstrate performing fps analysis to reduce the effort and time of annotation. We investigate several annotation methods for efficiently generating tight bounding boxes. Our results prove that interpolating bounding boxes between keyframes is the most efficient method of bounding box generation amongst several other methods and is 3x times faster than the naive baseline method. We introduce a novel way of aligning the identity of runners in disjoint cameras. Our inter-camera alignment tool integrated with the state-of-the-art person re-id system proves to be sufficient and effective in the alignment of the runners across multiple cameras with non-overlapping views. Our proposed framework of annotation reduces the annotation cost of the dataset by a factor of 16x, also effectively aligning 93.64% of the runners in the cross-camera setting.
△ Less
Submitted 6 April, 2021;
originally announced April 2021.
-
PVG at WASSA 2021: A Multi-Input, Multi-Task, Transformer-Based Architecture for Empathy and Distress Prediction
Authors:
Atharva Kulkarni,
Sunanda Somwase,
Shivam Rajput,
Manisha Marathe
Abstract:
Active research pertaining to the affective phenomenon of empathy and distress is invaluable for improving human-machine interaction. Predicting intensities of such complex emotions from textual data is difficult, as these constructs are deeply rooted in the psychological theory. Consequently, for better prediction, it becomes imperative to take into account ancillary factors such as the psycholog…
▽ More
Active research pertaining to the affective phenomenon of empathy and distress is invaluable for improving human-machine interaction. Predicting intensities of such complex emotions from textual data is difficult, as these constructs are deeply rooted in the psychological theory. Consequently, for better prediction, it becomes imperative to take into account ancillary factors such as the psychological test scores, demographic features, underlying latent primitive emotions, along with the text's undertone and its psychological complexity. This paper proffers team PVG's solution to the WASSA 2021 Shared Task on Predicting Empathy and Emotion in Reaction to News Stories. Leveraging the textual data, demographic features, psychological test score, and the intrinsic interdependencies of primitive emotions and empathy, we propose a multi-input, multi-task framework for the task of empathy score prediction. Here, the empathy score prediction is considered the primary task, while emotion and empathy classification are considered secondary auxiliary tasks. For the distress score prediction task, the system is further boosted by the addition of lexical features. Our submission ranked 1$^{st}$ based on the average correlation (0.545) as well as the distress correlation (0.574), and 2$^{nd}$ for the empathy Pearson correlation (0.517).
△ Less
Submitted 4 March, 2021;
originally announced March 2021.
-
Permutation-Based SGD: Is Random Optimal?
Authors:
Shashank Rajput,
Kangwook Lee,
Dimitris Papailiopoulos
Abstract:
A recent line of ground-breaking results for permutation-based SGD has corroborated a widely observed phenomenon: random permutations offer faster convergence than with-replacement sampling. However, is random optimal? We show that this depends heavily on what functions we are optimizing, and the convergence gap between optimal and random permutations can vary from exponential to nonexistent. We f…
▽ More
A recent line of ground-breaking results for permutation-based SGD has corroborated a widely observed phenomenon: random permutations offer faster convergence than with-replacement sampling. However, is random optimal? We show that this depends heavily on what functions we are optimizing, and the convergence gap between optimal and random permutations can vary from exponential to nonexistent. We first show that for 1-dimensional strongly convex functions, with smooth second derivatives, there exist permutations that offer exponentially faster convergence compared to random. However, for general strongly convex functions, random permutations are optimal. Finally, we show that for quadratic, strongly-convex functions, there are easy-to-construct permutations that lead to accelerated convergence compared to random. Our results suggest that a general convergence characterization of optimal permutations cannot capture the nuances of individual function classes, and can mistakenly indicate that one cannot do much better than random.
△ Less
Submitted 24 November, 2021; v1 submitted 18 February, 2021;
originally announced February 2021.
-
Roughsets-based Approach for Predicting Battery Life in IoT
Authors:
Rajesh Kaluri,
Dharmendra Singh Rajput,
Qin Xin,
Kuruva Lakshmanna,
Sweta Bhattacharya,
Thippa Reddy Gadekallu,
Praveen Kumar Reddy Maddikunta
Abstract:
Internet of Things (IoT) and related applications have successfully contributed towards enhancing the value of life in this planet. The advanced wireless sensor networks and its revolutionary computational capabilities have enabled various IoT applications become the next frontier, touching almost all domains of life. With this enormous progress, energy optimization has also become a primary conce…
▽ More
Internet of Things (IoT) and related applications have successfully contributed towards enhancing the value of life in this planet. The advanced wireless sensor networks and its revolutionary computational capabilities have enabled various IoT applications become the next frontier, touching almost all domains of life. With this enormous progress, energy optimization has also become a primary concern with the need to attend to green technologies. The present study focuses on the predictions pertinent to the sustainability of battery life in IoT frameworks in the marine environment. The data used is a publicly available dataset collected from the Chicago district beach water. Firstly, the missing values in the data are replaced with the attribute mean. Later, one-hot encoding technique is applied for achieving data homogeneity followed by the standard scalar technique to normalize the data. Then, rough set theory is used for feature extraction, and the resultant data is fed into a Deep Neural Network (DNN) model for the optimized prediction results. The proposed model is then compared with the state of the art machine learning models and the results justify its superiority on the basis of performance metrics such as Mean Squared Error, Mean Absolute Error, Root Mean Squared Error, and Test Variance Score.
△ Less
Submitted 8 February, 2021;
originally announced February 2021.
-
Glioblastoma Multiforme Patient Survival Prediction
Authors:
Snehal Rajput,
Rupal Agravat,
Mohendra Roy,
Mehul S Raval
Abstract:
Glioblastoma Multiforme is a very aggressive type of brain tumor. Due to spatial and temporal intra-tissue inhomogeneity, location and the extent of the cancer tissue, it is difficult to detect and dissect the tumor regions. In this paper, we propose survival prognosis models using four regressors operating on handcrafted image-based and radiomics features. We hypothesize that the radiomics shape…
▽ More
Glioblastoma Multiforme is a very aggressive type of brain tumor. Due to spatial and temporal intra-tissue inhomogeneity, location and the extent of the cancer tissue, it is difficult to detect and dissect the tumor regions. In this paper, we propose survival prognosis models using four regressors operating on handcrafted image-based and radiomics features. We hypothesize that the radiomics shape features have the highest correlation with survival prediction. The proposed approaches were assessed on the Brain Tumor Segmentation (BraTS-2020) challenge dataset. The highest accuracy of image features with random forest regressor approach was 51.5\% for the training and 51.7\% for the validation dataset. The gradient boosting regressor with shape features gave an accuracy of 91.5\% and 62.1\% on training and validation datasets respectively. It is better than the BraTS 2020 survival prediction challenge winners on the training and validation datasets. Our work shows that handcrafted features exhibit a strong correlation with survival prediction. The consensus based regressor with gradient boosting and radiomics shape features is the best combination for survival prediction.
△ Less
Submitted 26 January, 2021;
originally announced January 2021.
-
A Survey of Novel Cache Hierarchy Designs for High Workloads
Authors:
Pranjal Singh Rajput,
Sonnya Dellarosa,
Kanya Satis
Abstract:
Traditional on-die, three-level cache hierarchy design is very commonly used but is also prone to latency, especially at the Level 2 (L2) cache. We discuss three distinct ways of improving this design in order to have better performance. Performance is especially important for systems with high workloads. The first method proposes to eliminate L2 altogether while proposing a new prefetching techni…
▽ More
Traditional on-die, three-level cache hierarchy design is very commonly used but is also prone to latency, especially at the Level 2 (L2) cache. We discuss three distinct ways of improving this design in order to have better performance. Performance is especially important for systems with high workloads. The first method proposes to eliminate L2 altogether while proposing a new prefetching technique, the second method suggests increasing the size of L2, while the last method advocates the implementation of optical caches. After carefully contemplating the results in performance gains and the advantages and disadvantages of each method, we found the last method to be the best of the three.
△ Less
Submitted 24 January, 2021;
originally announced January 2021.
-
cGANs for Cartoon to Real-life Images
Authors:
Pranjal Singh Rajput,
Kanya Satis,
Sonnya Dellarosa,
Wenxuan Huang,
Obinna Agba
Abstract:
The image-to-image translation is a learning task to establish a visual mapping between an input and output image. The task has several variations differentiated based on the purpose of the translation, such as synthetic to real translation, photo to caricature translation, and many others. The problem has been tackled using different approaches, either through traditional computer vision methods,…
▽ More
The image-to-image translation is a learning task to establish a visual mapping between an input and output image. The task has several variations differentiated based on the purpose of the translation, such as synthetic to real translation, photo to caricature translation, and many others. The problem has been tackled using different approaches, either through traditional computer vision methods, as well as deep learning approaches in recent trends. One approach currently deemed popular and effective is using the conditional generative adversarial network, also known shortly as cGAN. It is adapted to perform image-to-image translation tasks with typically two networks: a generator and a discriminator. This project aims to evaluate the robustness of the Pix2Pix model by applying the Pix2Pix model to datasets consisting of cartoonized images. Using the Pix2Pix model, it should be possible to train the network to generate real-life images from the cartoonized images.
△ Less
Submitted 24 January, 2021;
originally announced January 2021.
-
A Detail Study of Security and Privacy issues of Internet of Things
Authors:
Mohan Krishna Kagita,
Navod Thilakarathne,
Dharmendra Singh Rajput,
Dr Surekha Lanka
Abstract:
The Internet of Things, or IoT, refers to the billions of physical objects around the planet that are now connected to the Internet, many of which store and exchange the data without human interaction. In recent years the Internet of Things (IoT) has incredibly become a groundbreaking technical innovation that has contributed to massive impact in the ways where all the information is handled incor…
▽ More
The Internet of Things, or IoT, refers to the billions of physical objects around the planet that are now connected to the Internet, many of which store and exchange the data without human interaction. In recent years the Internet of Things (IoT) has incredibly become a groundbreaking technical innovation that has contributed to massive impact in the ways where all the information is handled incorporate companies, computer devices, and even kitchen equipment and appliances, are designed and made. The main focus of this chapter is to systematically review the security and privacy of the Internet of Things in the present world. Most internet users are genuine, yet others are cybercriminals with individual expectations of misusing information. With such possibilities, users should know the potential security and privacy issues of IoT devices. IoT innovations are applied on numerous levels in a system that we use daily in our day-to-day life. Data confidentiality is a significant issue. The interconnection of various networks makes it impossible for users to assert extensive control of their data. Finally, this chapter discusses the IoT Security concerns in the literature and providing a critical review of the current approach and proposed solutions on present issues on the Privacy protection of IoT devices.
△ Less
Submitted 14 September, 2020;
originally announced September 2020.
-
Attack of the Tails: Yes, You Really Can Backdoor Federated Learning
Authors:
Hongyi Wang,
Kartik Sreenivasan,
Shashank Rajput,
Harit Vishwakarma,
Saurabh Agarwal,
Jy-yong Sohn,
Kangwook Lee,
Dimitris Papailiopoulos
Abstract:
Due to its decentralized nature, Federated Learning (FL) lends itself to adversarial attacks in the form of backdoors during training. The goal of a backdoor is to corrupt the performance of the trained model on specific sub-tasks (e.g., by classifying green cars as frogs). A range of FL backdoor attacks have been introduced in the literature, but also methods to defend against them, and it is cur…
▽ More
Due to its decentralized nature, Federated Learning (FL) lends itself to adversarial attacks in the form of backdoors during training. The goal of a backdoor is to corrupt the performance of the trained model on specific sub-tasks (e.g., by classifying green cars as frogs). A range of FL backdoor attacks have been introduced in the literature, but also methods to defend against them, and it is currently an open question whether FL systems can be tailored to be robust against backdoors. In this work, we provide evidence to the contrary. We first establish that, in the general case, robustness to backdoors implies model robustness to adversarial examples, a major open problem in itself. Furthermore, detecting the presence of a backdoor in a FL model is unlikely assuming first order oracles or polynomial time. We couple our theoretical results with a new family of backdoor attacks, which we refer to as edge-case backdoors. An edge-case backdoor forces a model to misclassify on seemingly easy inputs that are however unlikely to be part of the training, or test data, i.e., they live on the tail of the input distribution. We explain how these edge-case backdoors can lead to unsavory failures and may have serious repercussions on fairness, and exhibit that with careful tuning at the side of the adversary, one can insert them across a range of machine learning tasks (e.g., image classification, OCR, text prediction, sentiment analysis).
△ Less
Submitted 9 July, 2020;
originally announced July 2020.
-
Optimal Lottery Tickets via SubsetSum: Logarithmic Over-Parameterization is Sufficient
Authors:
Ankit Pensia,
Shashank Rajput,
Alliot Nagle,
Harit Vishwakarma,
Dimitris Papailiopoulos
Abstract:
The strong {\it lottery ticket hypothesis} (LTH) postulates that one can approximate any target neural network by only pruning the weights of a sufficiently over-parameterized random network. A recent work by Malach et al. \cite{MalachEtAl20} establishes the first theoretical analysis for the strong LTH: one can provably approximate a neural network of width $d$ and depth $l$, by pruning a random…
▽ More
The strong {\it lottery ticket hypothesis} (LTH) postulates that one can approximate any target neural network by only pruning the weights of a sufficiently over-parameterized random network. A recent work by Malach et al. \cite{MalachEtAl20} establishes the first theoretical analysis for the strong LTH: one can provably approximate a neural network of width $d$ and depth $l$, by pruning a random one that is a factor $O(d^4l^2)$ wider and twice as deep. This polynomial over-parameterization requirement is at odds with recent experimental research that achieves good approximation with networks that are a small factor wider than the target. In this work, we close the gap and offer an exponential improvement to the over-parameterization requirement for the existence of lottery tickets. We show that any target network of width $d$ and depth $l$ can be approximated by pruning a random network that is a factor $O(\log(dl))$ wider and twice as deep. Our analysis heavily relies on connecting pruning random ReLU networks to random instances of the \textsc{SubsetSum} problem. We then show that this logarithmic over-parameterization is essentially optimal for constant depth networks. Finally, we verify several of our theoretical insights with experiments.
△ Less
Submitted 11 March, 2021; v1 submitted 14 June, 2020;
originally announced June 2020.
-
A Review on End-To-End Methods for Brain Tumor Segmentation and Overall Survival Prediction
Authors:
Snehal Rajput,
Mehul S Raval
Abstract:
Brain tumor segmentation intends to delineate tumor tissues from healthy brain tissues. The tumor tissues include necrosis, peritumoral edema, and active tumor. In contrast, healthy brain tissues include white matter, gray matter, and cerebrospinal fluid. The MRI based brain tumor segmentation research is gaining popularity as; 1. It does not irradiate ionized radiation like X-ray or computed tomo…
▽ More
Brain tumor segmentation intends to delineate tumor tissues from healthy brain tissues. The tumor tissues include necrosis, peritumoral edema, and active tumor. In contrast, healthy brain tissues include white matter, gray matter, and cerebrospinal fluid. The MRI based brain tumor segmentation research is gaining popularity as; 1. It does not irradiate ionized radiation like X-ray or computed tomography imaging. 2. It produces detailed pictures of internal body structures. The MRI scans are input to deep learning-based approaches which are useful for automatic brain tumor segmentation. The features from segments are fed to the classifier which predict the overall survival of the patient. The motive of this paper is to give an extensive overview of state-of-the-art jointly covering brain tumor segmentation and overall survival prediction.
△ Less
Submitted 31 May, 2020;
originally announced June 2020.
-
HLVU : A New Challenge to Test Deep Understanding of Movies the Way Humans do
Authors:
Keith Curtis,
George Awad,
Shahzad Rajput,
Ian Soboroff
Abstract:
In this paper we propose a new evaluation challenge and direction in the area of High-level Video Understanding. The challenge we are proposing is designed to test automatic video analysis and understanding, and how accurately systems can comprehend a movie in terms of actors, entities, events and their relationship to each other. A pilot High-Level Video Understanding (HLVU) dataset of open sourc…
▽ More
In this paper we propose a new evaluation challenge and direction in the area of High-level Video Understanding. The challenge we are proposing is designed to test automatic video analysis and understanding, and how accurately systems can comprehend a movie in terms of actors, entities, events and their relationship to each other. A pilot High-Level Video Understanding (HLVU) dataset of open source movies were collected for human assessors to build a knowledge graph representing each of them. A set of queries will be derived from the knowledge graph to test systems on retrieving relationships among actors, as well as reasoning and retrieving non-visual concepts. The objective is to benchmark if a computer system can "understand" non-explicit but obvious relationships the same way humans do when they watch the same movies. This is long-standing problem that is being addressed in the text domain and this project moves similar research to the video domain. Work of this nature is foundational to future video analytics and video understanding technologies. This work can be of interest to streaming services and broadcasters hoping to provide more intuitive ways for their customers to interact with and consume video content.
△ Less
Submitted 1 May, 2020;
originally announced May 2020.
-
Closing the convergence gap of SGD without replacement
Authors:
Shashank Rajput,
Anant Gupta,
Dimitris Papailiopoulos
Abstract:
Stochastic gradient descent without replacement sampling is widely used in practice for model training. However, the vast majority of SGD analyses assumes data is sampled with replacement, and when the function minimized is strongly convex, an $\mathcal{O}\left(\frac{1}{T}\right)$ rate can be established when SGD is run for $T$ iterations. A recent line of breakthrough works on SGD without replace…
▽ More
Stochastic gradient descent without replacement sampling is widely used in practice for model training. However, the vast majority of SGD analyses assumes data is sampled with replacement, and when the function minimized is strongly convex, an $\mathcal{O}\left(\frac{1}{T}\right)$ rate can be established when SGD is run for $T$ iterations. A recent line of breakthrough works on SGD without replacement (SGDo) established an $\mathcal{O}\left(\frac{n}{T^2}\right)$ convergence rate when the function minimized is strongly convex and is a sum of $n$ smooth functions, and an $\mathcal{O}\left(\frac{1}{T^2}+\frac{n^3}{T^3}\right)$ rate for sums of quadratics. On the other hand, the tightest known lower bound postulates an $Ω\left(\frac{1}{T^2}+\frac{n^2}{T^3}\right)$ rate, leaving open the possibility of better SGDo convergence rates in the general case. In this paper, we close this gap and show that SGD without replacement achieves a rate of $\mathcal{O}\left(\frac{1}{T^2}+\frac{n^2}{T^3}\right)$ when the sum of the functions is a quadratic, and offer a new lower bound of $Ω\left(\frac{n}{T^2}\right)$ for strongly convex functions that are sums of smooth functions.
△ Less
Submitted 9 July, 2020; v1 submitted 24 February, 2020;
originally announced February 2020.
-
DETOX: A Redundancy-based Framework for Faster and More Robust Gradient Aggregation
Authors:
Shashank Rajput,
Hongyi Wang,
Zachary Charles,
Dimitris Papailiopoulos
Abstract:
To improve the resilience of distributed training to worst-case, or Byzantine node failures, several recent approaches have replaced gradient averaging with robust aggregation methods. Such techniques can have high computational costs, often quadratic in the number of compute nodes, and only have limited robustness guarantees. Other methods have instead used redundancy to guarantee robustness, but…
▽ More
To improve the resilience of distributed training to worst-case, or Byzantine node failures, several recent approaches have replaced gradient averaging with robust aggregation methods. Such techniques can have high computational costs, often quadratic in the number of compute nodes, and only have limited robustness guarantees. Other methods have instead used redundancy to guarantee robustness, but can only tolerate limited number of Byzantine failures. In this work, we present DETOX, a Byzantine-resilient distributed training framework that combines algorithmic redundancy with robust aggregation. DETOX operates in two steps, a filtering step that uses limited redundancy to significantly reduce the effect of Byzantine nodes, and a hierarchical aggregation step that can be used in tandem with any state-of-the-art robust aggregation method. We show theoretically that this leads to a substantial increase in robustness, and has a per iteration runtime that can be nearly linear in the number of compute nodes. We provide extensive experiments over real distributed setups across a variety of large-scale machine learning tasks, showing that DETOX leads to orders of magnitude accuracy and speedup improvements over many state-of-the-art Byzantine-resilient approaches.
△ Less
Submitted 7 March, 2020; v1 submitted 29 July, 2019;
originally announced July 2019.
-
Convergence and Margin of Adversarial Training on Separable Data
Authors:
Zachary Charles,
Shashank Rajput,
Stephen Wright,
Dimitris Papailiopoulos
Abstract:
Adversarial training is a technique for training robust machine learning models. To encourage robustness, it iteratively computes adversarial examples for the model, and then re-trains on these examples via some update rule. This work analyzes the performance of adversarial training on linearly separable data, and provides bounds on the number of iterations required for large margin. We show that…
▽ More
Adversarial training is a technique for training robust machine learning models. To encourage robustness, it iteratively computes adversarial examples for the model, and then re-trains on these examples via some update rule. This work analyzes the performance of adversarial training on linearly separable data, and provides bounds on the number of iterations required for large margin. We show that when the update rule is given by an arbitrary empirical risk minimizer, adversarial training may require exponentially many iterations to obtain large margin. However, if gradient or stochastic gradient update rules are used, only polynomially many iterations are required to find a large-margin separator. By contrast, without the use of adversarial examples, gradient methods may require exponentially many iterations to achieve large margin. Our results are derived by showing that adversarial training with gradient updates minimizes a robust version of the empirical risk at a $\mathcal{O}(\ln(t)^2/t)$ rate, despite non-smoothness. We corroborate our theory empirically.
△ Less
Submitted 22 May, 2019;
originally announced May 2019.
-
Does Data Augmentation Lead to Positive Margin?
Authors:
Shashank Rajput,
Zhili Feng,
Zachary Charles,
Po-Ling Loh,
Dimitris Papailiopoulos
Abstract:
Data augmentation (DA) is commonly used during model training, as it significantly improves test error and model robustness. DA artificially expands the training set by applying random noise, rotations, crops, or even adversarial perturbations to the input data. Although DA is widely used, its capacity to provably improve robustness is not fully understood. In this work, we analyze the robustness…
▽ More
Data augmentation (DA) is commonly used during model training, as it significantly improves test error and model robustness. DA artificially expands the training set by applying random noise, rotations, crops, or even adversarial perturbations to the input data. Although DA is widely used, its capacity to provably improve robustness is not fully understood. In this work, we analyze the robustness that DA begets by quantifying the margin that DA enforces on empirical risk minimizers. We first focus on linear separators, and then a class of nonlinear models whose labeling is constant within small convex hulls of data points. We present lower bounds on the number of augmented data points required for non-zero margin, and show that commonly used DA techniques may only introduce significant margin after adding exponentially many points to the data set.
△ Less
Submitted 8 May, 2019;
originally announced May 2019.
-
On Arrhythmia Detection by Deep Learning and Multidimensional Representation
Authors:
K. S. Rajput,
S. Wibowo,
C. Hao,
M. Majmudar
Abstract:
An electrocardiogram (ECG) is a time-series signal that is represented by one-dimensional (1-D) data. Higher dimensional representation contains more information that is accessible for feature extraction. Hidden variables such as frequency relation and morphology of segment is not directly accessible in the time domain. In this paper, 1-D time series data is converted into multi-dimensional repres…
▽ More
An electrocardiogram (ECG) is a time-series signal that is represented by one-dimensional (1-D) data. Higher dimensional representation contains more information that is accessible for feature extraction. Hidden variables such as frequency relation and morphology of segment is not directly accessible in the time domain. In this paper, 1-D time series data is converted into multi-dimensional representation in the form of multichannel 2-D images. Following that, deep learning was used to train a deep neural network based classifier to detect arrhythmias. The results of simulation on testing database demonstrate the effectiveness of the proposed methodology by showing an outstanding classification performance compared to other existing methods and hand-crafted annotations made by certified cardiologists.
△ Less
Submitted 11 April, 2019; v1 submitted 29 March, 2019;
originally announced April 2019.