Statistics
See recent articles
- [1] arXiv:2408.10218 [pdf, html, other]
-
Title: Constructive and consistent estimation of quadratic minimaxComments: arXiv admin note: substantial text overlap with arXiv:2307.15350Subjects: Statistics Theory (math.ST); Probability (math.PR)
We consider $k$ square integrable random variables $Y_1,...,Y_k$ and $k$ random (row) vectors of length $p$, $X_1,...,X_k$ such that $X_i(l)$ is square integrable for $1\le i\le k$ and $1\le l\le p$. No assumptions whatsoever are made of any relationship between the $X_i$:s and $Y_i$:s. We shall refer to each pairing of $X_i$ and $Y_i$ as an environment. We form the square risk functions $R_i(\beta)=\mathbb{E}\left[(Y_i-\beta X_i)^2\right]$ for every environment and consider $m$ affine combinations of these $k$ risk functions. Next, we define a parameter space $\Theta$ where we associate each point with a subset of the unique elements of the covariance matrix of $(X_i,Y_i)$ for an environment. Then we study estimation of the $\arg\min$-solution set of the maximum of a the $m$ affine combinations the of quadratic risk functions. We provide a constructive method for estimating the entire $\arg\min$-solution set which is consistent almost surely outside a zero set in $\Theta^k$. This method is computationally expensive, since it involves solving polynomials of general degree. To overcome this, we define another approximate estimator that also provides a consistent estimation of the solution set based on the bisection method, which is computationally much more efficient. We apply the method to worst risk minimization in the setting of structural equation models.
- [2] arXiv:2408.10340 [pdf, html, other]
-
Title: Can an unsupervised clustering algorithm reproduce a categorization system?Comments: 9 pages, 4 tables 28 figuresSubjects: Machine Learning (stat.ML); Machine Learning (cs.LG); Statistical Finance (q-fin.ST); Applications (stat.AP)
Peer analysis is a critical component of investment management, often relying on expert-provided categorization systems. These systems' consistency is questioned when they do not align with cohorts from unsupervised clustering algorithms optimized for various metrics. We investigate whether unsupervised clustering can reproduce ground truth classes in a labeled dataset, showing that success depends on feature selection and the chosen distance metric. Using toy datasets and fund categorization as real-world examples we demonstrate that accurately reproducing ground truth classes is challenging. We also highlight the limitations of standard clustering evaluation metrics in identifying the optimal number of clusters relative to the ground truth classes. We then show that if appropriate features are available in the dataset, and a proper distance metric is known (e.g., using a supervised Random Forest-based distance metric learning method), then an unsupervised clustering can indeed reproduce the ground truth classes as distinct clusters.
- [3] arXiv:2408.10381 [pdf, html, other]
-
Title: Efficient Reinforcement Learning in Probabilistic Reward MachinesComments: 33 pages, 4 figuresSubjects: Machine Learning (stat.ML); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
In this paper, we study reinforcement learning in Markov Decision Processes with Probabilistic Reward Machines (PRMs), a form of non-Markovian reward commonly found in robotics tasks. We design an algorithm for PRMs that achieves a regret bound of $\widetilde{O}(\sqrt{HOAT} + H^2O^2A^{3/2} + H\sqrt{T})$, where $H$ is the time horizon, $O$ is the number of observations, $A$ is the number of actions, and $T$ is the number of time-steps. This result improves over the best-known bound, $\widetilde{O}(H\sqrt{OAT})$ of \citet{pmlr-v206-bourel23a} for MDPs with Deterministic Reward Machines (DRMs), a special case of PRMs. When $T \geq H^3O^3A^2$ and $OA \geq H$, our regret bound leads to a regret of $\widetilde{O}(\sqrt{HOAT})$, which matches the established lower bound of $\Omega(\sqrt{HOAT})$ for MDPs with DRMs up to a logarithmic factor. To the best of our knowledge, this is the first efficient algorithm for PRMs. Additionally, we present a new simulation lemma for non-Markovian rewards, which enables reward-free exploration for any non-Markovian reward given access to an approximate planner. Complementing our theoretical findings, we show through extensive experiment evaluations that our algorithm indeed outperforms prior methods in various PRM environments.
- [4] arXiv:2408.10396 [pdf, other]
-
Title: Highly Multivariate High-dimensionality Spatial Stochastic Processes-A Mixed Conditional ApproachComments: 46 pages; 11 figuresSubjects: Methodology (stat.ME); Statistics Theory (math.ST)
We propose a hybrid mixed spatial graphical model framework and novel concepts, e.g., cross-Markov Random Field (cross-MRF), to comprehensively address all feature aspects of highly multivariate high-dimensionality (HMHD) spatial data class when constructing the desired joint variance and precision matrix (where both p and n are large). Specifically, the framework accommodates any customized conditional independence (CI) among any number of p variate fields at the first stage, alleviating dynamic memory burden. Meanwhile, it facilitates parallel generation of covariance and precision matrix, with the latter's generation order scaling only linearly in p. In the second stage, we demonstrate the multivariate Hammersley-Clifford theorem from a column-wise conditional perspective and unearth the existence of cross-MRF. The link of the mixed spatial graphical framework and the cross-MRF allows for a mixed conditional approach, resulting in the sparsest possible representation of the precision matrix via accommodating the doubly CI among both p and n, with the highest possible exact-zero-value percentage. We also explore the possibility of the co-existence of geostatistical and MRF modelling approaches in one unified framework, imparting a potential solution to an open problem. The derived theories are illustrated with 1D simulation and 2D real-world spatial data.
- [5] arXiv:2408.10401 [pdf, html, other]
-
Title: Spatial Knockoff Bayesian Variable Selection in Genome-Wide Association StudiesComments: 32 pages and 6 figuresSubjects: Methodology (stat.ME)
High-dimensional variable selection has emerged as one of the prevailing statistical challenges in the big data revolution. Many variable selection methods have been adapted for identifying single nucleotide polymorphisms (SNPs) linked to phenotypic variation in genome-wide association studies. We develop a Bayesian variable selection regression model for identifying SNPs linked to phenotypic variation. We modify our Bayesian variable selection regression models to control the false discovery rate of SNPs using a knockoff variable approach. We reduce spurious associations by regressing the phenotype of interest against a set of basis functions that account for the relatedness of individuals. Using a restricted regression approach, we simultaneously estimate the SNP-level effects while removing variation in the phenotype that can be explained by population structure. We also accommodate the spatial structure among causal SNPs by modeling their inclusion probabilities jointly with a reduced rank Gaussian process. In a simulation study, we demonstrate that our spatial Bayesian variable selection regression model controls the false discovery rate and increases power when the relevant SNPs are clustered. We conclude with an analysis of Arabidopsis thaliana flowering time, a polygenic trait that is confounded with population structure, and find the discoveries of our method cluster near described flowering time genes.
- [6] arXiv:2408.10416 [pdf, html, other]
-
Title: Issues of parameterization and computation for posterior inference in partially identified modelsSubjects: Computation (stat.CO)
A partially identified model, where the parameters can not be uniquely identified, often arises during statistical analysis. While researchers frequently use Bayesian inference to analyze the models, when Bayesian inference with an off-the-shelf MCMC sampling algorithm is applied to a partially identified model, the computational performance can be poor. It is found that using importance sampling with transparent reparameterization (TP) is one remedy. This method is preferable since the model is known to be rendered as identified with respect to the new parameterization, and at the same time, it may allow faster, i.i.d. Monte Carlo sampling by using conjugate convenience priors. In this paper, we explain the importance sampling method with the TP and a pseudo-TP. We introduce the pseudo-TP, an alternative to TP, since finding a TP is sometimes difficult. Then, we test the methods' performance in some scenarios and compare it to the performance of the off-the-shelf MCMC method - Gibbs sampling - applied in the original parameterization. While the importance sampling with TP (ISTP) shows generally better results than off-the-shelf MCMC methods, as seen in the compute time and trace plots, it is also seen that finding a TP which is necessary for the method may not be easy. On the other hand, the pseudo-TP method shows a mixed result and room for improvement since it relies on an approximation, which may not be adequate for a given model and dataset.
- [7] arXiv:2408.10478 [pdf, html, other]
-
Title: On a fundamental difference between Bayesian and frequentist approaches to robustnessSubjects: Methodology (stat.ME)
Heavy-tailed models are often used as a way to gain robustness against outliers in Bayesian analyses. On the other side, in frequentist analyses, M-estimators are often employed. In this paper, the two approaches are reconciled by considering M-estimators as maximum likelihood estimators of heavy-tailed models. We realize that, even from this perspective, there is a fundamental difference in that frequentists do not require these heavy-tailed models to be proper. It is shown what the difference between improper and proper heavy-tailed models can be in terms of estimation results through two real-data analyses based on linear regression. The findings of this paper make us ponder on the use of improper heavy-tailed data models in Bayesian analyses, an approach which is seen to fit within the generalized Bayesian framework of Bissiri et al. (2016) when combined with proper prior distributions yielding proper (generalized) posterior distributions.
- [8] arXiv:2408.10502 [pdf, html, other]
-
Title: Asymptotic Classification Error for Heavy-Tailed Renewal ProcessesComments: 11 pages, 2 figuresSubjects: Machine Learning (stat.ML); Machine Learning (cs.LG); Statistics Theory (math.ST)
Despite the widespread occurrence of classification problems and the increasing collection of point process data across many disciplines, study of error probability for point process classification only emerged very recently. Here, we consider classification of renewal processes. We obtain asymptotic expressions for the Bhattacharyya bound on misclassification error probabilities for heavy-tailed renewal processes.
- [9] arXiv:2408.10542 [pdf, html, other]
-
Title: High-Dimensional Covariate-Augmented Overdispersed Multi-Study Poisson Factor ModelSubjects: Methodology (stat.ME)
Factor analysis for high-dimensional data is a canonical problem in statistics and has a wide range of applications. However, there is currently no factor model tailored to effectively analyze high-dimensional count responses with corresponding covariates across multiple studies, such as the single-cell sequencing dataset from a case-control study. In this paper, we introduce factor models designed to jointly analyze multiple studies by extracting study-shared and specified factors. Our factor models account for heterogeneous noises and overdispersion among counts with augmented covariates. We propose an efficient and speedy variational estimation procedure for estimating model parameters, along with a novel criterion for selecting the optimal number of factors and the rank of regression coefficient matrix. The consistency and asymptotic normality of estimators are systematically investigated by connecting variational likelihood and profile M-estimation. Extensive simulations and an analysis of a single-cell sequencing dataset are conducted to demonstrate the effectiveness of the proposed multi-study Poisson factor model.
- [10] arXiv:2408.10558 [pdf, html, other]
-
Title: Multi-Attribute Preferences: A Transfer Learning ApproachSubjects: Methodology (stat.ME)
This contribution introduces a novel statistical learning methodology based on the Bradley-Terry method for pairwise comparisons, where the novelty arises from the method's capacity to estimate the worth of objects for a primary attribute by incorporating data of secondary attributes. These attributes are properties on which objects are evaluated in a pairwise fashion by individuals. By assuming that the main interest of practitioners lies in the primary attribute, and the secondary attributes only serve to improve estimation of the parameters underlying the primary attribute, this paper utilises the well-known transfer learning framework. To wit, the proposed method first estimates a biased worth vector using data pertaining to both the primary attribute and the set of informative secondary attributes, which is followed by a debiasing step based on a penalised likelihood of the primary attribute. When the set of informative secondary attributes is unknown, we allow for their estimation by a data-driven algorithm. Theoretically, we show that, under mild conditions, the $\ell_\infty$ and $\ell_2$ rates are improved compared to fitting a Bradley-Terry model on just the data pertaining to the primary attribute. The favourable (comparative) performance under more general settings is shown by means of a simulation study. To illustrate the usage and interpretation of the method, an application of the proposed method is provided on consumer preference data pertaining to a cassava derived food product: eba. An R package containing the proposed methodology can be found on xhttps://CRAN.this http URL.
- [11] arXiv:2408.10570 [pdf, html, other]
-
Title: A two-sample test based on averaged Wilcoxon rank sums over interpoint distancesSubjects: Methodology (stat.ME)
An important class of two-sample multivariate homogeneity tests is based on identifying differences between the distributions of interpoint distances. While generating distances from point clouds offers a straightforward and intuitive way for dimensionality reduction, it also introduces dependencies to the resulting distance samples. We propose a simple test based on Wilcoxon's rank sum statistic for which we prove asymptotic normality under the null hypothesis and fixed alternatives under mild conditions on the underlying distributions of the point clouds. Furthermore, we show consistency of the test and derive a variance approximation that allows to construct a computationally feasible, distribution-free test with good finite sample performance. The power and robustness of the test for high-dimensional data and low sample sizes is demonstrated by numerical simulations. Finally, we apply the proposed test to case-control testing on microarray data in genetic studies, which is considered a notorious case for a high number of variables and low sample sizes.
- [12] arXiv:2408.10612 [pdf, html, other]
-
Title: Extension of the one-sample Kolmogorov-Smirnov testComments: 18 pages, 3 figuresSubjects: Statistics Theory (math.ST)
We propose here a new goodness-of-fit test, named the one-sample OVL-q test (q = 1, 2, . . .), which can be considered an extension of the one-sample Kolmogorov-Smirnov test (equivalent to the one-sample OVL-1 test). We have analyzed the asymptotic properties of the one-sample OVL-2 test statistic and enabled the calculation of asymptotic p-values for the test statistic. We further conducted numerical experiments and demonstrated that the one-sample OVL-2 test can sometimes exceed the detection power of conventional goodness-of-fit tests.
- [13] arXiv:2408.10650 [pdf, html, other]
-
Title: Principal component analysis for max-stable distributionsComments: 34 pages, 11 FiguresSubjects: Methodology (stat.ME)
Principal component analysis (PCA) is one of the most popular dimension reduction techniques in statistics and is especially powerful when a multivariate distribution is concentrated near a lower-dimensional subspace. Multivariate extreme value distributions have turned out to provide challenges for the application of PCA since their constraint support impedes the detection of lower-dimensional structures and heavy-tails can imply that second moments do not exist, thereby preventing the application of classical variance-based techniques for PCA. We adapt PCA to max-stable distributions using a regression setting and employ max-linear maps to project the random vector to a lower-dimensional space while preserving max-stability. We also provide a characterization of those distributions which allow for a perfect reconstruction from the lower-dimensional representation. Finally, we demonstrate how an optimal projection matrix can be consistently estimated and show viability in practice with a simulation study and application to a benchmark dataset.
- [14] arXiv:2408.10856 [pdf, html, other]
-
Title: Conditional Delta-Method for Resampling Empirical Processes in Multiple Sample ProblemsSubjects: Statistics Theory (math.ST)
The functional delta-method has a wide range of applications in statistics. Applications on functionals of empirical processes yield various limit results for classical statistics. To improve the finite sample properties of statistical inference procedures that are based on the limit results, resampling procedures such as random permutation and bootstrap methods are a popular solution. In order to analyze the behaviour of the functionals of the resampling empirical processes, corresponding conditional functional delta-methods are desirable. While conditional functional delta-methods for some special cases already exist, there is a lack of more general conditional functional delta-methods for resampling procedures for empirical processes, such as the permutation and pooled bootstrap method. This gap is addressed in the present paper. Thereby, a general multiple sample problem is considered. The flexible application of the developed conditional delta-method is shown in various relevant examples.
- [15] arXiv:2408.10867 [pdf, html, other]
-
Title: Bye-Bye, Bye Advantage: Estimating the competitive impact of rest differential in the National Football LeagueComments: 10 figures, 4 tablesSubjects: Applications (stat.AP)
The National Football League (NFL) sets its regular season schedule to optimize viewership and minimize competitive inequities. One inequity assumed to impact team performance is rest differential, defined as the relative number of days between games. Using Bayesian state space models on both game outcomes and betting market data, we estimate the competitive effect of rest differential in American football. We find that the most commonly referred to inequities -- both the bye week rest advantage and the mini-bye week rest advantage -- currently show no significant evidence of providing the rested team a competitive edge. Further, we trace a decline in the advantage of a bye week to a 2011 change to the NFL's Collective Bargaining Agreement, which represents a natural experiment to test the relevance of rest and preparation in football. Prior to the agreement, NFL teams off a bye week received a significant advantage (+2.2 points per game), but since 2011, that benefit has been mitigated.
- [16] arXiv:2408.10876 [pdf, html, other]
-
Title: More Options for Prelabor Rupture of Membranes, A Bayesian AnalysisComments: To appear in the 2024 IEEE 11th International Conference on Data Science and Advanced Analytics (DSAA)Subjects: Applications (stat.AP); Machine Learning (cs.LG)
An obstetric goal for a laboring mother is to achieve a vaginal delivery as it reduces the risks inherent in major abdominal surgery (i.e., a Cesarean section). Various medical interventions may be used by a physician to increase the likelihood of this occurring while minimizing maternal and fetal morbidity. However, patients with prelabor rupture of membranes (PROM) have only two commonly used options for cervical ripening, Pitocin and misoprostol. Little research exists on the benefits/risks for these two key drugs for PROM patients. A major limitation with most induction-of-labor related research is the inability to account for differences in \textit{Bishop scores} that are commonly used in obstetrical practice to determine the next induction agent offered to the patient. This creates a confounding factor, which biases the results, but has not been realized in the literature. In this work, we use a Bayesian model of the relationships between the relevant factors, informed by expert physicians, to separate the confounding variable from its actual impact. In doing so, we provide strong evidence that pitocin and buccal misoprostol are equally effective and safe; thus, physicians have more choice in clinical care than previously realized. This is particularly important for developing countries where neither medication may be readily available, and prior guidelines may create an artificial barrier to needed medication.
- [17] arXiv:2408.10915 [pdf, html, other]
-
Title: Neural Networks for Parameter Estimation in Geometrically Anisotropic Geostatistical ModelsSubjects: Methodology (stat.ME)
This article presents a neural network approach for estimating the covariance function of spatial Gaussian random fields defined in a portion of the Euclidean plane. Our proposal builds upon recent contributions, expanding from the purely isotropic setting to encompass geometrically anisotropic correlation structures, i.e., random fields with correlation ranges that vary across different directions. We conduct experiments with both simulated and real data to assess the performance of the methodology and to provide guidelines to practitioners.
- [18] arXiv:2408.10976 [pdf, other]
-
Title: Kernel-Based Differentiable Learning of Non-Parametric Directed Acyclic Graphical ModelsComments: To be published in the Proceedings of Probabilistic Graphical Models (PGM) 2024Subjects: Machine Learning (stat.ML); Machine Learning (cs.LG)
Causal discovery amounts to learning a directed acyclic graph (DAG) that encodes a causal model. This model selection problem can be challenging due to its large combinatorial search space, particularly when dealing with non-parametric causal models. Recent research has sought to bypass the combinatorial search by reformulating causal discovery as a continuous optimization problem, employing constraints that ensure the acyclicity of the graph. In non-parametric settings, existing approaches typically rely on finite-dimensional approximations of the relationships between nodes, resulting in a score-based continuous optimization problem with a smooth acyclicity constraint. In this work, we develop an alternative approximation method by utilizing reproducing kernel Hilbert spaces (RKHS) and applying general sparsity-inducing regularization terms based on partial derivatives. Within this framework, we introduce an extended RKHS representer theorem. To enforce acyclicity, we advocate the log-determinant formulation of the acyclicity constraint and show its stability. Finally, we assess the performance of our proposed RKHS-DAGMA procedure through simulations and illustrative data analyses.
- [19] arXiv:2408.10996 [pdf, html, other]
-
Title: Approximation Rates for Shallow ReLU$^k$ Neural Networks on Sobolev Spaces via the Radon TransformSubjects: Machine Learning (stat.ML); Machine Learning (cs.LG); Numerical Analysis (math.NA)
Let $\Omega\subset \mathbb{R}^d$ be a bounded domain. We consider the problem of how efficiently shallow neural networks with the ReLU$^k$ activation function can approximate functions from Sobolev spaces $W^s(L_p(\Omega))$ with error measured in the $L_q(\Omega)$-norm. Utilizing the Radon transform and recent results from discrepancy theory, we provide a simple proof of nearly optimal approximation rates in a variety of cases, including when $q\leq p$, $p\geq 2$, and $s \leq k + (d+1)/2$. The rates we derive are optimal up to logarithmic factors, and significantly generalize existing results. An interesting consequence is that the adaptivity of shallow ReLU$^k$ neural networks enables them to obtain optimal approximation rates for smoothness up to order $s = k + (d+1)/2$, even though they represent piecewise polynomials of fixed degree $k$.
- [20] arXiv:2408.11003 [pdf, html, other]
-
Title: DEEPEAST technique to enhance power in two-sample tests via the same-attraction functionSubjects: Methodology (stat.ME)
Data depth has emerged as an invaluable nonparametric measure for the ranking of multivariate samples. The main contribution of depth-based two-sample comparisons is the introduction of the Q statistic (Liu and Singh, 1993), a quality index. Unlike traditional methods, data depth does not require the assumption of normal distributions and adheres to four fundamental properties. Many existing two-sample homogeneity tests, which assess mean and/or scale changes in distributions often suffer from low statistical power or indeterminate asymptotic distributions. To overcome these challenges, we introduced a DEEPEAST (depth-explored same-attraction sample-to-sample central-outward ranking) technique for improving statistical power in two-sample tests via the same-attraction function. We proposed two novel and powerful depth-based test statistics: the sum test statistic and the product test statistic, which are rooted in Q statistics, share a "common attractor" and are applicable across all depth functions. We further proved the asymptotic distribution of these statistics for various depth functions. To assess the performance of power gain, we apply three depth functions: Mahalanobis depth (Liu and Singh, 1993), Spatial depth (Brown, 1958; Gower, 1974), and Projection depth (Liu, 1992). Through two-sample simulations, we have demonstrated that our sum and product statistics exhibit superior power performance, utilizing a strategic block permutation algorithm and compare favourably with popular methods in literature. Our tests are further validated through analysis on Raman spectral data, acquired from cellular and tissue samples, highlighting the effectiveness of the proposed tests highlighting the effective discrimination between health and cancerous samples.
- [21] arXiv:2408.11012 [pdf, html, other]
-
Title: Discriminant Analysis in stationary time series based on robust cepstral coefficientsComments: This paper is the result of thesis defens presented to ICEX in UFMG contanning 39 pages and 17 figuresSubjects: Methodology (stat.ME)
Time series analysis is crucial in fields like finance, economics, environmental science, and biomedical engineering, aiding in forecasting, pattern identification, and understanding underlying mechanisms. While traditional time-domain methods focus on trends and seasonality, they often miss periodicities better captured in the frequency domain. Analyzing time series in the frequency domain uncovers spectral properties, offering deeper insights into underlying processes, aiding in differentiating data-generating processes of various populations, and assisting in classification. Common approaches use smoothed estimators, such as the smoothed periodogram, to minimize bias by averaging spectra from individual replicates within a population. However, these methods struggle with spectral variability among replicates, and abrupt values can skew estimators, complicating discrimination and classification. There's a gap in the literature for methods that account for within-population spectral variability, separate white noise effects from autocorrelations, and employ robust estimators in the presence of outliers. This paper fills that gap by introducing a robust framework for classifying time series groups. The process involves transforming time series into the frequency domain using the Fourier Transform, computing the power spectrum, and using the inverse Fourier Transform to obtain the cepstrum. To enhance spectral estimates' robustness and consistency, we apply the multitaper periodogram and the M-periodogram. These features are then used in Linear Discriminant Analysis (LDA) to improve classification accuracy and interpretability, offering a powerful tool for precise temporal pattern distinction and resilience to data anomalies.
New submissions for Wednesday, 21 August 2024 (showing 21 of 21 entries )
- [22] arXiv:2408.10147 (cross-list from cs.LG) [pdf, html, other]
-
Title: In-Context Learning with Representations: Contextual Generalization of Trained TransformersSubjects: Machine Learning (cs.LG); Computation and Language (cs.CL); Information Theory (cs.IT); Optimization and Control (math.OC); Machine Learning (stat.ML)
In-context learning (ICL) refers to a remarkable capability of pretrained large language models, which can learn a new task given a few examples during inference. However, theoretical understanding of ICL is largely under-explored, particularly whether transformers can be trained to generalize to unseen examples in a prompt, which will require the model to acquire contextual knowledge of the prompt for generalization. This paper investigates the training dynamics of transformers by gradient descent through the lens of non-linear regression tasks. The contextual generalization here can be attained via learning the template function for each task in-context, where all template functions lie in a linear space with $m$ basis functions. We analyze the training dynamics of one-layer multi-head transformers to in-contextly predict unlabeled inputs given partially labeled prompts, where the labels contain Gaussian noise and the number of examples in each prompt are not sufficient to determine the template. Under mild assumptions, we show that the training loss for a one-layer multi-head transformer converges linearly to a global minimum. Moreover, the transformer effectively learns to perform ridge regression over the basis functions. To our knowledge, this study is the first provable demonstration that transformers can learn contextual (i.e., template) information to generalize to both unseen examples and tasks when prompts contain only a small number of query-answer pairs.
- [23] arXiv:2408.10251 (cross-list from physics.data-an) [pdf, html, other]
-
Title: Impossible temperatures are not as rare as you thinkSubjects: Data Analysis, Statistics and Probability (physics.data-an); Atmospheric and Oceanic Physics (physics.ao-ph); Applications (stat.AP); Methodology (stat.ME)
The last decade has seen numerous record-shattering heatwaves in all corners of the globe. In the aftermath of these devastating events, there is interest in identifying worst-case thresholds or upper bounds that quantify just how hot temperatures can become. Generalized Extreme Value theory provides a data-driven estimate of extreme thresholds; however, upper bounds may be exceeded by future events, which undermines attribution and planning for heatwave impacts. Here, we show how the occurrence and relative probability of observed events that exceed a priori upper bound estimates, so-called "impossible" temperatures, has changed over time. We find that many unprecedented events are actually within data-driven upper bounds, but only when using modern spatial statistical methods. Furthermore, there are clear connections between anthropogenic forcing and the "impossibility" of the most extreme temperatures. Robust understanding of heatwave thresholds provides critical information about future record-breaking events and how their extremity relates to historical measurements.
- [24] arXiv:2408.10288 (cross-list from cs.LG) [pdf, html, other]
-
Title: Augmenting train maintenance technicians with automated incident diagnostic suggestionsSubjects: Machine Learning (cs.LG); Machine Learning (stat.ML)
Train operational incidents are so far diagnosed individually and manually by train maintenance technicians. In order to assist maintenance crews in their responsiveness and task prioritization, a learning machine is developed and deployed in production to suggest diagnostics to train technicians on their phones, tablets or laptops as soon as a train incident is declared. A feedback loop allows to take into account the actual diagnose by designated train maintenance experts to refine the learning machine. By formulating the problem as a discrete set classification task, feature engineering methods are proposed to extract physically plausible sets of events from traces generated on-board railway vehicles. The latter feed an original ensemble classifier to class incidents by their potential technical cause. Finally, the resulting model is trained and validated using real operational data and deployed on a cloud platform. Future work will explore how the extracted sets of events can be used to avoid incidents by assisting human experts in the creation predictive maintenance alerts.
- [25] arXiv:2408.10353 (cross-list from cs.LG) [pdf, html, other]
-
Title: On the Identifiability of Sparse ICA without Assuming Non-GaussianityComments: NeurIPS 2023Subjects: Machine Learning (cs.LG); Machine Learning (stat.ML)
Independent component analysis (ICA) is a fundamental statistical tool used to reveal hidden generative processes from observed data. However, traditional ICA approaches struggle with the rotational invariance inherent in Gaussian distributions, often necessitating the assumption of non-Gaussianity in the underlying sources. This may limit their applicability in broader contexts. To accommodate Gaussian sources, we develop an identifiability theory that relies on second-order statistics without imposing further preconditions on the distribution of sources, by introducing novel assumptions on the connective structure from sources to observed variables. Different from recent work that focuses on potentially restrictive connective structures, our proposed assumption of structural variability is both considerably less restrictive and provably necessary. Furthermore, we propose two estimation methods based on second-order statistics and sparsity constraint. Experimental results are provided to validate our identifiability theory and estimation methods.
- [26] arXiv:2408.10419 (cross-list from cs.LG) [pdf, html, other]
-
Title: Second-Order Forward-Mode Automatic Differentiation for OptimizationComments: 14 pages, 8 figuresSubjects: Machine Learning (cs.LG); Machine Learning (stat.ML)
This paper introduces a second-order hyperplane search, a novel optimization step that generalizes a second-order line search from a line to a $k$-dimensional hyperplane. This, combined with the forward-mode stochastic gradient method, yields a second-order optimization algorithm that consists of forward passes only, completely avoiding the storage overhead of backpropagation. Unlike recent work that relies on directional derivatives (or Jacobian--Vector Products, JVPs), we use hyper-dual numbers to jointly evaluate both directional derivatives and their second-order quadratic terms. As a result, we introduce forward-mode weight perturbation with Hessian information (FoMoH). We then use FoMoH to develop a novel generalization of line search by extending it to a hyperplane search. We illustrate the utility of this extension and how it might be used to overcome some of the recent challenges of optimizing machine learning models without backpropagation. Our code is open-sourced at this https URL.
- [27] arXiv:2408.10609 (cross-list from cs.LG) [pdf, html, other]
-
Title: PerturBench: Benchmarking Machine Learning Models for Cellular Perturbation AnalysisYan Wu, Esther Wershof, Sebastian M Schmon, Marcel Nassar, Błażej Osiński, Ridvan Eksi, Kun Zhang, Thore GraepelComments: 9 pages plus 19 pages supplementary material. Code is available at this https URLSubjects: Machine Learning (cs.LG); Genomics (q-bio.GN); Machine Learning (stat.ML)
We present a comprehensive framework for predicting the effects of perturbations in single cells, designed to standardize benchmarking in this rapidly evolving field. Our framework, PerturBench, includes a user-friendly platform, diverse datasets, metrics for fair model comparison, and detailed performance analysis. Extensive evaluations of published and baseline models reveal limitations like mode or posterior collapse, and underscore the importance of rank metrics that assess the ordering of perturbations alongside traditional measures like RMSE. Our findings show that simple models can outperform more complex approaches. This benchmarking exercise sets new standards for model evaluation, supports robust model development, and advances the potential of these models to use high-throughput and high-content genetic and chemical screens for disease target discovery.
- [28] arXiv:2408.10610 (cross-list from cs.LG) [pdf, html, other]
-
Title: On the Approximability of Stationary Processes using the ARMA ModelComments: 10 pages, 3 figuresSubjects: Machine Learning (cs.LG); Probability (math.PR); Methodology (stat.ME)
We identify certain gaps in the literature on the approximability of stationary random variables using the Autoregressive Moving Average (ARMA) model. To quantify approximability, we propose that an ARMA model be viewed as an approximation of a stationary random variable. We map these stationary random variables to Hardy space functions, and formulate a new function approximation problem that corresponds to random variable approximation, and thus to ARMA. Based on this Hardy space formulation we identify a class of stationary processes where approximation guarantees are feasible. We also identify an idealized stationary random process for which we conjecture that a good ARMA approximation is not possible. Next, we provide a constructive proof that Padé approximations do not always correspond to the best ARMA approximation. Finally, we note that the spectral methods adopted in this paper can be seen as a generalization of unit root methods for stationary processes even when an ARMA model is not defined.
- [29] arXiv:2408.10616 (cross-list from cs.CV) [pdf, other]
-
Title: A toolbox for calculating objective image properties in aesthetics researchChristoph Redies, Ralf Bartho, Lisa Koßmann, Branka Spehar, Ronald Hübner, Johan Wagemans, Gregor U. Hayn-LeichsenringComments: 41 pages, 6 figureSubjects: Computer Vision and Pattern Recognition (cs.CV); Applications (stat.AP)
Over the past two decades, researchers in the field of visual aesthetics have studied numerous quantitative (objective) image properties and how they relate to visual aesthetic appreciation. However, results are difficult to compare between research groups. One reason is that researchers use different sets of image properties in their studies. But even if the same properties are used, the image pre-processing techniques may differ and often researchers use their own customized scripts to calculate the image properties. To provide greater accessibility and comparability of research results in visual experimental aesthetics, we developed an open-access and easy-to-use toolbox (called the 'Aesthetics Toolbox'). The Toolbox allows users to calculate a well-defined set of quantitative image properties popular in contemporary research. The properties include lightness and color statistics, Fourier spectral properties, fractality, self-similarity, symmetry, as well as different entropy measures and CNN-based variances. Compatible with most devices, the Toolbox provides an intuitive click-and-drop web interface. In the Toolbox, we integrated the original scripts of four different research groups and translated them into Python 3. To ensure that results were consistent across analyses, we took care that results from the Python versions of the scripts were the same as those from the original scripts. The toolbox, detailed documentation, and a link to the cloud version are available via Github: this https URL. In summary, we developed a toolbox that helps to standardize and simplify the calculation of quantitative image properties for visual aesthetics research.
- [30] arXiv:2408.10762 (cross-list from cond-mat.soft) [pdf, html, other]
-
Title: Sparse Regression for Discovery of Constitutive Models from Oscillatory Shear MeasurementsComments: 45 pages, 8 figuresSubjects: Soft Condensed Matter (cond-mat.soft); Machine Learning (stat.ML)
We propose sparse regression as an alternative to neural networks for the discovery of parsimonious constitutive models (CMs) from oscillatory shear experiments. Symmetry and frame-invariance are strictly imposed by using tensor basis functions to isolate and describe unknown nonlinear terms in the CMs. We generate synthetic experimental data using the Giesekus and Phan-Thien Tanner CMs, and consider two different scenarios. In the complete information scenario, we assume that the shear stress, along with the first and second normal stress differences, is measured. This leads to a sparse linear regression problem that can be solved efficiently using $l_1$ regularization. In the partial information scenario, we assume that only shear stress data is available. This leads to a more challenging sparse nonlinear regression problem, for which we propose a greedy two-stage algorithm. In both scenarios, the proposed methods fit and interpolate the training data remarkably well. Predictions of the inferred CMs extrapolate satisfactorily beyond the range of training data for oscillatory shear. They also extrapolate reasonably well to flow conditions like startup of steady and uniaxial extension that are not used in the identification of CMs. We discuss ramifications for experimental design, potential algorithmic improvements, and implications of the non-uniqueness of CMs inferred from partial information.
- [31] arXiv:2408.10825 (cross-list from econ.EM) [pdf, other]
-
Title: Conditional nonparametric variable screening by neural factor regressionJianqing Fan (Princeton University), Weining Wang (University of Groningen), Yue Zhao (University of York)Comments: 80 pages, 14 figuresSubjects: Econometrics (econ.EM); Statistics Theory (math.ST)
High-dimensional covariates often admit linear factor structure. To effectively screen correlated covariates in high-dimension, we propose a conditional variable screening test based on non-parametric regression using neural networks due to their representation power. We ask the question whether individual covariates have additional contributions given the latent factors or more generally a set of variables. Our test statistics are based on the estimated partial derivative of the regression function of the candidate variable for screening and a observable proxy for the latent factors. Hence, our test reveals how much predictors contribute additionally to the non-parametric regression after accounting for the latent factors. Our derivative estimator is the convolution of a deep neural network regression estimator and a smoothing kernel. We demonstrate that when the neural network size diverges with the sample size, unlike estimating the regression function itself, it is necessary to smooth the partial derivative of the neural network estimator to recover the desired convergence rate for the derivative. Moreover, our screening test achieves asymptotic normality under the null after finely centering our test statistics that makes the biases negligible, as well as consistency for local alternatives under mild conditions. We demonstrate the performance of our test in a simulation study and two real world applications.
- [32] arXiv:2408.10862 (cross-list from cs.LG) [pdf, html, other]
-
Title: Feature Selection from Differentially Private CorrelationsComments: To appear in Proceedings of the 17th ACM Workshop on Artificial Intelligence and Security, 2024Subjects: Machine Learning (cs.LG); Machine Learning (stat.ML)
Data scientists often seek to identify the most important features in high-dimensional datasets. This can be done through $L_1$-regularized regression, but this can become inefficient for very high-dimensional datasets. Additionally, high-dimensional regression can leak information about individual datapoints in a dataset. In this paper, we empirically evaluate the established baseline method for feature selection with differential privacy, the two-stage selection technique, and show that it is not stable under sparsity. This makes it perform poorly on real-world datasets, so we consider a different approach to private feature selection. We employ a correlations-based order statistic to choose important features from a dataset and privatize them to ensure that the results do not leak information about individual datapoints. We find that our method significantly outperforms the established baseline for private feature selection on many datasets.
- [33] arXiv:2408.10939 (cross-list from cs.LG) [pdf, html, other]
-
Title: Conformalized Interval Arithmetic with Symmetric CalibrationSubjects: Machine Learning (cs.LG); Machine Learning (stat.ML)
Uncertainty quantification is essential in decision-making, especially when joint distributions of random variables are involved. While conformal prediction provides distribution-free prediction sets with valid coverage guarantees, it traditionally focuses on single predictions. This paper introduces novel conformal prediction methods for estimating the sum or average of unknown labels over specific index sets. We develop conformal prediction intervals for single target to the prediction interval for sum of multiple targets. Under permutation invariant assumptions, we prove the validity of our proposed method. We also apply our algorithms on class average estimation and path cost prediction tasks, and we show that our method outperforms existing conformalized approaches as well as non-conformal approaches.
Cross submissions for Wednesday, 21 August 2024 (showing 12 of 12 entries )
- [34] arXiv:2110.14992 (replaced) [pdf, html, other]
-
Title: Algebraic algorithm for direct sampling from toric models and hypergeometric functionsComments: 40 pages, 2 figuresSubjects: Statistics Theory (math.ST); Classical Analysis and ODEs (math.CA)
We show that Pfaffians or contiguity relations of hypergeometric functions of several variables give a direct sampling algorithm from toric models in statistics, which is a Markov chain on a lattice generated by a matrix $A$. A correspondence among graphical models and $A$-hypergeometric system is discussed and we give a sum formula of special values of $A$-hypergeometric polynomials. Some hypergeometric series which are interesting in view of statistics are presented.
- [35] arXiv:2206.00794 (replaced) [pdf, html, other]
-
Title: Sequential Bayesian Neural Subnetwork EnsemblesSanket Jantre, Shrijita Bhattacharya, Nathan M. Urban, Byung-Jun Yoon, Tapabrata Maiti, Prasanna Balaprakash, Sandeep MadireddySubjects: Machine Learning (stat.ML); Machine Learning (cs.LG); Statistics Theory (math.ST)
Deep ensembles have emerged as a powerful technique for improving predictive performance and enhancing model robustness across various applications by leveraging model diversity. However, traditional deep ensemble methods are often computationally expensive and rely on deterministic models, which may limit their flexibility. Additionally, while sparse subnetworks of dense models have shown promise in matching the performance of their dense counterparts and even enhancing robustness, existing methods for inducing sparsity typically incur training costs comparable to those of training a single dense model, as they either gradually prune the network during training or apply thresholding post-training. In light of these challenges, we propose an approach for sequential ensembling of dynamic Bayesian neural subnetworks that consistently maintains reduced model complexity throughout the training process while generating diverse ensembles in a single forward pass. Our approach involves an initial exploration phase to identify high-performing regions within the parameter space, followed by multiple exploitation phases that take advantage of the compactness of the sparse model. These exploitation phases quickly converge to different minima in the energy landscape, corresponding to high-performing subnetworks that together form a diverse and robust ensemble. We empirically demonstrate that our proposed approach outperforms traditional dense and sparse deterministic and Bayesian ensemble models in terms of prediction accuracy, uncertainty estimation, out-of-distribution detection, and adversarial robustness.
- [36] arXiv:2208.05949 (replaced) [pdf, html, other]
-
Title: Valid Inference After Causal DiscoverySubjects: Methodology (stat.ME); Machine Learning (cs.LG); Machine Learning (stat.ML)
Causal discovery and causal effect estimation are two fundamental tasks in causal inference. While many methods have been developed for each task individually, statistical challenges arise when applying these methods jointly: estimating causal effects after running causal discovery algorithms on the same data leads to "double dipping," invalidating the coverage guarantees of classical confidence intervals. To this end, we develop tools for valid post-causal-discovery inference. Across empirical studies, we show that a naive combination of causal discovery and subsequent inference algorithms leads to highly inflated miscoverage rates; on the other hand, applying our method provides reliable coverage while achieving more accurate causal discovery than data splitting.
- [37] arXiv:2208.08693 (replaced) [pdf, html, other]
-
Title: Matrix Quantile Factor ModelSubjects: Methodology (stat.ME); Econometrics (econ.EM)
This paper introduces a matrix quantile factor model for matrix-valued data with low-rank structure. We estimate the row and column factor spaces via minimizing the empirical check loss function with orthogonal rotation constraints. We show that the estimates converge at rate $(\min\{p_1p_2,p_2T,p_1T\})^{-1/2}$ in the average Frobenius norm, where $p_1$, $p_2$ and $T$ are the row dimensionality, column dimensionality and length of the matrix sequence, respectively. This rate is faster than that of the quantile estimates via ``flattening" the matrix model into a large vector model. To derive the central limit theorem, we introduce a novel augmented Lagrangian function, which is equivalent to the original constrained empirical check loss minimization problem. Via the equivalence, we prove that the Hessian matrix of the augmented Lagrangian function is locally positive definite, resulting in a locally convex penalized loss function around the true factors and their loadings. This easily leads to a feasible second-order expansion of the score function and readily established central limit theorems of the smoothed estimates of the loadings. We provide three consistent criteria to determine the pair of row and column factor numbers. Extensive simulation studies and an empirical study justify our theory.
- [38] arXiv:2210.02247 (replaced) [pdf, html, other]
-
Title: Modelling tree survival for investigating climate change effectsNicole N. Augustin, Axel Albrecht, Karim Anaya-Izquierdo, Alice Davis, Stefan Meining, Heike Puhlmann, Simon N. WoodComments: revised versionSubjects: Applications (stat.AP); Computation (stat.CO)
Using German forest health monitoring data we investigate the main drivers leading to tree mortality and the association between defoliation and mortality; in particular (a) whether defoliation is a proxy for other covariates (climate, soil, water budget); (b) whether defoliation is a tree response that mitigates the effects of climate change and (c) whether there is a threshold of defoliation which could be used as an early warning sign for irreversible damage.
Results show that environmental drivers leading to tree mortality differ by species, but some are always required in the model. The defoliation effect on mortality differs by species but it is always strong and monotonic. There is some evidence that a defoliation threshold exists for spruce, fir and beech.
We model tree survival with a smooth additive Cox model allowing for random effects taking care of dependence between neighbouring trees and non-linear functions of spatial time varying and functional predictors on defoliation, climate, soil and hydrology characteristics. Due to the large sample size and large number of parameters, we use parallel computing combined with marginal discretization of covariates. We propose a 'boost forward penalise backward' model selection scheme based on combining component-wise gradient boosting with integrated backward selection. - [39] arXiv:2212.08867 (replaced) [pdf, html, other]
-
Title: Specification tests for normal/gamma and stable/gamma stochastic frontier models based on empirical transformsSubjects: Statistics Theory (math.ST)
Goodness--of--fit tests for the distribution of the composed error term in a Stochastic Frontier Model (SFM) are suggested. The focus is on the case of a normal/gamma SFM and the heavy--tailed stable/gamma SFM. In the first case the moment generating function is used as tool while in the latter case the characteristic function of the error term is employed. In both cases our test statistics are formulated as weighted integrals of properly standardized data. The new normal/gamma test is consistent, and is shown to have an intrinsic relation to moment--based tests. The finite--sample behavior of resampling versions of both tests is investigated by Monte Carlo simulation, while several real--data applications are also included.
- [40] arXiv:2305.03205 (replaced) [pdf, html, other]
-
Title: Risk management in the use of published statistical results for policy decisionsComments: 29 pages, 1 figure; updated version has edits and several additions to originalSubjects: Other Statistics (stat.OT); Econometrics (econ.EM)
Statistical inferential results generally come with a measure of reliability for decision-making purposes. For a policy implementer, the value of implementing published policy research depends critically upon this reliability. For a policy researcher, the value of policy implementation may depend weakly or not at all upon the policy's outcome. Some researchers might benefit from overstating the reliability of statistical results. Implementers may find it difficult or impossible to determine whether researchers are overstating reliability. This information asymmetry between researchers and implementers can lead to an adverse selection problem where, at best, the full benefits of a policy are not realized or, at worst, a policy is deemed too risky to implement at any scale. Researchers can remedy this by guaranteeing the policy outcome. Researchers can overcome their own risk aversion and wealth constraints by exchanging risks with other researchers or offering only partial insurance. The problem and remedy are illustrated using a confidence interval for the success probability of a binomial policy outcome.
- [41] arXiv:2307.02188 (replaced) [pdf, html, other]
-
Title: Improving Algorithms for Fantasy BasketballComments: 19 pagesSubjects: Methodology (stat.ME)
Fantasy basketball has a rich underlying mathematical structure which makes optimal drafting strategy unclear. A central issue for category leagues is how to aggregate a player's statistics from all categories into a single number representing general value. It is shown that under a simplified model of fantasy basketball, a novel metric dubbed the "G-score" is appropriate for this purpose. The traditional metric used by analysts, "Z-score", is a special case of the G-score under the condition that future player performances are known exactly. The distinction between Z-score and G-score is particularly meaningful for head-to-head formats, because there is a large degree of uncertainty in player performance from one week to another. Simulated fantasy basketball seasons with head-to-head scoring provide evidence that G-scores do in fact outperform Z-scores in that context.
- [42] arXiv:2308.07248 (replaced) [pdf, other]
-
Title: Maintaining the validity of inference from linear mixed models in stepped-wedge cluster randomized trials under misspecified random-effects structuresComments: update a typo in abstractSubjects: Methodology (stat.ME); Applications (stat.AP)
Linear mixed models are commonly used in analyzing stepped-wedge cluster randomized trials (SW-CRTs). A key consideration for analyzing a SW-CRT is accounting for the potentially complex correlation structure, which can be achieved by specifying a random effects structure. Common random effects structures for a SW-CRT include random intercept, random cluster-by-period, and discrete-time decay. Recently, more complex structures, such as the random intervention structure, have been proposed. In practice, specifying appropriate random effects can be challenging. Robust variance estimators (RVE) may be applied to linear mixed models to provide consistent estimators of standard errors of fixed effect parameters in the presence of random-effects misspecification. However, there has been no empirical investigation of RVE for SW-CRT. In this paper, we first review five RVEs (both standard and small-sample bias-corrected RVEs) that are available for linear mixed models. We then describe a comprehensive simulation study to examine the performance of these RVEs for SW-CRTs with a continuous outcome under different data generators. For each data generator, we investigate whether the use of a RVE with either the random intercept model or the random cluster-by-period model is sufficient to provide valid statistical inference for fixed effect parameters, when these working models are subject to misspecification. Our results indicate that the random intercept and random cluster-by-period models with RVEs performed similarly. The CR3 RVE estimator, coupled with the number of clusters minus two degrees of freedom correction, consistently gave the best coverage results, but could be slightly conservative when the number of clusters was below 16. We summarize the implications of our results for linear mixed model analysis of SW-CRTs in practice.
- [43] arXiv:2308.16650 (replaced) [pdf, html, other]
-
Title: Optimal confidence interval for the difference of proportionsSubjects: Computation (stat.CO)
Estimating the probability of the binomial distribution is a basic problem, which appears in almost all introductory statistics courses and is performed frequently in various studies. In some cases, the parameter of interest is a difference between two probabilities, and the current work studies the construction of confidence intervals for this parameter when the sample size is small. Our goal is to find the shortest confidence intervals under the constraint of coverage probability being at least as large as a predetermined level. For the two-sample case, there is no known algorithm that achieves this goal, but different heuristics procedures have been suggested, and the present work aims at finding optimal confidence intervals. In the one-sample case, there is a known algorithm that finds optimal confidence intervals presented by Blyth and Still (1983). It is based on solving small and local optimization problems and then using an inversion step to find the global optimum solution. We show that this approach fails in the two-sample case and therefore, in order to find optimal confidence intervals, one needs to solve a global optimization problem, rather than small and local ones, which is computationally much harder. We present and discuss the suitable global optimization problem. Using the Gurobi package we find near-optimal solutions when the sample sizes are smaller than 15, and we compare these solutions to some existing methods, both approximate and exact. We find that the improvement in terms of lengths with respect to the best competitor varies between 1.5\% and 5\% for different parameters of the problem. Therefore, we recommend the use of the new confidence intervals when both sample sizes are smaller than 15. Tables of the confidence intervals are given in the Excel file in this link.
- [44] arXiv:2310.02273 (replaced) [pdf, html, other]
-
Title: A New measure of income inequalitySubjects: Methodology (stat.ME); Statistics Theory (math.ST)
A new measure of income inequality that captures the heavy tail behavior of the income distribution is proposed. We discuss two different approaches to find the estimators of the proposed measure. We show that these estimators are consistent and have an asymptotically normal distribution. We also obtain a jackknife empirical likelihood (JEL) confidence interval of the income inequality measure. A Monte Carlo simulation study is conducted to evaluate the finite sample properties of the estimators and JEL-based confidence inerval. Finally, we use our measure to study the income inequality of three states in India.
- [45] arXiv:2310.11683 (replaced) [pdf, html, other]
-
Title: Treatment bootstrapping: A new approach to quantify uncertainty of average treatment effect estimatesSubjects: Methodology (stat.ME); Applications (stat.AP)
This paper proposes a new non-parametric bootstrap method to quantify the uncertainty of average treatment effect estimate for the treated from matching estimators. More specifically, it seeks to quantify the uncertainty associated with the average treatment effect estimate for the treated by bootstrapping the treatment group only and finding the counterpart control group by pair matching on estimated propensity score without replacement. We demonstrate the validity of this approach and compare it with existing bootstrap approaches through Monte Carlo simulation and analysis of a real world data set. The results indicate that the proposed approach constructs confidence intervals and standard errors that have 95 percent or above coverage rate and better precision compared with existing bootstrap approaches, while these measures also depend on percent treated in the sample data and the sample size.
- [46] arXiv:2402.18697 (replaced) [pdf, html, other]
-
Title: Inferring Dynamic Networks from Marginals with Iterative Proportional FittingComments: Conference version available from https://proceedings.mlr.press/v235/chang24b.htmlJournal-ref: Proceedings of the 41st International Conference on Machine Learning, PMLR 235:6202-6252, 2024Subjects: Machine Learning (stat.ML); Machine Learning (cs.LG); Social and Information Networks (cs.SI); Optimization and Control (math.OC); Statistics Theory (math.ST)
A common network inference problem, arising from real-world data constraints, is how to infer a dynamic network from its time-aggregated adjacency matrix and time-varying marginals (i.e., row and column sums). Prior approaches to this problem have repurposed the classic iterative proportional fitting (IPF) procedure, also known as Sinkhorn's algorithm, with promising empirical results. However, the statistical foundation for using IPF has not been well understood: under what settings does IPF provide principled estimation of a dynamic network from its marginals, and how well does it estimate the network? In this work, we establish such a setting, by identifying a generative network model whose maximum likelihood estimates are recovered by IPF. Our model both reveals implicit assumptions on the use of IPF in such settings and enables new analyses, such as structure-dependent error bounds on IPF's parameter estimates. When IPF fails to converge on sparse network data, we introduce a principled algorithm that guarantees IPF converges under minimal changes to the network structure. Finally, we conduct experiments with synthetic and real-world data, which demonstrate the practical value of our theoretical and algorithmic contributions.
- [47] arXiv:2403.10136 (replaced) [pdf, other]
-
Title: Response Style Characterization for Repeated Measures Using the Visual Analogue ScaleComments: Accepted to IEEE Access. Accessible at this https URLSubjects: Methodology (stat.ME); Artificial Intelligence (cs.AI)
Self-report measures (e.g., Likert scales) are widely used to evaluate subjective health perceptions. Recently, the visual analog scale (VAS), a slider-based scale, has become popular owing to its ability to precisely and easily assess how people feel. These data can be influenced by the response style (RS), a user-dependent systematic tendency that occurs regardless of questionnaire instructions. Despite its importance, especially in between-individual analysis, little attention has been paid to handling the RS in the VAS (denoted as response profile (RP)), as it is mainly used for within-individual monitoring and is less affected by RP. However, VAS measurements often require repeated self-reports of the same questionnaire items, making it difficult to apply conventional methods on a Likert scale. In this study, we developed a novel RP characterization method for various types of repeatedly measured VAS data. This approach involves the modeling of RP as distributional parameters ${\theta}$ through a mixture of RS-like distributions, and addressing the issue of unbalanced data through bootstrap sampling for treating repeated measures. We assessed the effectiveness of the proposed method using simulated pseudo-data and an actual dataset from an empirical study. The assessment of parameter recovery showed that our method accurately estimated the RP parameter ${\theta}$, demonstrating its robustness. Moreover, applying our method to an actual VAS dataset revealed the presence of individual RP heterogeneity, even in repeated VAS measurements, similar to the findings of the Likert scale. Our proposed method enables RP heterogeneity-aware VAS data analysis, similar to Likert-scale data analysis.
- [48] arXiv:2405.05638 (replaced) [pdf, html, other]
-
Title: A Correlation-induced Finite Difference EstimatorSubjects: Methodology (stat.ME); Machine Learning (cs.LG); Numerical Analysis (math.NA); Optimization and Control (math.OC)
Finite difference (FD) approximation is a classic approach to stochastic gradient estimation when only noisy function realizations are available. In this paper, we first provide a sample-driven method via the bootstrap technique to estimate the optimal perturbation, and then propose an efficient FD estimator based on correlated samples at the estimated optimal perturbation. Furthermore, theoretical analyses of both the perturbation estimator and the FD estimator reveal that, {\it surprisingly}, the correlation enables the proposed FD estimator to achieve a reduction in variance and, in some cases, a decrease in bias compared to the traditional optimal FD estimator. Numerical results confirm the efficiency of our estimators and align well with the theory presented, especially in scenarios with small sample sizes. Finally, we apply the estimator to solve derivative-free optimization (DFO) problems, and numerical studies show that DFO problems with 100 dimensions can be effectively solved.
- [49] arXiv:2405.13073 (replaced) [pdf, other]
-
Title: A graph-structured distance for mixed-variable domains with meta variablesComments: 25 pages (without references), 5 figures, data and scripts available at this https URLSubjects: Machine Learning (stat.ML); Machine Learning (cs.LG)
Heterogeneous datasets emerge in various machine learning and optimization applications that feature different input sources, types or formats. Most models or methods do not natively tackle heterogeneity. Hence, such datasets are often partitioned into smaller and simpler ones, which may limit the generalizability or performance, especially if data is limited. The first main contribution of this work is a modeling framework that generalizes hierarchical, tree-structured, variable-size or conditional search frameworks. The framework models mixed-variable domains in which variables may be continuous, integer, or categorical, with some identified as meta when they influence the structure of the problem. The second main contribution is a novel distance that compares any pair of mixed-variable points that do not share the same variables, allowing to use whole heterogeneous datasets that reside in mixed-variable domains with meta variables. The contributions are illustrated on several regression experiments, in which the performance of a multilayer perceptron with respect to its hyperparameters is modeled.
- [50] arXiv:2405.18020 (replaced) [pdf, other]
-
Title: The short-term association between environmental variables and mortality: evidence from EuropeSubjects: Applications (stat.AP)
Using fine-grained, publicly available data, this paper studies the short-term association between environmental factors, i.e., weather and air pollution characteristics, and weekly mortality rates in small geographical regions in Europe. Hereto, we develop a mortality modeling framework where a baseline model describes a region-specific, seasonal trend observed within the historical weekly mortality rates. Using a machine learning algorithm, we then explain deviations from this baseline using features constructed from environmental data to identify anomalies and extreme events. We illustrate our proposed modeling framework through a case study on more than 550 NUTS 3 regions (Nomenclature of Territorial Units for Statistics, level 3) in 20 European countries. Using interpretation tools, we unravel insights into which environmental features are most important when estimating excess or deficit mortality relative to the baseline and explore how these features interact. Moreover, we investigate harvesting effects through our constructed weekly mortality modelling framework. Our findings show that temperature-related features are most influential in explaining mortality deviations from the baseline over short time periods. Furthermore, we find that environmental features prove particularly beneficial in southern regions for explaining elevated levels of mortality, and we observe evidence for a harvesting effect related to heat waves.
- [51] arXiv:2405.18055 (replaced) [pdf, html, other]
-
Title: Dimension-free uniform concentration bound for logistic regressionComments: 28 pages; revised introduction and added numerical experimentsSubjects: Statistics Theory (math.ST); Machine Learning (stat.ML)
We provide a novel dimension-free uniform concentration bound for the empirical risk function of constrained logistic regression. Our bound yields a milder sufficient condition for a uniform law of large numbers than conditions derived by the Rademacher complexity argument and McDiarmid's inequality. The derivation is based on the PAC-Bayes approach with second-order expansion and Rademacher-complexity-based bounds for the residual term of the expansion.
- [52] arXiv:2407.12288 (replaced) [pdf, other]
-
Title: Information-Theoretic Foundations for Machine LearningSubjects: Machine Learning (stat.ML); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
The staggering progress of machine learning in the past decade has been a sight to behold. In retrospect, it is both remarkable and unsettling that these milestones were achievable with little to no rigorous theory to guide experimentation. Despite this fact, practitioners have been able to guide their future experimentation via observations from previous large-scale empirical investigations. However, alluding to Plato's Allegory of the cave, it is likely that the observations which form the field's notion of reality are but shadows representing fragments of that reality. In this work, we propose a theoretical framework which attempts to answer what exists outside of the cave. To the theorist, we provide a framework which is mathematically rigorous and leaves open many interesting ideas for future exploration. To the practitioner, we provide a framework whose results are very intuitive, general, and which will help form principles to guide future investigations. Concretely, we provide a theoretical framework rooted in Bayesian statistics and Shannon's information theory which is general enough to unify the analysis of many phenomena in machine learning. Our framework characterizes the performance of an optimal Bayesian learner, which considers the fundamental limits of information. Throughout this work, we derive very general theoretical results and apply them to derive insights specific to settings ranging from data which is independently and identically distributed under an unknown distribution, to data which is sequential, to data which exhibits hierarchical structure amenable to meta-learning. We conclude with a section dedicated to characterizing the performance of misspecified algorithms. These results are exciting and particularly relevant as we strive to overcome increasingly difficult machine learning challenges in this endlessly complex world.
- [53] arXiv:2407.13314 (replaced) [pdf, html, other]
-
Title: NIRVAR: Network Informed Restricted Vector AutoregressionComments: 27 pagesSubjects: Methodology (stat.ME); Applications (stat.AP)
High-dimensional panels of time series arise in many scientific disciplines such as neuroscience, finance, and macroeconomics. Often, co-movements within groups of the panel components occur. Extracting these groupings from the data provides a course-grained description of the complex system in question and can inform subsequent prediction tasks. We develop a novel methodology to model such a panel as a restricted vector autoregressive process, where the coefficient matrix is the weighted adjacency matrix of a stochastic block model. This network time series model, which we call the Network Informed Restricted Vector Autoregression (NIRVAR) model, yields a coefficient matrix that has a sparse block-diagonal structure. We propose an estimation procedure that embeds each panel component in a low-dimensional latent space and clusters the embedded points to recover the blocks of the coefficient matrix. Crucially, the method allows for network-based time series modelling when the underlying network is unobserved. We derive the bias, consistency and asymptotic normality of the NIRVAR estimator. Simulation studies suggest that the NIRVAR estimated embedded points are Gaussian distributed around the ground truth latent positions. On three applications to finance, macroeconomics, and transportation systems, NIRVAR outperforms competing models in terms of prediction and provides interpretable results regarding group recovery.
- [54] arXiv:2001.05989 (replaced) [pdf, html, other]
-
Title: Conformal e-predictionComments: This version is greatly expanded; 33 pages and 1 figureSubjects: Machine Learning (cs.LG); Machine Learning (stat.ML)
This paper discusses a counterpart of conformal prediction for e-values, "conformal e-prediction". Conformal e-prediction is conceptually simpler and had been developed in the 1990s as precursor of conformal prediction. When conformal prediction emerged as result of replacing e-values by p-values, it seemed to have important advantages over conformal e-prediction without obvious disadvantages. This paper re-examines relations between conformal prediction and conformal e-prediction systematically from a modern perspective. Conformal e-prediction has advantages of its own, such as the ease of designing conditional conformal e-predictors and the guaranteed validity of cross-conformal e-predictors (whereas for cross-conformal predictors validity is only an empirical fact and can be broken with excessive randomization). Even where conformal prediction has clear advantages, conformal e-prediction can often emulate those advantages, more or less successfully. Conformal e-prediction can also serve as basis for testing; the resulting "conformal e-testing" looks very different from but inherits some strengths of conformal testing.
- [55] arXiv:2207.00625 (replaced) [pdf, html, other]
-
Title: Projectivity revisitedComments: 30 pagesJournal-ref: International Journal of Approximate Reasoning 162:109031 (2023)Subjects: Artificial Intelligence (cs.AI); Probability (math.PR); Statistics Theory (math.ST)
The behaviour of statistical relational representations across differently sized domains has become a focal area of research from both a modelling and a complexity viewpoint.Recently, projectivity of a family of distributions emerged as a key property, ensuring that marginal probabilities are independent of the domain size. However, the formalisation used currently assumes that the domain is characterised only by its size. This contribution extends the notion of projectivity from families of distributions indexed by domain size to functors taking extensional data from a database. This makes projectivity available for the large range of applications taking structured input. We transfer key known results on projective families of distributions to the new setting. This includes a characterisation of projective fragments in different statistical relational formalisms as well as a general representation theorem for projective families of distributions. Furthermore, we prove a correspondence between projectivity and distributions on countably infinite domains, which we use to unify and generalise earlier work on statistical relational representations in infinite domains. Finally, we use the extended notion of projectivity to define a further strengthening, which we call $\sigma$-projectivity, and which allows the use of the same representation in different modes while retaining projectivity.
- [56] arXiv:2305.00050 (replaced) [pdf, html, other]
-
Title: Causal Reasoning and Large Language Models: Opening a New Frontier for CausalityComments: Added three novel datasets. To be published in TMLR. Authors listed alphabeticallySubjects: Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Computers and Society (cs.CY); Human-Computer Interaction (cs.HC); Machine Learning (cs.LG); Methodology (stat.ME)
The causal capabilities of large language models (LLMs) are a matter of significant debate, with critical implications for the use of LLMs in societally impactful domains such as medicine, science, law, and policy. We conduct a "behavorial" study of LLMs to benchmark their capability in generating causal arguments. Across a wide range of tasks, we find that LLMs can generate text corresponding to correct causal arguments with high probability, surpassing the best-performing existing methods. Algorithms based on GPT-3.5 and 4 outperform existing algorithms on a pairwise causal discovery task (97%, 13 points gain), counterfactual reasoning task (92%, 20 points gain) and event causality (86% accuracy in determining necessary and sufficient causes in vignettes). We perform robustness checks across tasks and show that the capabilities cannot be explained by dataset memorization alone, especially since LLMs generalize to novel datasets that were created after the training cutoff date.
That said, LLMs exhibit unpredictable failure modes, and we discuss the kinds of errors that may be improved and what are the fundamental limits of LLM-based answers. Overall, by operating on the text metadata, LLMs bring capabilities so far understood to be restricted to humans, such as using collected knowledge to generate causal graphs or identifying background causal context from natural language. As a result, LLMs may be used by human domain experts to save effort in setting up a causal analysis, one of the biggest impediments to the widespread adoption of causal methods. Given that LLMs ignore the actual data, our results also point to a fruitful research direction of developing algorithms that combine LLMs with existing causal techniques. Code and datasets are available at this https URL. - [57] arXiv:2305.02158 (replaced) [pdf, html, other]
-
Title: Shotgun crystal structure prediction using machine-learned formation energiesChang Liu (1), Hiromasa Tamaki (2), Tomoyasu Yokoyama (2), Kensuke Wakasugi (2), Satoshi Yotsuhashi (2), Minoru Kusaba (1), Artem R. Oganov (3), Ryo Yoshida (1 and 4) ((1) The Institute of Statistical Mathematics, (2) Panasonic Holdings Corporation, (3) Skolkovo Innovation Center, (4) The Graduate University for Advanced Studies)Subjects: Computational Physics (physics.comp-ph); Materials Science (cond-mat.mtrl-sci); Machine Learning (stat.ML)
Stable or metastable crystal structures of assembled atoms can be predicted by finding the global or local minima of the energy surface within a broad space of atomic configurations. Generally, this requires repeated first-principles energy calculations, which is often impractical for large crystalline systems. Here, we present significant progress toward solving the crystal structure prediction problem: we performed noniterative, single-shot screening using a large library of virtually created crystal structures with a machine-learning energy predictor. This shotgun method (ShotgunCSP) has two key technical components: transfer learning for accurate energy prediction of pre-relaxed crystalline states, and two generative models based on element substitution and symmetry-restricted structure generation to produce promising and diverse crystal structures. First-principles calculations were performed only to generate the training samples and to refine a few selected pre-relaxed crystal structures. The ShotunCSP method is computationally less intensive than conventional methods and exhibits exceptional prediction accuracy, reaching 93.3% in benchmark tests with 90 different crystal structures.
- [58] arXiv:2309.13333 (replaced) [pdf, html, other]
-
Title: mdendro: An R package for extended agglomerative hierarchical clusteringComments: 29 pages, 15 figures. Software available at CRAN (this https URL) and Github (this https URL)Subjects: Information Retrieval (cs.IR); Data Analysis, Statistics and Probability (physics.data-an); Computation (stat.CO)
"mdendro" is an R package that provides a comprehensive collection of linkage methods for agglomerative hierarchical clustering on a matrix of proximity data (distances or similarities), returning a multifurcated dendrogram or multidendrogram. Multidendrograms can group more than two clusters at the same time, solving the nonuniqueness problem that arises when there are ties in the data. This problem causes that different binary dendrograms are possible depending both on the order of the input data and on the criterion used to break ties. Weighted and unweighted versions of the most common linkage methods are included in the package, which also implements two parametric linkage methods. In addition, package "mdendro" provides five descriptive measures to analyze the resulting dendrograms: cophenetic correlation coefficient, space distortion ratio, agglomeration coefficient, chaining coefficient and tree balance.
- [59] arXiv:2311.08815 (replaced) [pdf, html, other]
-
Title: Self-Supervised Disentanglement by Leveraging Structure in Data AugmentationsCian Eastwood, Julius von Kügelgen, Linus Ericsson, Diane Bouchacourt, Pascal Vincent, Bernhard Schölkopf, Mark IbrahimSubjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computer Vision and Pattern Recognition (cs.CV); Machine Learning (stat.ML)
Self-supervised representation learning often uses data augmentations to induce some invariance to "style" attributes of the data. However, with downstream tasks generally unknown at training time, it is difficult to deduce a priori which attributes of the data are indeed "style" and can be safely discarded. To deal with this, current approaches try to retain some style information by tuning the degree of invariance to some particular task, such as ImageNet object classification. However, prior work has shown that such task-specific tuning can lead to significant performance degradation on other tasks that rely on the discarded style. To address this, we introduce a more principled approach that seeks to disentangle style features rather than discard them. The key idea is to add multiple style embedding spaces where: (i) each is invariant to all-but-one augmentation; and (ii) joint entropy is maximized. We formalize our structured data-augmentation procedure from a causal latent-variable-model perspective, and prove identifiability of both content and individual style variables. We empirically demonstrate the benefits of our approach on both synthetic and real-world data.
- [60] arXiv:2405.10839 (replaced) [pdf, html, other]
-
Title: Model orthogonalization and Bayesian forecast mixing via Principal Component AnalysisComments: 10 pages, 4 figuresSubjects: Nuclear Theory (nucl-th); Data Analysis, Statistics and Probability (physics.data-an); Machine Learning (stat.ML)
One can improve predictability in the unknown domain by combining forecasts of imperfect complex computational models using a Bayesian statistical machine learning framework. In many cases, however, the models used in the mixing process are similar. In addition to contaminating the model space, the existence of such similar, or even redundant, models during the multimodeling process can result in misinterpretation of results and deterioration of predictive performance. In this work we describe a method based on the Principal Component Analysis that eliminates model redundancy. We show that by adding model orthogonalization to the proposed Bayesian Model Combination framework, one can arrive at better prediction accuracy and reach excellent uncertainty quantification performance.
- [61] arXiv:2406.05185 (replaced) [pdf, html, other]
-
Title: Tree balance in phylogenetic modelsSubjects: Populations and Evolution (q-bio.PE); Applications (stat.AP)
Tree shape statistics, particularly measures of tree (im)balance, play an important role in the analysis of the shape of phylogenetic trees. With applications ranging from testing evolutionary models to studying the impact of fertility inheritance and selection, or tumor development and language evolution, the assessment of measures of tree balance is important. Currently, a multitude of at least 30 (im)balance indices can be found in the literature, alongside numerous other tree shape statistics.
This diversity prompts essential questions: How can we assist researchers to choose only a small number of indices to mitigate the challenges of multiple testing? Is there a preeminent balance index tailored to specific tasks?
This research expands previous studies on the examination of index power, encompassing almost all established indices and a broader array of alternative models, such as a variety of trait-based models. Our investigation reveals distinct groups of balance indices better suited for different tree models, suggesting that decisions on balance index selection can be enhanced with prior knowledge. Furthermore, we present the \textsf{R} software package \textsf{poweRbal} which allows the inclusion of new indices and models, thus facilitating future research on the power of tree shape statistics. - [62] arXiv:2408.05431 (replaced) [pdf, html, other]
-
Title: Simple and Nearly-Optimal Sampling for Rank-1 Tensor Completion via Gauss-JordanComments: 16 pages; corrected typos in Prior Work section & Theorem 1.5Subjects: Data Structures and Algorithms (cs.DS); Machine Learning (cs.LG); Statistics Theory (math.ST); Machine Learning (stat.ML)
We revisit the sample and computational complexity of completing a rank-1 tensor in $\otimes_{i=1}^{N} \mathbb{R}^{d}$, given a uniformly sampled subset of its entries. We present a characterization of the problem (i.e. nonzero entries) which admits an algorithm amounting to Gauss-Jordan on a pair of random linear systems. For example, when $N = \Theta(1)$, we prove it uses no more than $m = O(d^2 \log d)$ samples and runs in $O(md^2)$ time. Moreover, we show any algorithm requires $\Omega(d\log d)$ samples.
By contrast, existing upper bounds on the sample complexity are at least as large as $d^{1.5} \mu^{\Omega(1)} \log^{\Omega(1)} d$, where $\mu$ can be $\Theta(d)$ in the worst case. Prior work obtained these looser guarantees in higher rank versions of our problem, and tend to involve more complicated algorithms. - [63] arXiv:2408.08493 (replaced) [pdf, html, other]
-
Title: Fishers Harvest Parallel Unlearning in Inherited Model NetworksSubjects: Machine Learning (cs.LG); Machine Learning (stat.ML)
Unlearning in various learning frameworks remains challenging, with the continuous growth and updates of models exhibiting complex inheritance relationships. This paper presents a novel unlearning framework, which enables fully parallel unlearning among models exhibiting inheritance. A key enabler is the new Unified Model Inheritance Graph (UMIG), which captures the inheritance using a Directed Acyclic Graph (DAG).Central to our framework is the new Fisher Inheritance Unlearning (FIUn) algorithm, which utilizes the Fisher Information Matrix (FIM) from initial unlearning models to pinpoint impacted parameters in inherited models. By employing FIM, the FIUn method breaks the sequential dependencies among the models, facilitating simultaneous unlearning and reducing computational overhead. We further design to merge disparate FIMs into a single matrix, synchronizing updates across inherited models. Experiments confirm the effectiveness of our unlearning framework. For single-class tasks, it achieves complete unlearning with 0\% accuracy for unlearned labels while maintaining 94.53\% accuracy for retained labels on average. For multi-class tasks, the accuracy is 1.07\% for unlearned labels and 84.77\% for retained labels on average. Our framework accelerates unlearning by 99\% compared to alternative methods.
- [64] arXiv:2408.09632 (replaced) [pdf, html, other]
-
Title: MoDeGPT: Modular Decomposition for Large Language Model CompressionChi-Heng Lin, Shangqian Gao, James Seale Smith, Abhishek Patel, Shikhar Tuli, Yilin Shen, Hongxia Jin, Yen-Chang HsuComments: 31 pages, 9 figuresSubjects: Machine Learning (cs.LG); Computation and Language (cs.CL); Machine Learning (stat.ML)
Large Language Models (LLMs) have reshaped the landscape of artificial intelligence by demonstrating exceptional performance across various tasks. However, substantial computational requirements make their deployment challenging on devices with limited resources. Recently, compression methods using low-rank matrix techniques have shown promise, yet these often lead to degraded accuracy or introduce significant overhead in parameters and inference latency. This paper introduces \textbf{Mo}dular \textbf{De}composition (MoDeGPT), a novel structured compression framework that does not need recovery fine-tuning while resolving the above drawbacks. MoDeGPT partitions the Transformer block into modules comprised of matrix pairs and reduces the hidden dimensions via reconstructing the module-level outputs. MoDeGPT is developed based on a theoretical framework that utilizes three well-established matrix decomposition algorithms -- Nyström approximation, CR decomposition, and SVD -- and applies them to our redefined transformer modules. Our comprehensive experiments show MoDeGPT, without backward propagation, matches or surpasses previous structured compression methods that rely on gradient information, and saves 98% of compute costs on compressing a 13B model. On \textsc{Llama}-2/3 and OPT models, MoDeGPT maintains 90-95% zero-shot performance with 25-30% compression rates. Moreover, the compression can be done on a single GPU within a few hours and increases the inference throughput by up to 46%.