Extension of the one-sample Kolmogorov-Smirnov test
Abstract
We propose here a new goodness-of-fit test, named the one-sample OVL- test (), which can be considered an extension of the one-sample Kolmogorov-Smirnov test (equivalent to the one-sample OVL-1 test). We have analyzed the asymptotic properties of the one-sample OVL-2 test statistic and enabled the calculation of asymptotic p-values for the test statistic. We further conducted numerical experiments and demonstrated that the one-sample OVL-2 test can sometimes exceed the detection power of conventional goodness-of-fit tests.
MSC2020 subject classifications: Primary 62G10; Secondary 62G20, 62-04.
Keywords and phrases: Nonparametric statistics, Kolmogorov-Smirnov test, One-sample testing.
1 Introduction
The Kolmogorov-Smirnov (KS) test is a nonparametric method used to determine whether a sample originates from a specific probability distribution (one-sample KS test) or to assess whether two samples come from the same distribution (two-sample KS test). In our previous study, we devised an extended version of the two-sample KS test, named the (two-sample) OVL- test (), and demonstrated its utility particularly for the two-sample OVL-2 test [6].
In this study, we extended the one-sample KS test using a similar approach to our previous work [6], and named it the one-sample OVL- test (). We analyze the asymptotic properties of the one-sample OVL-2 test statistic and calculate its asymptotic p-values. Furthermore, we assess the detection power of the one-sample OVL-2 test relative to conventional goodness-of-fit tests by conducting numerical experiments.
In this paper, we describe the analytical framework in Section 2. Experimental results are shown in Section 3. Conclusion follows in Section 4. The proofs of Theorems 2.3, 2.4 and 2.5 are given in Section 5. The source code for the experiments in Section 3 is provided in the Supplementary Material.
General notation
We denote by , , , , and the sets of integers, nonnegative integers, positive integers, rational numbers, and real numbers, respectively. If and if there is no confusion, we write , , , and as (extended) real intervals. For , let be the Euclidean -dimensional space and . For a topological space , we denote by the -algebra of Borel sets in . For a set , denotes the cardinality of . For a real function on a set and , we write . We denote by the indicator function of a set . For a random variable , denotes its expectation.
2 Analytical framework
Let be the set of distribution functions on , where each is nondecreasing, is continuous from the right, and satisfies and . For and ,
(1) |
where
(2) |
for , , and . Note that for all , so that .
Theorem 2.1.
(See [7] for reference.) The KS metric on equals , that is,
Theorem 2.2.
For each , is a complete metric space.
Theorem 2.1 can be proved similarly as in the proof of [6, Proposition 2.7]. Theorem 2.2 will be proved in Section 5.1. By these theorems, we can see that are extension of the KS metric. Furthermore, by Theorem 5.11, any and generate the same topology on .
2.1 One-sample KS test and its extension
As a null hypothesis , we assume that are independent and identically distributed (i.i.d.) random variables on a probability space with a given distribution function . Let be the corresponding empirical distribution function, i.e.,
(3) |
Here we propose () as an extension of the one-sample KS test statistic, which equals by Theorem 2.1. The p-value (function) of the extended test is given by
(4) |
and the upper limit of a % confidence interval () of is
(5) |
This can be regarded as the one-sample OVL- test since the (two-sample) OVL- test statistic in [6, Definition 2.2] equals (see [6, Definition 2.1]), whose p-value is equal to that of as described in [6, Section 2.3].
Theorem 2.3.
For each , converges completely to as , i.e.,
for any .
Note that complete convergence implies almost sure convergence, as described in [5, Remark 4.4].
Theorem 2.4.
For each , the distribution of is the same for all continuous .
Theorem 2.5.
If is continuous on ,
(6) |
for any .
Note that in 6 is independent of any continuous by Theorem 2.4. By this theorem, we have the asymptotic distribution function of :
(7) |
See Sections 5.2, 5.3 and 5.4 for the proofs of Theorems 2.3, 2.4 and 2.5, respectively.
3 Numerical experiments
We conducted a computer-based experiment to compare the statistical power of the one-sample OVL-2 test with that of conventional statistical tests, including the one-sample KS test.
Beforehand, we computed , as defined in 4, using the Monte Carlo method for , because exact p-values for the one-sample OVL-2 test could not be computed. More specifically, instead of , we used the empirical distribution function of computed from 100,000 samples of size drawn from the standard uniform distribution, whose distribution function is defined as
(8) |
Here, represents in the case . The empirical distribution functions for are shown in Fig. 1 together with the theoretical asymptotic distribution function given in 7.
![Refer to caption](x1.png)
The probability density functions used in the experiments are defined as follows:
(, , ), | ||||
(), | ||||
and they are illustrated in Fig. 2.
![Refer to caption](x2.png)
First, we chose two different distributions as the sampling distribution and the reference distribution. Then we repeated the following trial 100,000 times for each sample size : a random sample of size was drawn from the sampling distribution, and tested under the null hypothesis that it were drawn from the reference distribution. We performed the one-sample OVL-2 test, the one-sample KS test, and the Cramér-von Mises test for each sample, and counted the number of times that the null hypothesis was rejected at 0.05 level of significance. The rate of rejection out of 100,000 trials was assumed to represent the statistical power of the test. The entire source code for the experiment, written in Python 3.11.8, is provided as the Supplementary Material on pages Supplementary Material–Supplementary Material.
The result is shown in Fig. 3. When the sampling distribution was and the reference distribution was , the powers of the Cramér-von mises test and the one-sample KS test were respectively the first and second highest of the three, while the power of the one-sample OVL-2 test was lower. When the sampling distribution was , , or and the reference distribution was , the power of the one-sample OVL-2 test was the highest among the three, and the powers of the other two were almost equally lower. When the sampling distribution was and the reference distribution was , or vice versa, the power of the one-sample OVL-2 test was the highest, followed by that of the one-sample KS test, and that of the Cramér-von Mises test.
4 Conclusion
In this study, we have developed the one-sample OVL- test () as a new goodness-of-fit test, which can also be considered an extended version of the one-sample KS test (because the one-sample KS test is equivalent to the one-sample OVL-1 test). We analyzed the asymptotic properties of the one-sample OVL-2 test statistic and enabled the calculation of its asymptotic p-values.
We conducted numerical experiments to compare the detection power of the one-sample OVL-2 test with conventional goodness-of-fit tests, including the one-sample KS test. In several instances, the one-sample OVL-2 test demonstrated superior performance, suggesting its potential utility.
The limitations of this study are as follows:
-
•
We have not presented a calculation method for the one-sample OVL- test when .
-
•
We calculated asymptotic p-values for the one-sample OVL-2 test statistic, but these are not applicable when the sample size is small.
-
•
We have not proposed a method for calculating p-values when the sample size is small, particularly exact methods.
To make the one-sample OVL-2 test practical, these issues need to be addressed in future research.
![Refer to caption](x3.png)
5 Proofs
5.1 Proof of Theorem 2.2
Let us denote by the set of bounded right-continuous real functions on , and by the supremum norm on , i.e.,
(9) |
Remark 5.1.
We can easily see that is a normed linear space with norm , and that is a convex subset of .
Theorem 5.2.
is a Banach space with norm .
Proof.
Suppose is a Cauchy sequence in . For each , implies that is a Cauchy sequence, so that there exists by the completeness of the real line.
For any , there exists such that implies , so that for all , i.e., , which also implies that .
For any , there exists with by the argument above. For each , there exists such that for all since is right-continuous, so that
for all . Hence is right-continuous.
Now we see that and , and the proof is complete. ∎
Lemma 5.3.
is a closed subspace of .
Proof.
Let be a convergent sequence in and .
For each , () implies that . Hence for any , since for all . This means that is nondecreasing.
For any , there exists with . Since , there exists such that for all . Hence for all . Therefore, . The proof for is similar.
Taken together, we have shown that . ∎
The following theorem follows immediately from Theorems 2.1, 5.2 and 5.3.
Theorem 5.4.
is a complete metric space.
Remark 5.5.
We can see that is not separable. For example, the collection of open subsets
is pairwise disjoint and uncountable. Here note that if .
Lemma 5.6.
For any , .
Proof.
We can assume that without loss of generality. If , then . If , then . ∎
Theorem 5.7.
For each , is a metric space.
Proof.
We have to show that, for all ,
-
(a)
.
-
(b)
if and only if .
-
(c)
.
-
(d)
.
Lemma 5.8.
For any , if .
Proof.
Since by definition, holds. ∎
Lemma 5.9.
For each , for all .
Proof.
It follows from definition that
and the proof is complete. ∎
Remark 5.10.
As shown in the proof above, we obtain the equation
The following theorem follows immediately from Lemmas 5.8 and 5.9.
Theorem 5.11.
For each , for all .
Now Theorem 2.2 follows from Theorems 5.4, 5.7 and 5.11.
5.2 Proof of Theorem 2.3
Theorem 5.12.
(The Glivenko-Cantelli theorem. See the proof of [8, Theorem A, Section 2.1.4].) converges completely to 0 as , i.e.,
for any .
5.3 Proof of Theorem 2.4
Let us denote by the quantile function of , i.e.,
(10) |
with the convention that and . Let be the standard uniform distribution function defined in 8.
Theorem 5.13.
Let be i.i.d. random variables on a probability space with , for , and
As described in Section 2.1, are i.i.d. random variables on with and
The following colloraries are immediate consequences of Theorem 5.13.
Corollary 5.14.
The random variables are i.i.d. with the same distribution function . The probability measure on induced by and that by are the same, i.e.,
Corollary 5.15.
It holds almost surely that .
Note that holds obviously.
Theorem 5.16.
For each , almost surely. If is continuous on , almost surely.
Proof.
Let us define, for each ,
We also define, for each and ,
Remark 5.17.
We see that and on , and on .
Theorem 5.18.
For each and , is a Borel measurable function on .
Proof.
Let us put . It is obvious that
(11) |
For any , there exists such that . Let be a sequence in such that for each , converges to from the right as . Since and are right-continuous on , . Hence for some . With 11, we have
(12) |
so that
(13) |
by definition.
It is immediate from definition that and are Borel measurable for each and , respectively. Hence can be described by the countable infimum of Borel measurable functions by 13. This implies the claim. ∎
The next corollary follows from Remarks 5.17 and 5.18.
Corollary 5.19.
and are random variables on , and is a random variable on .
Theorem 5.20.
For each , the probability measure on induced by and that by are the same if is continuous on .
Proof.
This theorem implies Theorem 2.4.
5.4 Proof of Theorem 2.5
Definition 5.21.
For , , and , define
We also put and . Note that since .
Lemma 5.22 (cf. [6, Lemma 7.4]).
For any , .
Proof.
Given , we have
If and , then
If and , then
If , then , hence
Taken together, holds in general. Since was arbitrary, we obtain .
On the other hand, for any , there exist such that and . Putting , , and , we have
Since was arbitrary, we obtain . ∎
Lemma 5.23.
For any , .
Proof.
Definition 5.24.
(See [2, pages 353 and 443] for reference.) Let be a set and a probability space. A mapping is called a stochastic process if is measurable for each . We say that is Gaussian if is Gaussian for any .
Definition 5.25.
(See [2, page 445] for reference.) Let be a Gaussian stochastic process with . If the following conditions hold:
-
•
for any ,
-
•
for any with ,
-
•
is sample continuous, i.e., is continuous for any ,
then is called a Brownian bridge.
Theorem 5.26.
[2, Proposition 12.3.4]. For a Brownian bridge and any ,
Theorem 5.27.
[2, Proposition 12.3.6]. For a Brownian bridge and any ,
Definition 5.28.
(See [1, Section 12] for reference.) Let be the space of real functions on that are right-continuous and have left-hand limits (such functions are called càdlàg functions). Let be the set of strictly increasing, continuous mappings of onto itself. For , define
where denotes the identity map on . The function is a metric on , which defines the Skorohod topology.
Lemma 5.29.
(See [1, page 124] for reference.) For an element and a sequence in , if and only if and for some sequence in .
Proof.
If , then . Since there exists for each such that
and hold.
On the other hand, suppose and for some sequence in . Then for any , there exists such that implies and , so that . Hence . ∎
Lemma 5.30.
For , the following inequalities hold:
Proof.
For any , there exists such that . Then . Since was arbitrary, holds. Similarly, holds, and the first inequality is proved. The second one can be proved in a similar way. ∎
Theorem 5.31.
The function defined by is continuous.
Proof.
For a given , let be a sequence in such that . It suffices to prove that . By Lemma 5.29, there exists a sequence in such that and . By Lemma 5.30, we have
Hence . ∎
Definition 5.32.
(See [1, pages 7 and 15–16] for reference.) Suppose is a Borel probability measure and a sequence of Borel probability measures on a metric space . We say that converges weakly to (denoted by ) if for all bounded continuous functions , where . A set whose boundary satisfies is called a -continuity set.
Theorem 5.33.
[1, Theorem 2.1]. Suppose is a Borel probability measure and a sequence of Borel probability measures on a metric space . Then these five conditions are equivalent:
-
(i)
.
-
(ii)
for all bounded, uniformly continuous functions .
-
(iii)
for any closed set .
-
(iv)
for any open set .
-
(v)
for any -continuity set .
Definition 5.34.
(See [1, pages 24–26] for reference.) We call a map from a probability space to a metric space a random element if it is Borel measurable. (As is customary, we call it a random variable if, in addition, .) Suppose is a random element and a sequence of random elements from to . The law of is the Borel probability measure on . We say that converges in distribution to (denoted by ) if . A set with is called a -continuity set.
Theorem 5.35.
[1, page 26]. Suppose is a random element and a sequence of random elements from a probability space to a metric space . Then these five conditions are equivalent:
-
(i)
.
-
(ii)
for all bounded, uniformly continuous functions .
-
(iii)
for any closed set .
-
(iv)
for any open set .
-
(v)
for any -continuity set .
Theorem 5.36.
[1, page 20]. Suppose is a Borel probability measure and a sequence of Borel probability measures on a metric space . Let be another metric space and a continuous map. If , then .
Remark 5.37 (See [1, page 135] for reference).
Note that is a random element if and only if is a stochastic process (i.e., is a random variable for all ).
Theorem 5.38.
Suppose are i.i.d. random variables on a probability space with a continuous distribution function , and for . Then
(14) |
for any .
Proof.
Let be defined as in Section 5.3 on , for , and a Brownian bridge. Then and are random elements on to , and (which means ) by [1, Theorem 14.3]. Since
for any by Theorem 5.27, and by Theorems 5.31, 5.36 and 5.33, we obtain
(15) |
Note that is an -continuity set. It holds almost surely that , . The continuity of implies that . Therefore, we have
by Corollaries 5.14, 5.15 and 5.23. With 15, we obtain 14. ∎
This theorem is equivalent to Theorem 2.5.
Acknowledgement
We thank Akitomo Amakawa (University of Yamanashi) for the validation of our numerical results.
Funding
This study was partially supported by JSPS KAKENHI Grant Numbers JP21K15762, JP20K03509, and JP24K06686.
Supplementary Material
The source code of the experiment in Section 3. The following packages are required:
-
•
numpy==1.26.4
-
•
pandas==2.2.2
-
•
scipy==1.13.0
-
•
matplotlib==3.8.4
This work © 2024 by Atsushi Komaba is licensed under Creative Commons Attribution 4.0 International. To view a copy of this license, visit https://creativecommons.org/licenses/by/4.0/
References
- [1] P. Billingsley. Convergence of probability measures. Wiley Series in Probability and Statistics: Probability and Statistics. John Wiley & Sons Inc., New York, second edition, 1999. A Wiley-Interscience Publication.
- [2] R. M. Dudley. Real Analysis and Probability. Cambridge Studies in Advanced Mathematics. Cambridge University Press, second edition, 2002.
- [3] R. M. Dudley. Uniform Central Limit Theorems. Cambridge Studies in Advanced Mathematics. Cambridge University Press, second edition, 2014.
- [4] P. Embrechts and M. Hofert. A note on generalized inverses. Mathematical Methods of Operations Research, 77(3):423–432, 2013.
- [5] H. Johno and K. Nakamoto. Decision tree-based estimation of the overlap of two probability distributions. Journal of Mathematical Sciences, the University of Tokyo, 30(1):21–54, 2023.
- [6] A. Komaba, H. Johno, and K. Nakamoto. A novel statistical approach for two-sample testing based on the overlap coefficient. Journal of Mathematical Sciences, the University of Tokyo, 30(2):205–240, 2023.
- [7] L. Molnár and P. Szokol. Kolmogorov-Smirnov isometries of the space of generalized distribution functions. Mathematica Slovaca, 64(2):433–444, 2014.
- [8] R. J. Serfling. Approximation theorems of mathematical statistics. John Wiley & Sons, New York, 1980.