Professional Documents
Culture Documents
Distribution System
Distribution System
by
Sanjoy Dasgupta
in
Computer Science
in the
GRADUATE DIVISION
of the
UNIVERSITY of CALIFORNIA at BERKELEY
Committee in charge:
Spring 2000
The dissertation of Sanjoy Dasgupta is approved:
Chair Date
Date
Date
Spring 2000
1
Abstract
by
Sanjoy Dasgupta
Doctor of Philosophy in Computer Science
The first part of this thesis presents an algorithm which takes data from an unknown
mixture of Gaussians in arbitrarily high dimension and recovers the parameters of this
mixture to within the precision desired by the user. There are two restrictions on the
mixture: its component Gaussians must be well-separated in a precise sense, and they
must have a common (though of course unknown) non-singular covariance matrix. The
running time of the algorithm is linear in the dimension of the data and polynomial in the
number of Gaussians.
This algorithm is very simple, and relies crucially upon a particular procedure for
dimensionality reduction. Data from a mixture of k Gaussians are projected to a randomly
chosen O(log k)-dimensional subspace, regardless of the original dimension and the number
of data points, and it is shown that not only does this retain enough information for cluster-
ing, but it also causes a drastic reduction in the eccentricity of the Gaussians. Experiments
are performed to illustrate some of the promise of this random projection technique.
The second half of this thesis studies the problem of learning the maximum-likelihood
directed probabilistic net given data. Three combinatorial optimization tasks are distin-
guished: SL(k), learning the optimal probabilistic net in which each node has at most k
parents; PT, learning the optimal polytree; and PT(k), learning the optimal polytree with
an indegree bound of k. It is well-known that SL(1) is efficiently solvable while SL(2) is
an NP-hard optimization problem. We demonstrate an even more damaging hardness bar-
rier, that for some constant c > 1, unless P = NP there is no polynomial-time algorithm
which can c-approximate SL(2), that is, which can consistently return a solution whose
2
Contents
I Introduction 1
1 Massive data sets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
2 Learning probability distributions . . . . . . . . . . . . . . . . . . . . . . . . 3
3 Current practice . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
4 A summary of results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
II Mixtures of Gaussians 12
1 Assessing the quality of a fit . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
2 Learning algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
2.1 The method of moments . . . . . . . . . . . . . . . . . . . . . . . . . 15
2.2 Expectation-maximization . . . . . . . . . . . . . . . . . . . . . . . . 16
3 High-dimensional Gaussians . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
3.1 Some counter-intuitive effects . . . . . . . . . . . . . . . . . . . . . . 17
3.2 Formalizing separation . . . . . . . . . . . . . . . . . . . . . . . . . . 19
3.3 The problem of dimension . . . . . . . . . . . . . . . . . . . . . . . . 20
4 Dimensionality reduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
4.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
4.2 Principal component analysis . . . . . . . . . . . . . . . . . . . . . . 21
4.3 Random projection . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
4.4 Random projection for mixtures of Gaussians . . . . . . . . . . . . . 22
4.5 Illustrative experiments . . . . . . . . . . . . . . . . . . . . . . . . . 27
IV Probabilistic nets 57
1 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
1.1 Conditional probability functions . . . . . . . . . . . . . . . . . . . . 58
1.2 Inference . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
2 A maximum-likelihood learning model . . . . . . . . . . . . . . . . . . . . . 59
2.1 Entropy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
2.2 Maximizing likelihood . . . . . . . . . . . . . . . . . . . . . . . . . . 60
2.3 Other learning problems . . . . . . . . . . . . . . . . . . . . . . . . . 61
3 SL(k) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
3.1 Previous work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
3.2 SL(2) is hard to approximately solve . . . . . . . . . . . . . . . . . . 64
V Learning polytrees 72
1 Why polytrees? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72
2 An approximation algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . 73
2.1 A simple bound . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
2.2 A better performance guarantee . . . . . . . . . . . . . . . . . . . . 74
2.3 The final improvement . . . . . . . . . . . . . . . . . . . . . . . . . . 78
3 A hardness result . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80
Literature cited 93
v
Acknowledgements
I would like to thank my advisor, Umesh Vazirani, for guiding me with care through
my time in graduate school, and providing practical advice and help on the many occasions
when I needed them. My research also owes a lot to discussions with Yoav Freund, Nir
Friedman, Anupam Gupta, Mike Jordan, Daphne Koller, Christos Papadimitriou, Stuart
Russell, Leonard Schulman, Eric Vigoda, and David Zuckerman, and to the generosity and
encouragement of Peter Bickel.
1
Chapter I
Introduction
The wide proliferation of computers has raised hopes that it might be possible to
automatically analyze collections of data and provide concise, useful models of them. For
a simple example of the kind of computational task involved, consider a gel electrophoresis
experiment in which protein molecules in solution are mobilized by the application of a
potential difference (Longsworth, 1942; Titterington, Smith and Makov, 1985). Speeds of
some of these are then measured in the hope of gauging the relative proportions of different
proteins. Molecules of a particular protein can be thought of as having an ideal speed
which is perturbed by a host of environmental factors, like convection, of combined standard
deviation some . The measured speed of one of these molecules can then conveniently
be modeled as a Gaussian (normal) distribution with mean and variance 2 , denoted
N (, 2 ). This particular choice of distribution is prevalent throughout the physical sciences
and has a universality that is captured in the central limit theorem (Feller, 1966).
If there are k distinct proteins in the solution, with ideal speeds 1 , 2 , . . . , k and
noise levels 1 , . . . , k , then the distribution of the measurements as a whole is a mixture of
Gaussians,
w1 N (1 , 12 ) + w2 N (2 , 22 ) + + wk N (k , k2 ),
where wi is the proportion in which the ith protein is present. The experimenter may decide
that molecules of different proteins are subject to the same noise processes, in which case
he can set 1 = 2 = = k . The task then is to fit a mixture of Gaussians to the
observations, as a useful summary of them. Once this is done, a potentially confusing list
of measurements has been condensed into a brief characterization of each protein, which
Introduction 2
can be used to answer general questions like is one protein much more common than the
others?, and, if need be, to classify new measurements.
Can this kind of modelling be reliably computerized? What assurances could we
reasonably expect? This issue is particularly important in the case of high-dimensional
data, where it is difficult to visually check the computers results. The need for performance
guarantees in data analysis is the central preoccupation of this thesis.
the very high-dimensional space Rn . A possible goal is to group together similar documents
in the hope that this will facilitate subsequent queries.
The impressive number of samples in such a database poses various computational
difficulties. It brings up issues of caching and paging, and strongly discourages too many
passes through the data. However the more serious and fundamental concern is the high
number of attributes per data point, which has been nicknamed the curse of dimensional-
ity (Bellman, 1961).
What exactly is this curse? First, high-dimensional data is very hard to visualize.
Finding the mean and covariance, or looking at various two-dimensional projections, can be
helpful but also misleading. Diaconis and Freedman (1984) have demonstrated simple fami-
lies of non-Gaussian distributions most of whose two-dimensional projections look Gaussian.
Second, high-dimensional geometry contains many counter-intuitive pitfalls. One such ef-
fect can be observed by choosing points uniformly at random in an n-sphere and noticing
that they almost always lie close to the surface of the sphere. We shall encounter quite a
few more in the coming chapters. These two problems partially explain the basic difficulty,
which is this: much of current algorithmic technology is very clearly directed toward low-
dimensional data and becomes impractical when naively scaled to higher dimension. For
instance, Voronoi diagrams are a staple of two- and three-dimensional computational geom-
etry but lead to exponentially complex partitions of space when the dimension is increased;
m points in Rn will typically generate Voronoi cells with (mn ) faces.
This thesis studies in detail two popular families of probability distributions. They
are most easily introduced by example.
The Infrared Astronomical Satellite (IRAS) data set, described by Cheeseman et
al. (1995), contains 5425 points in 100-dimensional space, each representing information
from one of an unknown number of distinct galaxies. The authors decided to model the
data from each galaxy by a high-dimensional Gaussian. These distributions are natural
generalizations of the univariate Gaussian presented earlier. A Gaussian in n dimensions is
indexed by its mean Rn and covariance matrix , and is denoted N (, ). It assigns
Introduction 4
Fever
SoreThroat Headache
Findings
Diseases
other, but that they both make Fever more likely. This can be represented by the graph
in Figure 1 or equivalently by the factorization
A more sophisticated example is the QMR-DT medical network (Shwe et al., 1989; Jaakkola
and Jordan, 1999), a graphical model which attempts to encapsulate medical knowledge
about certain diseases and findings (symptoms). It contains nodes for approximately 600
diseases and 4,000 findings. These are arranged into a bipartite graph, with edges from
diseases to findings (Figure 2); the common jargon for this is that the disease nodes are
sources of the graph, and are parents of the symptom nodes. The network is annotated
with the conditional probability distribution of each node given its parents, and these then
specify a joint probability distribution over all the variables. A particularly appealing
feature of such networks is that they are easy to read and immediately communicate some
basic correlations between the different variables. The network structure can also in some
cases be constructed, or constrained, by consultation with experts. In such cases, as in the
QMR-DT net, it is common for the edges to reflect some interpretation of causality among
the variables.
Directed probabilistic nets are always acyclic. A maximally connected acyclic net can
Introduction 6
X+Y
X Y Z X Y Z
represent any distribution over the variables and requires potentially enormous conditional
probability distributions to be stored. In constructing these networks, therefore, a primary
concern is to limit connectivity, or more specifically, to limit the number of parents of each
node.
Probabilistic nets have become popular because they permit some complicated proba-
bility distributions to be expressed in a compact and easily understandable manner. Some-
times hidden nodes are introduced, representing random variables which are not directly
observed but which either permit a more concise model or correspond to some hypothesized
underlying causal mechanism. Figure 3, for instance, shows how a graph on six observable
variables can be simplified by the creation of a hidden node.
Once a probabilistic net has been created, it can be used to address conditional
probability queries about the distribution. For example, given the QMR-DT network, one
might reasonably ask, what is the chance that I have so-and-so disease if I am exhibiting
these symptoms? Answering such questions is called inference and in general is NP-hard
to perform with even approximate accuracy (Cooper, 1990; Dagum and Luby, 1993). There
are certain classes of networks for which efficient exact inference is possible, however. The
most natural such class is that of directed trees, called polytrees in the artificial intelligence
literature (Pearl, 1988).
It is becoming increasingly common to construct complex probability models out of
simpler ones. For instance, many speech recognition systems use both mixture models and
probabilistic nets as subcomponents. Depending upon style of speaking and environmental
factors, there are many different sequences of sound signals which can convey the same word,
or syllable, or phoneme. Therefore, for any particular phoneme, there is some distribution
over these various possible acoustic sequences. It is standard to summarize this distribution
by a hidden Markov model (Rabiner, 1989; Jelinek, 1997). Figure 4 is an example of such
Introduction 7
0.5
0.6
1 2
0.4 1.0
1.0
0.5
5
3
0.2
4 0.1
0.7
Figure 4: The underlying state graph of a simple hidden Markov model. The numbers on
the edges are transition probabilities.
q1 q2 q3
...
O1 O2 O3
Figure 5: A time-slice depiction of the Markov model of Figure 4. Here qt is the state (1-5)
at time t, and Ot is the observation at time t.
a model. The graph is annotated by an initial distribution over the five states (nodes) and
by transition probabilities on the edges. In addition, each state has its own distribution
over sound signals, called its observable distribution. A sound sequence can be generated
by performing a standard random walk on this graph, and at each time step choosing a
sound from the observable distribution of the current state. The graph is sometimes drawn
as a directed probabilistic net with nodes for the state and observation at each time step
(Figure 5). The observable distribution of each state is typically a mixture of Gaussians.
Sometimes, different states share the same set of Gaussians and are allowed to vary only
the mixing weights (tied mixtures).
For our purposes, the task of learning a mixture of Gaussians given a data set consists
of finding the parameters of a mixture which fits the data well. In order to assess the quality
of this fit, we will consider a simplified situation in which the data is actually generated from
a mixture of Gaussians, which we call the true mixture, and the number of Gaussians is
Introduction 8
, an accuracy parameter;
, a confidence parameter;
wmin , a lower bound on the true mixing weights of the Gaussians; and finally,
A learning algorithm must now return a mixture of k Gaussians, such that with proba-
bility > 1 , the L2 distance between each learned center and the corresponding true
center is at most times the radius of that true Gaussian (defined in the next chapter).
A learning algorithm is considered efficient if it runs in time polynomial in k and 1/,
and polynomial in the dimension of the data.
known. The learned mixture is then evaluated on the basis of its proximity to the truth,
specifically the Euclidean (L2 ) distance between the learned Gaussian centers and the true
centers. This learning scenario, summarized in Figure 6, is inspired by Valiants (1984)
probably approximately correct (PAC) model.
Our learning model for directed probabilistic nets is rather different. It is framed as
a combinatorial optimization problem: given data, find the model with highest likelihood,
that is, the model which assigns the highest probability to the data set. In this case, there
are no assumptions whatever about the data and no notion of a true model. An efficient
learning algorithm can then routinely be defined as one whose time and sample complexity
are polynomial in the number of nodes.
It is always easy to find a good directed probabilistic net which has high connectivity,
but this defeats the purpose of such models. A natural restriction on the complexity of a
net is a bound on the number of parents its nodes are allowed to have. This motivates the
triplet of optimization problems presented in Figure 7. In each case, there are no hidden
nodes, so the number of nodes in the graph equals the number of attributes of the data set.
Introduction 9
SL(k) Find a directed probabilistic net of minimal cost in which each node has at
most k parents.
PT(k) Find a polytree of minimal cost in which each node has at most k parents.
Figure 7: Learning directed probabilistic nets: some optimization problems. In each case,
the cost of a model is its negative log-likelihood.
The cost of any given model is defined as its negative log-likelihood (that is, minus the log
of its likelihood).
3 Current practice
The most popular technique for learning a mixture of Gaussians from data is currently
the expectation-maximization (EM) algorithm. This is a local search heuristic of appealing
simplicity, whose principal goal is convergence to a local maximum in the space of Gaussian
mixtures ranked by likelihood. No performance guarantees of the kind outlined above are
known for this algorithm.
For learning probabilistic nets, a variety of different local search algorithms have been
proposed, including variants of gradient descent and EM. One important subcase stands
out, that of branchings (sometimes called Chow-Liu trees in the literature), directed nets in
which each node has at most one parent. A simple and computationally efficient algorithm
for determining the maximum-likelihood branching (in the absence of hidden nodes) was
discovered by Edmonds in 1967. In our terminology, this means that SL(1) (or equivalently,
PT(1)) is efficiently solvable. To date this remains the central positive result in the literature
on learning the structure of probabilistic nets.
The research that will be presented here is motivated principally by the lack of simple
and provably good algorithms for learning important classes of probability distributions. As
in the two learning frameworks just described, we shall think of a provably good algorithm
as one which is computationally efficient and is accompanied by strong guarantees on the
quality of fit of the learned model. Such guarantees give the user some indication of how
much confidence he can have in the final answer and provide a scale along which different
Introduction 10
algorithms can be meaningfully compared. They are crucial in the construction of complex
probabilistic models which rely upon the correct functioning of each component submodel.
4 A summary of results
Chapters II and III present an efficient learning algorithm, in the sense of Figure 6,
for mixtures of Gaussians in arbitrarily high dimension. There are two restrictions: the
component Gaussians of the true mixture must be well-separated (this notion is made pre-
cise in the next chapter), and they must have a common (though unknown) non-singular
covariance matrix. The running time is linear in the dimension of the data and polynomial
in the number of Gaussians.
This algorithm is very simple and relies crucially upon a particular procedure for
dimensionality reduction. Data from a mixture of k Gaussians are projected to a randomly
chosen O(log k)-dimensional subspace, regardless of the original dimension and the number
of data points, and it is shown that not only does this retain enough information for clus-
tering, but it also causes a drastic reduction in the eccentricity of the Gaussians. This latter
quantity will be defined later; for the time being, it should be interpreted as some natural
measure of how non-spherical a Gaussian looks. Experiments in Chapter II illustrate some
of the promise of this random projection technique.
Can something similar be done when the clusters do not look Gaussian, for instance
if the data is discrete-valued? A celebrated result of Diaconis and Freedman (1984) demon-
strates that many families of distributions, including for instance product distributions on
the hypercube, look more Gaussian when randomly projected to low dimension, in a sense
that we will make precise in Chapter III. Thus the techniques we develop might work for a
fairly broad class of mixture models.
The last two chapters study probabilistic nets in terms of SL and PT. These are NP-
hard optimization problems and therefore the main question is whether good approximation
algorithms exist for them. A c-approximation algorithm for a minimization problem is
defined as one which always returns a solution whose cost is within a multiplicative factor
c of optimal. We have already seen that SL(1) is efficiently solvable. Chapters IV and V
will demonstrate that SL(2) and PT(2) are, however, hard to even approximately solve.
Specifically, there is some constant c > 1 for which c-approximating SL(2) is an NP-hard
optimization problem. A similar result holds for PT(2).
Introduction 11
Chapter II
Mixtures of Gaussians
(x) being the density of the mixture at point x. In other words, the goal is to maximize
the probability density of the data S with respect to , and there are no assumptions about
the data in particular, it is not assumed that the data actually come from a mixture of
Gaussians.
This framework has an appealing generality. A formal justification would ideally
include a finite-sample guarantee of the form: for any mixture , for any > 0, there
exists an m such that the maximum-likelihood mixture of Gaussians with respect to m
i.i.d. samples from will with high probability have centers which are -close to those of ,
under some natural measure of closeness. Unfortunately, this is quite dramatically false.
Given any data points x1 , . . . , xm , consider a mixture b which centers one Gaussian on x1
b 1 ) approaches infinity.
and allows its variance (or covariance matrix) to go to zero. Then (x
In short, given any finite amount of data, the ML solution is ill-defined because a mixture
with arbitrarily high likelihood can be found.
p
Define the radius of a Gaussian N (, ) to be trace() we will soon see the
reasons for this choice. Suppose that during ML learning, some lower bound is placed on
Mixtures of Gaussians 13
the radius of the Gaussians in the mixture. Then the singularities we just encountered will
no longer occur to quite the same alarming extent. Nevertheless, similar difficulties will
continue to be present, since the creation of one (or more) clusters of very small radius can
artificially boost the overall likelihood. It is perhaps fair to say that the ML framework is
inadequate in situations where different clusters may have widely differing radii. However,
this should not be judged too harshly since it is an important open problem in clustering
to develop goodness-of-fit criteria that can satisfactorily handle such cases. The algorithms
presented in this thesis all assume clusters of approximately the same radius.
Open Problem 1 Say m i.i.d. data points are generated from an unknown mixture of
k Gaussians with the same covariance matrix, and then the maximum-likelihood solution
b is found within this same class of mixtures. Can it be shown that with high probability
the Gaussian centers of b will be close in L2 distance to those of ? What bounds can be
given for this error as a function of k and m?
Open Problem 2 What kind of guarantees can be given for EMs performance on finite
data sets drawn i.i.d. from a mixture of Gaussians?
The next chapter describes an algorithm which takes data from a mixture of Gaussians
and then recovers that mixture with high probability, to any specified accuracy. Why is it
at all acceptable to make such a strong distributional assumption about the data? First,
any algorithm with such a performance guarantee must be able to handle a certain amount
of sampling error; thus it can automatically handle component distributions which are close
to Gaussian in a specific sense that we will discuss in Chapter III. Second, the Gaussian is a
distribution which occurs naturally in many data sets, for instance as noise in physical data.
This phenomenon can be explained, after the fact, by appeal to entropy considerations or
Mixtures of Gaussians 14
to the central limit theorem. Third, we will see that many non-Gaussian distributions can
be made to look more Gaussian under a simple form of projection; this is related to the
previous point and gives the Gaussian a certain universality.
For many clustering problems, it is impossible to give algorithms with useful per-
formance guarantees under completely arbitrary conditions. Consider, for instance, the
k-center problem.
k-center
Input: n data points in Rd ; integer k.
Output: k centers 1 , 2 , . . . , k Rd and radius r such that the data set is
contained within B(1 ; r) B(k ; r).
Objective: Minimize r.
It has been shown that there is no polynomial-time algorithm which can consistently find
an r within a multiplicative factor two of optimal, unless P = NP, and that this factor can
be achieved by a simple greedy algorithm (Hochbaum and Shmoys, 1985; Gonzalez, 1985;
Feder and Greene, 1988). This is a very weak guarantee which can be quite unacceptable
for clustering. For instance, suppose each cluster in the data looks like the surface of a
sphere. Then choosing centers on the surfaces of these spheres will yield a value of r
within a factor two of optimal, and yet it should be possible to do much better.
A big challenge in designing an algorithm for a combinatorial optimization problem
is to state the problem in an useful manner. If it is stated in too much generality, then it
might be impossible to concoct an algorithm with any meaningful performance guarantee.
In this case, there may be many algorithms to choose between, but one will not be able to
differentiate between them the goodness criterion is so stringent that they all fail miserably.
One could run simulations or make a few heuristic arguments for specific cases, but overall
it will be hard to make a definitive and satisfactory choice. This may well be the case with
the ML formulation of the mixture-of-Gaussians problem without any assumptions on the
data.
Clustering is a hard problem which is the subject of intensive research. One possible
mode of attack is to alternate exploratory data analysis with algorithm design:
Explore data sets of interest to gain some insights into them. What do clusters
look like?
Using these insights, formalize some assumptions about the data. Design an
algorithm which works well under these assumptions.
Mixtures of Gaussians 15
A reasonable way of starting this process is by designing a clustering algorithm which works
under what is perhaps the most widely used assumption that the data look like a mixture
of Gaussians.
2 Learning algorithms
The reference book of Titterington, Smith and Makov (1985) contains a brief history
of the many fascinating and idiosyncratic techniques that have been applied to learning
mixtures of Gaussians. These range from manual univariate procedures requiring graph
paper and sharpened pencils to more sophisticated numerical methods based upon the
manipulation of empirical Fourier coefficients of the data. Further details and more recent
advances can be found in an excellent survey article by Redner and Walker (1984) and in a
monograph by Lindsay (1995). Among the various algorithms, two of the most prominent
are the method of moments and EM.
The theory of mixture models is often thought to have started with Pearsons (1894)
clustering of crabs. The data were collected from 1000 crabs in Naples. For each crab, the
ratio of the forehead to the body length was recorded. Pearson wanted to divide these
measurements into two clusters, and did so by fitting a mixture of two univariate Gaussians
to the data.
The technique he used has been widely studied since, and is called the method of
moments. Pick a function t(), and compute its expectation under the empirical distribution
of data points x1 , . . . , xm ,
b = t(x1 ) + + t(xm ) .
Et
m
b Often a
Now choose a model for which the expectation of t under is equal to Et.
number of different functions ti are chosen and the fit need not be exact.
Pearson chose the first few moment functions ti (x) = xi . He reduced the resulting
equivalences to a single nonic equation whose solution would yield a mixture of Gaussians.
In scaling this technique to mixtures with more parameters, a few considerations are
important. First, the functions t() should not have high variance, otherwise their empirical
estimates are unreliable. This can sometimes rule out high-order moments, suggesting
Mixtures of Gaussians 16
instead the zero-order moments tA (x) = 1(x A), for sets A. It seems difficult to find a
set of moments to use with mixtures of Gaussians which will make the necessary equation-
solving easy to automate. Perhaps this is why it appears to have died out as a viable
algorithm, despite its intuitive appeal.
2.2 Expectation-maximization
The principal goal of the EM algorithm (Dempster, Laird and Rubin, 1977; Redner
and Walker, 1984; Xu and Jordan, 1996) when applied to our learning problem is to find a
local maximum in the space of Gaussian mixtures ranked by likelihood. We have already
seen that the global optimum is not necessarily desirable, which puts the whole enterprise
in question. Nonetheless EM has an appealing simplicity which, together with a marked
absence of significantly better alternatives, has made it the current algorithm of choice for
learning mixtures of Gaussians.
We will briefly sketch the algorithm. It starts by guessing some values for the mixture
parameters, and then alternates between two stages.
Stage E: Taking the current guess as truth, compute, for each data point and
each Gaussian, the probability that the point came from that Gaussian. This
gives a soft clustering of the data set each point is not definitively assigned
to a Gaussian, but is split between all of them in suitable proportion.
Stage M: Based on this interim clustering, update the estimates of the param-
eters.
As this process continues, the log-likelihood of the estimated mixture increases monotoni-
cally. A possible stopping point is when the increase becomes very small, although of course
this could just be a misleading lull.
One issue that has received a fair amount of attention is how the initial parameters
should be chosen, particularly the initial Gaussian centers (for instance, Meila and Hecker-
man, 1998). Viable options for the initial centers are: (i) choose a few of the data points
at random; (ii) use the results of k-means, a quicker clustering heuristic; or (iii) find the
bounding box of the data and choose some points uniformly at random from this box.
Mixtures of Gaussians 17
3 High-dimensional Gaussians
If is a multiple of the identity matrix, then the Gaussian is called spherical. Some
important intuition about the behavior of Gaussians in high dimension can quickly be
gained by examining the spherical Gaussian N (0, 2 In ). Although its density is highest at
the origin, it turns out that for large n most of the probability mass lies far away from this
center. This is the first of many surprises that high-dimensional space will spring upon us.
A point X Rn chosen randomly from this Gaussian has coordinates Xi which are i.i.d.
P
N (0, 2 ). Therefore its expected squared Euclidean norm is E(kXk2 ) = i EXi2 = n 2 .
In fact, it can be shown quite routinely, by writing out the moment-generating function of
kXk2 , that the distribution of kXk2 will be tightly concentrated around its expected value.
Specifically,
2 /24
P(|kXk2 2 n| > 2 n) 2en .
That is to say, for big enough n, almost the entire distribution lies in a thin shell of radius
approximately n. Thus the natural scale of this Gaussian is in units of n.
This effect might arouse some initial skepticism because it is not observable in one or
two dimensions. But it can perhaps be made more plausible by the following explanation.
2
The Gaussian N (0, In ) assigns density proportional to e n/2 to points on the surface of
the sphere centered at the origin and of radius n, 1. But the surface area of this
sphere is proportional to ( n)n1 . For large n, as 1, this surface area is growing much
faster than the density is decaying, and thus most of the probability mass lies at distance
about n from the origin. Figure 1 is a graphical depiction of this effect for various values
of n.
The more general Gaussian N (0, ) has ellipsoidal contours of equal density. Each
such ellipsoid is of the form {x : xT 1 x = r 2 }, corresponding to points at a fixed Maha-
lanobis distance kxk = xT 1 x from the center of the Gaussian. The principal axes of
any of these ellipsoids are given by the eigenvectors of . The radius along a particular axis
is proportional to the square root of the corresponding eigenvalue. Denote the eigenvalues by
Mixtures of Gaussians 18
0.8 0.45
0.4
0.7 n=1 n=2
0.35
0.6
0.3
0.5 0.25
0.4 0.2
0.15
0.3
0.1
0.2
0.05
0.1 0
0 0.5 1 1.5 2 0 0.5 1 1.5 2
Radius r from center, in units of sqrt(n) Radius r from center, in units of sqrt(n)
5 13
x 10 x 10
2 2.5
n = 10 n = 20
2
Probability mass at radius r
Probability mass at radius r
1.5
1.5
1
1
0.5
0.5
0 0
0 0.5 1 1.5 2 0 0.5 1 1.5 2
Radius from center, in units of sqrt(n) Radius r from center, in units of sqrt(n)
Figure 1: Probability mass of N (0, In ) at radius r n from the origin, for various n.
Mixtures of Gaussians 19
4 Dimensionality reduction
4.1 Overview
The naive mode-finding algorithm we just considered requires at least about 2d data
points to learn a mixture of Gaussians in Rd . Is it possible to reduce the dimension of the
data so dramatically that this requirement actually becomes reasonable?
Dimensionality reduction has been the subject of keen study for the past few decades,
and instead of trying to summarize this work we will focus upon two very popular contem-
porary techniques: principal component analysis (PCA) and random projection. They are
both designed for data with Euclidean (L2 ) interpoint distances and are both achieved via
linear mappings.
The linear projection of a Gaussian remains a Gaussian. Therefore, projecting a mix-
ture of high-dimensional Gaussians onto a single line will produce a mixture of univariate
Gaussians. However, these projected clusters might be so close together as to be indistin-
guishable. The main question then is, how much can the dimension be reduced while still
maintaining a reasonable amount of separation between different clusters?
Mixtures of Gaussians 21
Principal component analysis is an extremely important tool for data analysis which
has found use in many experimental and theoretical studies. It finds a d-dimensional sub-
space of Rn which captures as much of the variation in the data set as possible. Specifically,
given data S = {x1 , . . . , xm } it finds the linear projection to Rd for which
m
X
kxi k2
i=1
is maximized, where xi is the projection of point xi and is the mean of the projected
data.
This projection is quite easy to obtain. Let , denote the mean and covariance of
the high-dimensional data S. The positive semidefinite matrix can be written in the form
B T DB, where D = diag(1 , . . . , n ) is a diagonal matrix containing the eigenvalues of ,
and B is an orthogonal n n matrix (that is, B T B = BB T = In ). If the data points are
rotated by B, the resulting data Bx1 , . . . , Bxm has mean B and covariance
m
1 X
(Bxi B)(Bxi B)T = BB T = D.
m
i=1
A much faster technique for dimensionality reduction, which has received a warm
welcome in the theoretical computer science community, is expressed in the following lemma.
Lemma 1 (Johnson-Lindenstrauss, 1984) Fix some 0 < < 1 and 0 < < 1. Pick
any m data points x1 , . . . , xm Rn . If these points are projected into a randomly chosen
12 2
subspace of dimension d 2
ln m then with probability > 1 , the projected points
x1 , . . . , xm satisfy
d kxi xj k2 d
(1 ) (1 + ) for all i 6= j.
n kxi xj k2 n
This remarkable lemma asserts, roughly, that any m data points in arbitrarily high
dimension can be linearly mapped into an O(log m)-dimensional subspace in such a way
that pairwise Euclidean distances between the points are only slightly distorted. Moreover,
this particular subspace is not hard to choose; a random subspace will with high probability
(> 1 m(1) ) do the job. The original proof was simplified by Frankl and Maehara (1988),
and a yet more elementary version, the result of joint work with Gupta, is presented in
Appendix A.3.
The choice of random projection matrix does not depend upon the data in any way,
and so can be accomplished quickly, in time O(d2 n). This technique is therefore a very
efficient generic means of coping with high dimensionality. However, for our purposes this
reduced dimension is still far too high. The rough heuristic outlined above needs 2d data
points, and this exceeds m by many orders of magnitude.
Later in this chapter we will see that for the particular case of mixtures of Gaussians,
we can reduce the dimension of the data far more drastically than in the generic bound given
above. By using projection to a randomly chosen subspace as in the Johnson-Lindenstrauss
lemma, we can map the data into just d = O(log k) dimensions, where k is the number of
Gaussians. Therefore the amount of data we will need is only polynomial in k.
This might puzzle readers who are familiar with random projection, because the usual
motive behind such projections is to approximately preserve relative distances between
data points. However, in our situation we expressly do not want this. We want some
Mixtures of Gaussians 23
of the pairwise distances to contract significantly, so that the fraction of points within
distance d of any Gaussian center in the reduced space Rd is much greater than the
fraction of points within distance n of the same center in the original space Rn . At
the same time, we do not want the distances between different Gaussians to contract; we
must make sure that Gaussians which are well-separated remain so when they are projected.
These conflicting requirements are accommodated admirably by a projection to just O(log k)
dimensions.
Definition For a positive definite matrix , let max () and min () refer to its largest
and smallest eigenvalues, respectively, and denote by e() the eccentricity of the matrix,
p
that is, max ()/min ().
Proof. Consider a single line segment in Rn , of squared length L. If the original space is
projected onto a random d-dimensional subspace, the squared length of this line segment
becomes some L , of expected value EL = Ld/n. The proof of the Johnson-Lindenstrauss
2
lemma (Appendix A.3) shows that P(L < (1 )Ld/n) ed /4 .
Apply this bound to the k2 line segments joining pairs of Gaussian centers in the
original space. Then
p k d2 /4
P i 6= j : ki j k < ki j k (1 )d/n e .
2
This keeps the centers far apart; to satisfy our definition of separatedness, we must also check
that the original Gaussians do not spread out when projected, that is, max (i ) max (i ).
Mixtures of Gaussians 24
.
. . .
.
. . . . .
. . . .
. . . .
. . .
. . .
. . .
. .
. . . .
. .
.
. . . .
. . . .
. .
. . . .
. . .
. . . .
. . .
. .
. . . .
Figure 2: The effects of random projection: the dimension is drastically reduced while the
clusters remain well-separated and become more spherical.
uT (P T P )u (P T v)T (P T P )(P T v)
max (P T P ) = max = maxn
uRd uT u vR (P T v)T (P T v)
(P P T v)T (P P T v)
= maxn
vR (P P T v)T (P P T v)
v T v
maxn T = max ().
vR v v
The denominator in the second line uses P T P = Id . In similar fashion we can show that
min (i ) min (i ), completing the proof.
This method of projection has another tremendous benefit: we show that even if
the original Gaussians are highly skewed (have ellipsoidal contours of high eccentricity),
their projected counterparts will be more spherical (Figure 2). Since it is conceptually
much easier to design algorithms for spherical clusters than ellipsoidal ones, this feature of
random projection can be expected to simplify the learning of the projected mixture.
Consider a Gaussian N (0, ) in Rn , and think of the random projection from Rn to
Rd as a random rotation in Rn , represented by some orthogonal matrix U T , followed by a
projection P T onto the first d coordinates. The columns of U T are an orthonormal basis
{u1 , . . . , un } of Rn . Denote the restriction of these vectors to their first d coordinates by
u1 , . . . , un , respectively. The high-dimensional covariance matrix has some eigenvalues
p
1 n , with eccentricity e = n /1 1, and normalized trace = n1 (1 + +n ).
It will turn out that under suitable conditions on e the covariance matrix of the projected
Mixtures of Gaussians 25
Lemma 3 (Variance of a projected Gaussian) For any unit vector x Rd , V (x) has
P
the same distribution as ni=1 i vi2 , where v is chosen uniformly at random from the surface
of the unit sphere in Rn . Therefore EV (x) = , over the choice of random projection.
whereby
n
X
V (x) = (P x)i (P x)j (U T U )ij
i,j=1
d
X n
X
= xi xj k Uki Ukj
i,j=1 k=1
n
X d
X
= k (xi Uki )(xj Ukj )
k=1 i,j=1
n
X
= k (x uk )2 ,
k=1
Since UT is a random orthogonal matrix, its first row (u11 , . . . , un1 ) is a random unit vector.
We now have a simple formulation of the distribution of V (x). For any given x, this
value is likely to be close to its expectation because it is the sum of n almost-independent
Mixtures of Gaussians 26
Lemma 4 (Eccentricity reduction) There is a universal constant C such that for any
0 < 1 and 0 < < 1, if the original dimension satisfies n > C e2 (log 1 + d log d ),
2
then with probability > 1 over the choice of random projection, the eccentricity e of
the projected covariance matrix will be at most 1 + . In particular, if the high-dimensional
1
eccentricity e is at most n1/2 C 1/2 (log + d log d)1/2 then with probability at least 1 ,
the projected Gaussians will have eccentricity e 2.
Moreover, if x and y are unit vectors which are close to each other then V (y) cannot
differ too much from V (x):
n
X
|V (x) V (y)| i (ui x)2 (ui y)2
i=1
n
X
= i |ui (x y)| |ui (x + y)|
i=1
n
X
i kui k2 kx + yk kx yk
i=1
n
!
X
2 kx yk i kui k2 .
i=1
The third line follows by the Cauchy-Schwarz inequality and the fourth uses the fact that
x and y are unit vectors. The final parenthesized quantity has expected value d (each ui
consists of the first d coordinates of a random unit vector in Rn and therefore Ekui k2 = nd ).
The chance that it exceeds 2d is at most de(n/e ) , by Lemma A.5. Choosing kx yk
2
24d will then ensure |V (x) V (y)| 61 .
Bounding V (x) effectively bounds V (y) for y B(x; 24d ). How many points x must
be chosen to cover the unit sphere in this way? It turns out that ( 24d d
) points will do the
trick, for some constant > 0. This last result is shown by considering the dual question,
Mixtures of Gaussians 27
how many non-overlapping spherical caps of fixed radius can be packed onto the surface of
a sphere? An upper bound can be obtained by a surface area calculation (Gupta, 1999).
Let S denote this finite cover; then
Random projection offers many clear benefits over principal component analysis. As
explained earlier, PCA cannot in general be used to reduce the dimension of a mixture of
k Gaussians to below (k), whereas random projection can reduce the dimension to just
O(log k). Moreover, PCA may not reduce the eccentricity of Gaussians. However, if a
projection to k dimensions is acceptable, then a PCA-projected mixture could easily be
far better separated than a randomly projected mixture. For this reason PCA remains an
important tool in the study of Gaussian clusters.
The lemmas of the previous section can be illustrated by a few simple experiments.
The first two of these examine what happens when a 1-separated mixture of k Gaussians in
Rn is randomly projected into Rd . The main question is, in order to achieve a fixed level of
separation in the projected mixture, what value of d must be chosen? How will this d vary
with n and k?
The first experiment, depicted in Figure 3, is intended to demonstrate that d does not
depend upon n, that is, the projected dimension is independent of the original dimension
of the data. Here two 1-separated spherical Gaussians are projected into R20 and their
separation is noted as a function of n. The error bars are for one standard deviation in
either direction; there are 40 trials per value of n.
The second series of tests (Figure 4) randomly projects 1-separated mixtures of k
spherical Gaussians in R100 into d = 10 ln k dimensions, and then notes the separation of
the resulting mixtures. The results directly corroborate Lemma 2. The mixtures created for
these tests are maximally packed, that is, each pair of constituent Gaussians is 1-separated.
There are 40 measurements taken for each value of k.
Mixtures of Gaussians 28
0.5
0
0 200 400 600 800 1000 1200
Figure 3: The projected dimension (d = 20) does not depend upon the original dimension.
0.9
0.8
Separation of projected mixture
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
0 20 40 60 80 100 120
Number of Gaussians in mixture (k)
Eccentricity e n
in Rn 25 50 75 100 200
50 9.5 3.80 3.4 0.62 2.5 0.29 2.2 0.17 1.7 0.07
100 13.1 5.79 3.5 0.57 2.5 0.26 2.2 0.19 1.7 0.08
150 13.0 7.40 3.5 0.55 2.5 0.25 2.2 0.14 1.7 0.07
200 14.7 8.04 3.4 0.50 2.5 0.22 2.2 0.19 1.7 0.06
The last two experiments, shown in Figures 5 and 6, document the dramatic decrease
in eccentricity that accompanies the random projection of a Gaussian. The first of these
projects a Gaussian of high eccentricity e from Rn into R20 and measures the eccentricity
e of the projection, over a range of values of e and n. Here matrices of eccentricity e are
constructed by sampling the square roots of their eigenvalues uniformly from the range [1, e],
and making sure to include the endpoints 1 and e. The last experiment fixes a particular
Gaussian in R50 of enormous eccentricity e = 1000, and then projects this Gaussian into
successively lower dimensions 49, 48, 47, . . . , 25. Notice that the y-axis of the graph has a
logarithmic scale. Also, the error bars no longer represent standard deviations but in fact
span the maximum and minimum eccentricities observed over 40 trials per value of d.
Mixtures of Gaussians 30
2
10
1
10
0
10
20 25 30 35 40 45 50
Chapter III
1 Overview
In this chapter we will examine a simple algorithm which learns an unknown mix-
ture of Gaussians with arbitrary common covariance matrix and arbitrary mixing weights,
in time which scales only linearly with dimension and polynomially with the number of
Gaussians. With high probability, it learns the true centers of the Gaussians to within
the precision specified by the user. Previous heuristics have been unable to offer any such
performance guarantee, even for highly restricted subcases like mixtures of two Gaussians.
The learning model was described in the introductory chapter (Figure I.6). The user
furnishes: a data set S consisting of (say) m data points in Rn ; , the accuracy within which
the centers are to be learned; , a confidence parameter; k, the number of Gaussians; and
wmin , the smallest mixing weight that will be considered.
The algorithm, shown in Figure 1, works in four phases and is very simple to im-
plement. Some brief intuition about its functioning will hopefully ease the way for the
subsequent formal analysis.
The data are projected from Rn to Rd via a linear map. Since any linear transfor-
mation of a Gaussian conveniently remains a Gaussian, we can pretend that the projected
data themselves come from a mixture of low-dimensional Gaussians.
The second step of the algorithm estimates the means of these projected Gaussians.
Learning mixtures of Gaussians 32
For each x S , compute rx , the smallest radius such that there are p
points within distance rx of x.
Start with S = S .
For i = 1 . . . k:
Regions of higher density will tend to contain more points, and we can roughly imagine the
density around any data point x to be inversely related to the radius rx . In particular, the
point with lowest rx will be near the center of some (projected) Gaussian. If the Gaussians
all share the same covariance, then this data point will most likely be close to the center of
the Gaussian with highest mixing weight.
Once we have a good estimate for the center of one Gaussian, how do we handle the
rest of them? The problem is that one Gaussian may be responsible for the bulk of the data
if it has a particularly high mixing weight. All the data points with low rx might come from
this one over-represented Gaussian, and need to be eliminated from consideration somehow.
This is done by growing a wide region around the estimated center, and removing
from contention all the points in it. The region should be large enough to remove all high-
density points in that particular Gaussian, but should at the same time leave intact the
high-density points of other Gaussians. The reader may wonder, how can we possibly know
how large this region should be if we have no idea of either the covariance or the mixing
weights? First, we pick the q points closest to the estimated center rather than using a
preset radius; this accomplishes a natural scaling. Second, the probability of encountering
a data point at a distance r from the center of the Gaussian grows exponentially with
r, and this rapid growth tends to eclipse discrepancies of mixing weight and directional
variance.
Both the techniques described that of choosing the point with the next lowest rx
as a center-estimate, and then subtracting the points close to it rely heavily on the
accuracy of spherical density estimates. That is, they assume that for any sphere in Rd ,
the number of data points which fall within that sphere is close to its expected value under
the mixture distribution. That this is in fact the case follows from the happy circumstance
that the concept class of spheres in Rd has VC dimension only d + 1.
At this stage, projected centers in hand, we recall that our actual task was to find the
Gaussian means in the original high-dimensional space. This is not too difficult conceptually.
bi , we pick the l data points closest to it in Rd ,
For each low-dimensional estimated center
call them Si , and then average these same points in Rn . We expect Si to be relatively
uncontaminated with points from other Gaussians (although we cannot of course avoid the
Learning mixtures of Gaussians 34
The estimates obtained in the second and third phases of the algorithm will be very
crude, of much lower precision than the user demands. Nevertheless, it can be shown that
assigning each point in S to its closest high-dimensional center-estimate
bi will cluster the
data almost perfectly. Computing statistics of each cluster will then yield sharp estimates
of its mean, covariance, and mixing weight.
How exactly did the projection help us? It enabled us to find, for each Gaussian, a
set of data points drawn mostly from that Gaussian.
Theorem 1 In what follows, C1 , C2 , and C3 are universal constants. Suppose data is drawn
from a mixture of k Gaussians in Rn which is 1-separated, has smallest mixing weight wmin ,
and has some common (unknown) covariance matrix . If the user specifies confidence
and accuracy parameters , (0, 1) respectively, then the algorithm will set the reduced
1
dimension to some d = C1 log wmin .
2
Let max denote the maximum eigenvalue of and let e denote its eccentricity. If
(1) e C2 n1/2 (d log d)1/2 , (2) n C2 (d + log 1 ) and (3) the number of data points is
1 C3
m ( wmin ) , then with probability > 1 , the center estimates returned by the algorithm
will be accurate within L2 distance max n.
The algorithm runs in time O(m2 d + mdn), plus an additional O(mn2 ) if an estimate
of the covariance matrix is sought.
This algorithm can in fact efficiently handle Gaussians whose separation is an arbi-
trarily small constant c > 0. It is only to curtail the proliferation of symbols that we insist
upon 1-separation in this theorem. A similar comment applies to the eccentricity.
Learning mixtures of Gaussians 35
1.5 Notation
The following battery of notation will be used consistently through the chapter.
, Accuracy and confidence, supplied by user
0 Accuracy of spherical density estimates
m Overall number of data points
n Original dimension of data
d Reduced dimension
k Number of Gaussians
wi N (i , ) A mixture component (Gaussian) in Rn
wmin Lower bound on the wi , supplied by user
cij Separation between ith and j th Gaussians in Rn
c mini6=j cij
cij , c Separation of Gaussians in Rd
wi N (i , ) Projection of ith Gaussian into Rd
() Density of the entire projected mixture
B(x; r) Sphere of radius r centered at x
B(r ; r) B(x; r) for some x with kxk = r
l, p, q Integer parameters needed by algorithm
The accuracy of phases two and three (a small constant)
S Data set in Rn
S Data
p set in p Rd
max , min max (), min ()
e Eccentricity max /min
max , e
, min Similar, but in the projected space
() N (0, Id )
() N (0, )
T A useful linear transformation in Rd
k k Mahalanobis distance, kxk = xT 1 x
E(z; r; ) Ellipsoid {x : kx zk r}
As we have already seen, we can think of e as a small constant even if e is large,
and this will help us tremendously.
Learning mixtures of Gaussians 36
2 Projection
Theorem 2 The first phase of the algorithm takes as input (1) a data set S Rn and (2)
k, the number of Gaussians (or an upper bound on it). It returns a projection of the data
set, S . If cij denotes the separation between the ith and j th projected Gaussians, and e is
the eccentricity of the projected covariance matrix,
64 2
if d 9 log k then with probability at least 1 , for all i 6= j, cij 12 cij ; and
3 Low-dimensional clustering
The second phase of the algorithm, which learns the centers of Gaussians in low
dimension, is ad hoc though correct. The technical tools used in its analysis might be
helpful in developing other similar routines, so we start with a brief overview of them.
The first question is, in what way is the Gaussian nature of the data being used?
Using VC bounds, it can be shown that with just a modest number of samples, all spheres
in Rd will contain roughly the expected number of points of under the mixture distribution.
This is a convenient guarantee; we need no other control on the sampling error. More
formally, the clustering algorithm will work under a
Weak Gaussian assumption For each sphere in Rd , the fraction of S that falls in
this sphere is the expected fraction under the mixture distribution, 0 , where 0 reflects
sampling error (and is in our case proportional to m1/2 ). The class of spheres may be
substituted by some other class of small VC dimension.
Learning mixtures of Gaussians 37
A
B
Figure 2: A sphere near the center of a Gaussian has higher probability mass than a sphere
further away.
Strong Gaussian assumption The data in S are drawn i.i.d. from the projected mixture
of Gaussians.
The latter enables us to conclude, for instance, that if d is large and S consists
of just two data points, then with high probability these two points are not close to one
another. This does not follow from the weak assumption; in fact, the sampling error is so
high for such a tiny data set that no non-trivial conclusions whatsoever can be drawn. The
deliberate use of such a restriction is not mere conceit; it will turn out to be important for
some of the extensions proposed later.
The weak Gaussian assumption forces us to examine the relative probability masses
of different spherical regions in Rd . Assume that the Gaussians are spherical. Each point x
in the sample is assigned a radius rx , and we hope that points with low rx will be close to
the centers of the Gaussians. In order to prove this, we must show that in cases such as that
depicted in Figure 2 (where the outer sphere conceptually denotes a Gaussian), sphere A
has a significantly higher probability mass than sphere B, which has the same radius but is
further from the center of the Gaussian. This can be shown easily by a pointwise coupling
of A and B.
bi has
Next, assume the Gaussians are spherical with unit variance. Once a center
been chosen, we will eliminate the q points closest to it, where q is the number of points
expected to fall within B(i ; 83 d), assuming a mixing weight of wmin . It turns out that
bi is a reasonably accurate estimate of i then whatever wi might actually be, this
if
process will eliminate all points in B(i ; 41 d) and nothing that does not lie in B(i ; 21 d).
In effect, it eliminates all the high-density points in the ith Gaussian while leaving intact
Learning mixtures of Gaussians 38
Sphere in Ellipse in a
an ellipsoidal spherical
Gaussian Gaussian
Linear
transformation
or by the equivalent
By switching between these two options, we will avoid various unpleasant dependencies
between the data set and the estimated centers.
Learning mixtures of Gaussians 39
3.2 Summary
Theorem 3 The second phase of the algorithm takes as input a (projected) data set S Rd
1
which satisfies the weak Gaussian assumption with error 0 with respect to a 2 -separated
mixture of Gaussians {(wi , i , )}. The other inputs are: k, the number of Gaussians; the
eccentricity e , or some upper bound on it; an accuracy parameter > 0; and an integer l.
The output of this phase is a set of k subsets {S1 , . . . , Sk } S , each of size l.
The quantity should be thought of as a small constant which is typically unrelated
to the final precision required by the user. If the following conditions are met,
l mwmin ( 12 )d m0 , and
30 2
l wmin ln k ,
then with probability at least 1 , each Si is contained within B(i ; max
d) and the
lwj
number of points in Si from the j th Gaussian Gj , j 6= i, is 10cij .
Our algorithm relies heavily upon the hope that in the projected space, every spherical
region will contain roughly its expected number of points under the mixture distribution.
This can shown effortlessly by a VC dimension argument.
Proof. For any closed ball B Rd , let 1B (x) = 1(x B) denote the indicator function for
B. The concept class {1B : B Rd is a sphere} has VC dimension d + 1 (Dudley, 1979).
The rest follows from well-known results about sample complexity; details can be found,
for instance, in an article by Haussler (1992) or in the book by Pach and Agarwal (1995).
We will henceforth assume that m meets the conditions of this lemma and that
all spherical density estimates are accurate within 0 . The next problem we face is that
because Gaussians in general have ellipsoidal contours, it is not easy to get tight bounds on
the probability mass of a given spherical region. We will content ourselves with rather loose
bounds, obtained via the mediation of a linear transformation T which converts ellipsoids
into spheres.
Write the d d covariance matrix as B T DB, where B is orthogonal and D is
diagonal with the eigenvalues of as entries. Define T = B T D1/2 B; notice that T is its
own transpose. The table below hints at the uses to which T shall be put.
Our first step will be to relate the density of the ellipsoidal Gaussian N (0, ) to
the more manageable density of N (0, Id ).
Proof. This is easy if T is used appropriately. For instance, since E(z; r ; ) B(z; r)
max
we can write
Similarly we can bound the relative densities of displaced spheres. Consider two
spheres of equal radius r, one close to the center of the Gaussian, at Mahalanobis distance
s, and the other at some distance s + . By how much must the probability mass of the
closer sphere exceed that of the farther one, given that they may lie in different directions
Learning mixtures of Gaussians 41
from the center? Although the spheres have equal radius, it might be the case that the
closer sphere lies in a direction of higher variance than the farther sphere, in which case its
radius is effectively scaled down. The following lemma gives a bound that will work for all
spatial configurations of the spheres.
Lemma 6 Pick any point z and set s = kzk . If kz k s + for some > 0 and if
radius r smax then
(B(z; r)) ( + 2s)( 2se )
exp .
(B(z ; r)) 2
Proof. We will use the fact that Mahalanobis distance satisfies the triangle inequality and
. For any point x in B(z; r),
that kuk kuk/min
r
kxk kzk + kx zk s +
min s + se ,
where the last inequality follows from our restriction on r. Similarly, for any point x in
B(z ; r),
kx k kz k kx z k s(e 1).
Since (y) is proportional to exp(kyk2 /2) for any point y, the ratio of probabilities of
the two spheres must be at least
e(s(1+e )) /2
2
( 2se )( + 2s)
= exp ,
e(s(e 1)) /2
2
2
as anticipated.
Finally we need a bound on the rate at which the probability mass of a sphere, under
distribution , grows as its radius increases.
Lemma 7 If radii r and s satisfy r + s 21 min
d then
d/2
(B(0; r + s)) r+s
.
(B(0; r)) r
We will bound this ratio of integrals by considering a pointwise ratio. For any y B(0; r+s),
we know kyk (r + s)/min and so
(y) kyk2 r2
r = exp 1
(y r+s ) 2 (r + s)2
(r + s)2 r 2
exp 2
2min
d/2
r
,
r+s
Proof. Let Vd denote the volume of the unit ball in d dimensions. We will use the lower
d/2 (2)d/2
bound Vd = (1+d/2) 2(d/2)d/2
which follows from the observation (1+k) kk 2(k1) for
k 1. Now center a sphere at the mean of the Gaussian. A crude bound on its probability
mass is
2
e( d) /2 d
(B(0; d)) V d ( d) d .
(2)d/2
Continuing in the same vein, this time for a displaced sphere, we get bound (b).
We are now in a position to prove that for an appropriate choice of the parameters
p and q, the algorithm will find one data point close to each projected center, within
Mahalanobis distance 12 d. This value should be thought of as a small constant which is
unrelated to (and typically much larger than) the precision demanded by the user. Denote
by i the means of the projected Gaussians and by their common covariance matrix.
Let be the density of the projected mixture.
Parameter settings The parameters of the algorithm will be set to the following easily
computable values,
p = m(wmin ( 8e )d 0 ), and
Learning mixtures of Gaussians 43
q = mwmin (B(0; 8e3 d)).
Lemma 9 There is at least one data point within Mahalanobis distance 8e d of each
center. Any such point x has at least p data points close by, in B(x; ( 8e )max
d), and
thus rx ( 8e )max
d.
Proof. Since all the density estimates are accurate within 0 , we need only show that
wmin (E(0; ( 8e ) d; )) 0 and that wmin (B(x; ( 8e )max
p
d)) m + 0 for
points x with kxk ( 8e ) d. Transformation T and Lemma 5 convert statements about
into statements about , in particular,
(E(0; ( 8e ) d; )) = (B(0; ( 8e ) d)) and
(B(x; ( 8e )max
d)) (B(( 8e ) d; ( 8e ) d)).
By Lemma 8 it is enough to check that wmin ( 8e )d p
m + 0 . This follows from the choice
of p.
This lemma gives an upper bound on rx for points x close to a center. We next need
to show that rx will be significantly larger for points further away, at Mahalanobis distance
12 d from the center.
Lemma 10 Suppose rx ( 8e )max d for some point x which is at Mahalanobis distance
1
2 d from the closest center i and at L2 distance 4e1 min
d from all other centers.
1 1 1
Then, if d 38e4 ln wmin 2 and min{ 16e , 8e2 }, any point z within Mahalanobis
distance 8e d of i will have rz < rx .
wi (B(z i ; rx )) exp( 12 ( 12
+ 8e )( 14 8e )d)
+ (B(z i ; rx )) exp( 12 ( 4e12 + 8e )( 4e12 8e 41 )d)
wi (B(z i ; rx ))F
(B(z; rx ))F
Learning mixtures of Gaussians 44
Lemma 10 implies roughly that within any Gaussian, the lowest rx values come from
data points which are within distance 21 d of the center.
A potential problem is that a few of the Gaussians might have much higher mixing
weights than the rest and consequently have a monopoly over small rx values. In order to
handle this, after selecting a center-estimate we eliminate the q points closest to it, and
guarantee that this knocks out the high-density points near the current center while leaving
intact the high-density regions near other centers.
q
Proof. Rewriting m as wmin (E(0; 8e3 d; )), we notice that
q
wmin (B(0; 8e3 min
d)) wmin (B(0; 8e3 max
d)).
m
For the first inclusion of the lemma, it is enough to show that B(x; ( 4e1 + e2 )min
d),
1
which contains B(i ; 4e min d), has fewer than q points in it, or more specifically that
q
(B(x; ( 4e1 + e2 )min
d)) 0
m
The left-hand side of this expression is maximized by choosing x = i and wi = 1. Moreover
q
the choice of 0 , q forces 0 2m ; therefore it is enough to check that
(B(0; ( 4e1 + e2 )min
d)) 21 wmin (B(0; 8e3 min
d)),
2
which is a direct consequence of Lemma 7 if 8e12 and d 11 ln wmin . The second half
of the proof is executed similarly, and requires 161e and d 10.
Proof. This will proceed by induction on the number of centers selected so far.
Referring back to the algorithm, the first center-estimate chosen is the point x S
with lowest rx . By Lemma 9, this rx ( 8e )max
d. Let i be the projected center closest
to x. Since the Gaussians are 21 -separated, x is at distance at least 14 min
d from all the
other projected centers. By Lemma 10, we then see that x must be within Mahalanobis
distance 21 d of i .
b have already been chosen,
Say that at some stage in the algorithm, center-estimates C
b 1, and that these correspond to true centers C. Select any y C;
|C| b by the induction
hypothesis there is a j for which ky j k 21 d. S does not contain the q points
closest to y. By Lemma 11, this removes B(j ; 4e1 min d) from S , yet no point outside
B(j ; ( 2e1 8e )max
d) is eliminated from S on account of y.
Let z be the next point chosen, and let i be the center closest to it which is not
in C. We have seen that z must be at distance at least 4e1 min
d from centers in C.
Because of the separation of the mixture, z must be at distance at least 41 min
d from all
other centers except possibly i . Again due to the separation of the Gaussians, all points
within distance ( 8e )max
d of i remain in S , and therefore z is potentially one of these,
whereupon, by Lemma 9, rz ( 8e )max d. By Lemma 10 then, kz i k 21 d.
Lemma 13 Under the assumptions of Theorem 3, with probability at least 1 , for all
i 6= j,
lij wj
.
l 10cij
Learning mixtures of Gaussians 46
Proof. Pick any x Si and any j = 6 i. We know kx i k max
d and therefore
kx j k cij max
d max d. Then, applying the general principle that kuk
max
kuk
kuk
min , we can conclude
kx i k e d and kx j k (cij ) d.
Therefore
2
P(x comes from Gj ) wj e(cij ) d/2 wj 3c2
e ij d/8
wi e(e )2 d/2
P(x comes from Gi ) wmin
1
since 16e while cij 12 . Under the assumption d 16 ln w10
min
+ 8e , and using the simple
x wj 3cij /2
inequality ln x e (for x 0), we find that P(x comes from Gj ) is at most 10cij e .
We can now use a Chernoff inequality (Appendix A.1) to bound the chance that lij is very
30 2
much more than l times this probability. Specifically, since by assumption l wmin ln k ,
!
lij wj
P > elwj /30 2 ,
l 10cij k
as claimed.
4 High-dimensional reconstruction
4.1 Summary
Theorem 14 The third phase of the algorithm takes as input the sets Si Rd of size l,
bi of the corresponding sets Si Rn . If the following conditions
and returns the averages
are satisfied:
Si B(i ; max
d) for all i, for some 41 ,
lij wj
the proportion of Si from a different Gaussian Gj , j 6= i, is l 10cij ,
n 16 log k , and
d n2 , cij 12 cij 21 ,
4.2 Details
So far we have data points Si near each of the projected means. Let Si correspond
bi in Rn is the
to these same points in the original space. The high-dimensional estimate
mean of Si .
The random projection from Rn to Rd can be thought of as a composition of two
linear transformations: a random rotation in Rn followed by a projection onto the first d
coordinates. Since rotations preserve L2 distance, we can assume, for the purpose of bound-
ing the L2 accuracy of our high-dimensional center-estimates, that the random projection
consists solely of a mapping onto the first d coordinates. We will write high-dimensional
points in the form (x, y) Rd Rnd , and will assume that each such point is projected
down to x. We have already bounded the error on the first portion. How do we deal with
the rest? We start with an important
Definition Tij = points in Si drawn from the j th Gaussian Gj . As before, let lij = |Tij |.
Fact If a point drawn randomly from N (0, ) has x as its first d coordinates, then its
last n d coordinates have the induced (conditional) distribution N (Ax, C), where A =
yx 1 1
xx and C = yy yx xx xy . This well-known result can be found, for instance, in
Lauritzens (1996) book on graphical models.
We will need to tackle the question: for a point (x, y) drawn from N (0, ), what is
the distribution of kyk given kxk? In order to answer this, we need to study the matrix A
a bit more carefully.
Learning mixtures of Gaussians 48
p
Lemma 15 kAxk max kxk n/d for any x Rd .
n
Proof. A = yx 1
xx is an (n d) d matrix; divide it into d 1 square matrices
B1 , . . . , Bn/d1 by taking d rows at a time. Fix attention on one such Bi . The rows of
Bi correspond to some d consecutive coordinates of x; call these coordinates z. Then
we can write Bi = zx 1
xx . It is well-known see, for instance, the textbook by Horn
and Johnson (1985), or consider the inverse of the 2d 2d positive definite covariance
matrix of (z, x) that (xx xz 1 d
zz zx ) is positive definite. Therefore, for any u R ,
uT xx u > uT xz 1 1
zz zx u, and by choosing u = xx v, we find
kBi vk2
kvk2 = uT xx u > v T BiT 1 2 1
zz Bi v kBi vk min (zz ) 2
.
max
Therefore kBi vk max kvk . The pieces now come neatly together,
For the analysis we are pretending that we know the x coordinates of all the data
points but that their y coordinates have not yet been chosen. We have seen that points in
i j , N (A(b
Tij roughly have conditional distribution j + (b i j ), C)). What bounds
can be given for the average of these points? In particular we would like to bound the
deviation of mean(Tij ) from j and thereby from i .
Proof. For the sake of convenience translate the Gaussians and data points by j .
This centers the j th Gaussian Gj at the origin and centers Gi at i j . The translated
Si now lies within B(i j ; max
d). Recall that Tij consists of those points in Si
which come from Gaussian Gj . Therefore, for our purposes we can pretend that each point
(Xa , Ya ) Tij , in the translated space, is generated in the following fashion:
Pick Xa B(i j ; max
d) Rd in some unknown manner.
d
Choose Ya = N (AXa , C).
Learning mixtures of Gaussians 49
In this manner we choose lij = |Tij | points, with mean value some (X, Y ). The range of
the Xa coordinates constrains kXk to be at most ki j k + max
d. To understand the
d d
distribution of Ya , we notice (Ya AXa ) = N (0, C) = C 1/2 N (0, Ind ), and taking averages,
d
Y = AX + C 1/2 N (0, l1ij Ind ).
Armed with this result we can prove the correctness of this phase of the algorithm.
Lemma 17 Under the conditions of Theorem 14, with probability at least 1 , for all
q
i i k e2 + 2l max n.
1 i k, kb
Proof. Fix any i. The normed difference between i and the mean of Si , which we hope
is close to zero, is given by
kmean(Si ) i k
k
X lij
X lij
(mean(Tij ) j ) l
+
kj i k
j=1
j6=i l
k r
X lij
max n X lij
ki j k + max d 1+ + kC 1/2 Nl k + kj i k
l min d l
j=1 j6=i
k
X lij
5cij e max n + kC 1/2 Nl k
l
j=1
where Nl has distribution N (0, 1l Ind ) and the second inequality uses Lemmas 15 and 16.
It remains to bound these last two terms.
(a) Since C = yy yx 1
xx xy and each of these two right-hand terms is positive semidef-
2
inite, max (C) max (yy ) max and therefore kC 1/2 Nl k max kNl k. Bounding kNl k
is routine (Lemma A.3). In particular, if d 21 n,
q
P(kNl k > 2n l ) e
n/16
k.
lwj
(b) By assumption, lij 10cij . Applying this bound immediately yields the lemma.
The first three phases of the algorithm obtain very rough estimates of the high-
dimensional centers. These are enough to cluster the data almost perfectly, and with this
clustering we can obtain sharp estimates of the means, covariances, and mixing weights.
Associate each data point in S with its closest center-estimate
bi . It can be shown
that the expected proportion of misclassified points in this hard clustering is small, roughly
Learning mixtures of Gaussians 50
O(ke(n) ). The proof is not difficult but requires some care because of the dependence
between the data and the estimated centers.
Lemma 18 Suppose the true mixture {(wi , i , )} is 1-separated, and that the estimated
centers
bi satisfy
1
kb
i i k 16 max n.
In the hard clustering, a point x drawn from N (i , ) will not be misclassified as belonging
to N (j , ), j 6= i, if
(a) kx i k 45 cij max n and
j i
(b) (x i ) ki j k 16 cij max n.
Proof. Pick any x from N (i , ) and any j 6= i. We will show that under the conditions
above, kx
bi k < kx
bj k, and thus x will not be misclassified.
We start by rewriting kx bj k2 kx bi k2 , which we hope is positive, as kb bj k2
i
2(x
bi ) (b
j
bi ). The quantity kb
j bi k differs from kj i k by at most 18 max n, and
thus is at least 78 cij max n. The second term, meanwhile, depends upon the interaction of
the data and the estimated centers, and so we would like to rewrite it in terms of the data
and the true centers.
(x
bi ) (b
j
bi ) =
(x i ) (j i ) (x i ) ((j
bj ) (i
bi )) (b
i i ) (b
j
bi )
(x i ) (j i ) + kx i k (kj
bj k + ki
bi k) + kb
i i k kb
j
bi k
1 5 1 1
cij kj i kmax n + cij max n max n + max n kb j bi k
6 4 8 16
1 2 2 5 2 1
= cij max n + cij max n + max n kb j
bi k
6 32 16
1
< kb
j bi k2 .
2
bj k2 kx
Therefore kx bi k2 > 0, as we had hoped.
This lemma makes it easy to show that the hard clustering is close to optimal.
Lemma 19 Fix any two indices i 6= j, and draw X randomly from N (i , ). Then the
probability that X violates either condition (a) or (b) of the previous lemma is at most
2 c2ij n/72
ecij n/20 + 6
cij 2n
e , assuming as before that cij 1.
Learning mixtures of Gaussians 51
For any unit direction , the random variable (X i ) has distribution N (0, T ), and
this variance T is at most max
2 . Therefore, for any j 6= i,
j i
P((X i ) kj i k > 16 cij max n) P(Z > 61 cij n),
where Z is a standard normal random variable. This last probability can routinely be
bounded (Durrett, 1996, p.7) to complete the lemma.
This more or less completes the proof. The chance that a particular point is misclassi-
fied in phase four is at most k times this value, which is miniscule if n is large. Specifically,
2
if m = 2 wmin
ln k points are chosen randomly from the data set, then with probability
2
at least 1 , there will be at least 2 points from each Gaussian. The chance that none
of these points is misclassified is very high, at least 1 if n (d + log 1 ). And, 22
points from a Gaussian are enough to specify its mean within L2 distance max n, with
probability at least 1 en/8 . In other words, by computing statistics of the points in each
hard cluster, we can expect to get good estimates of the means, as well the mixing weights
and covariance matrix.
The final phase of the algorithm is similar to a single M step of EM (although it
uses a hard clustering like k-means) while phases one, two, and three are devoted to finding
a reasonable initial solution. In particular, the accuracy required of the first three phases,
which we have been calling , is unrelated to the accuracy the user demands.
Well-separated spherical Gaussians in very high dimension are easy to learn. Data
drawn from them can be clustered just by looking at interpoint distances. However, spherical
clusters are unlikely to occur naturally in data because different attributes are typically
measured along different scales. Random projection helps to make arbitrary Gaussians
look spherical. We will also see that it makes a wide range of different distributions look
Gaussian, in the sense of the weak Gaussian assumption. These beneficial effects get
more pronounced as the projected dimension is lowered.
Our algorithm therefore lowers the dimension as much as possible, in the hope of
making the data look almost spherical-Gaussian. This dimension is so low that efficient
clustering can be managed but is no longer trivial.
Learning mixtures of Gaussians 52
6 Shortcomings
The least satisfactory part of this algorithm is the subroutine which estimates centers
in low dimension. Some of its problems are fairly minor. For instance, the need to assess
distances between all pairs of points, unthinkable for very large data sets, can be made
tolerable by instead computing distances from all points to a small random sample of the
data. However, there are two particular flaws which deserve careful attention. First, the
weak Gaussian assumption might not be quite weak enough. The practical applicability
of the algorithm would be more believable if it could be shown to robustly handle data
which at some time satisfied the weak assumption but which has since been corrupted by
a small amount of adversarial noise. The second inadequacy is the requirement that all
clusters have approximately the same radius. Rectifying this would be a significant advance
in current clustering technology.
Open Problem 1 Is there a simple algorithm which correctly learns the centers of a
mixture of spherical Gaussians with widely differing radii?
The recent machine learning literature contains some interesting work on clusters
whose empirical distributions are non-Gaussian in the sense of being discrete or fat-tailed.
Kearns, Mansour, Ron, Rubinfeld, Schapire and Sellie (1994) have considered a dis-
crete analogue of spherical Gaussians which they call Hamming balls. These are specified
by a center {0, 1}n and a corruption probability p (0, 1). The distribution is then
(B(p) B(p) B(p)) , where B(p) denotes a Bernoulli random variable with bias p
(it has value one with probability p and value zero otherwise) and is exclusive-or. Given
data from a mixture of k such balls, the authors show how to find a mixture of O(k log k)
Learning mixtures of Gaussians 53
balls which comes close in distribution to the original mixture. Freund and Mansour (1999)
consider the related model of product distributions over {0, 1}n , that is, distributions of the
form B(p1 ) B(pn ) for some p1 , . . . , pn (0, 1). They present a simple algorithm for
estimating the parameters of a mixture of two such distributions, in a learning framework
similar to ours.
Discrete product distributions seem similar enough in spirit to Gaussians that it
should be possible to extend the techniques of this chapter to accommodate them. Fat-
tailed distributions, on the other hand, can contain strong dependencies between different
coordinates. Rather than attempt a precise definition we shall, roughly speaking, describe
as fat-tailed any distribution over vectors X Rn for which kXk2 falls off significantly more
slowly than in the Gaussian case, in other words, either EkXk2 does not exist, or if it does,
2 n)
P(kXk2 > cEkXk2 ) = eo(c .
Example 1 One generic way of constructing such a distribution is from a stable law of
exponent less than two. The standard Cauchy density in R1 ,
1 t
t (x) = ,
t 2 + x2
approaches zero with such hesitation that its expectation does not exist. Therefore, a
high-dimensional distribution with Cauchy coordinates will be fat-tailed.
Basu and Micchelli (1998) report experiments which attempt to model speech data
using mixtures of fat-tailed densities
p(x) exp c((x )T 1 (x ))
Learning mixtures of Gaussians 54
where < 1 (the case = 1 produces a Gaussian). It turns out that this particular family
of distributions can be accommodated by an algorithm similar to EM. In reading this and
other such highly specialized results, the main question that comes to mind is whether
there exists a unified way of dealing with a heterogeneous collection of distributions, some
of which may be discrete or fat-tailed?
1 d
X = (X1 + + Xn ) N (0, 1)
n
as n increases to infinity. That is to say, the projection of this discrete distribution in the
particular direction looks like a Gaussian. In fact, this will hold for almost any randomly
chosen direction.
A celebrated paper of Diaconis and Freedman (1984) studies this phenomenon in
detail. They find a large family of high-dimensional distributions almost all of whose one-
dimensional projections are close to Gaussian. This is terrible news for anyone who is
trying to visualize a cluster by projecting it onto random lines, but for us it immediately
suggests a simple and unexpected methodology for learning a large class of mixture models:
project the data into a random low-dimensional subspace in the hope that this will make
the clusters more Gaussian, learn these clusters under a Gaussian assumption, and then
perform some kind of high-dimensional reconstruction. Before we get carried away, we
should take a closer look at the fine print. First we review a classical result of Berry and
Esseen (see Feller, 1966, or Petrov, 1995).
Theorem 21 (Diaconis and Freedman, 1984) Suppose vectors X Rn have i.i.d. co-
ordinates with mean zero, variance one and third absolute moment 3 . Then for almost all
unit vectors , the distribution Fn of X is close to normal in the sense that
Similar statements can be made when the coordinates of the vector X are independent
but not identically distributed. Presumably this closeness to normality continues to hold
when the random projection is to a d-dimensional space rather than a line, for small d. In
this more general case, we are interested in the discrepancy on spherical regions, for the
weak Gaussian assumption.
Open Problem 3 Do these results still hold for projections onto random subspaces of
(small) dimension d? Let B denote the class of all spheres in Rd . By how much does the
projected distribution differ from normal on sets in B? How does this difference scale as a
function of d?
Y = (Y1 , . . . , Yn ) uniformly at random from {0, 1} n and then setting Xi = Yi/n .
What can we expect of its random projection?
For a unit direction ,
d
X = 1 Y1 + + n Yn ,
where i = (i1)n+1 + + in . If is chosen randomly then (1 , . . . , n ), when
normalized to unit length, will itself be a random unit vector; therefore the results for
the i.i.d. case will apply once more, except this time the distance from normality will be
proportional to n1/4 rather than n1/2 .
Open Problem 4 If a distribution over vectors X {0, 1}n has entropy at least n(1) ,
it is true that for most random unit directions , the projected distribution Fn = X,
suitably normalized, satisfies supa<b |Fn (a, b) (a, b)| n(1) ?
Example 3 This one is due to David Zuckerman. Consider a distribution D on {0, 1}n
which is uniform over
T = {strings with exactly 31 n ones} {strings with exactly 31 n zeros}
and is zero elsewhere. D has high entropy (n) because the size of T is exponential in n.
However, a vector X = (X1 , . . . , Xn ) drawn randomly from D will not satisfy the law of
1
large numbers because n (X1 + + Xn ) cannot be close to its expected value 12 ; it will
1
always be either 3 or 23 . This means that the distribution of (X1 + + Xn )/ n is far from
Gaussian. The open problem expresses the hope that, nevertheless, for most directions ,
the projection X will look Gaussian.
57
Chapter IV
Probabilistic nets
1 Background
Directed probabilistic nets can express distributions far more complicated than Gaus-
sian mixtures. They pay for this representational power by residing in a thick swamp of
computational intractability. However there are a few positive algorithmic results (Ed-
monds, 1967; Chow and Liu, 1968; Pearl, 1988; Lauritzen and Spiegelhalter, 1988; Jaakkola
and Jordan, 1999), quite modest in scope, and to which the current popularity of proba-
bilistic nets can directly be traced. Before reviewing these we as usual need to agree upon
a barrage of notation.
A directed probabilistic net over the random variables X1 , . . . , Xn assigns each a node
and imposes a directed acyclic graph structure upon them. Let i {X1 , . . . , Xn } denote
the parents of node Xi , that is, the nodes which point to Xi . Then Xi is reciprocally called
a child of any member of i . The indegree of Xi is the number of parents it has, |i |.
The acyclic structure makes it inevitable that at least one node will have no parent. Such
nodes are called sources. A distribution over the net is specified via conditional probability
distributions P(Xi |i ) whose product is the joint distribution on (X1 , . . . , Xn ).
A maximal structure has nodes with indegrees 1, 2, . . . , n1 (Figure 1, left-hand side)
and can represent all distributions over the random variables. Often, however, a distribution
exhibits various approximate conditional independencies which can be exploited to reduce
the complexity of the structure graph. A tension is thus created, in which model size is
reduced at the expense of accuracy.
Probabilistic nets 58
If the variables can take on a continuous range of values and a decision is made not to
discretize them, the most common choice of conditional probability function is the linear-
Gaussian. For X with parents Z1 , . . . , Zk as before, this is specified by a vector v Rk and
a noise parameter :
d
X = v (Z1 , . . . , Zk ) + N (0, 2 ).
In other words, X is assumed to be a linear function of its parents, with some independent
Gaussian noise added in to make whole thing look stochastic.
1.2 Inference
that he has the following symptoms: . . . ? The process of answering such queries is called
inference in the artificial intelligence literature.
Since probabilistic nets are a generalization of Boolean circuits, any inference pro-
cedure can be used to solve NP-complete problems like does this Boolean circuit ever
produce the result zero? It follows immediately that inference is NP-hard to perform even
approximately, to within any multiplicative factor (Cooper, 1990). Dagum and Luby (1993)
show that inference is also NP-hard to perform within a constant additive accuracy, even
for probabilistic nets in which all indegrees and outdegrees are small, say three or less.
These hardness results very seriously call into question the practicality of probabilistic
nets, and make it critically important to find subclasses of nets for which efficient, possibly
approximate, inference is possible. The most notable such class is that of directed trees,
called polytrees in some of the literature, which permit exact inference to be performed
quickly by dynamic programming, or by a convenient message-passing formalism (Pearl,
1988).
There are a few other positive results for inference. The junction tree algorithm (Lau-
ritzen, 1996) works efficiently for a small class of graphs whose characterization is somewhat
intricate but which includes polytrees. For nets composed of layers of noisy-OR gates (or
various other classes of conditional probability functions), variational approximations can
be used as part of an inference heuristic (Jaakkola and Jordan, 1999). The quality of these
approximations differs between instances, but in each case the final result is accompanied
by nontrivial bounds on its error. This technique has worked impressively on the QMR-DT
net.
2.1 Entropy
This value is best thought of as a measure of the inherent randomness content of X, scaled
so that a fair coin has unit entropy. A biased coin has a slightly less random outcome and
therefore has entropy less than one. A deterministic event has zero entropy.
Probabilistic nets 60
This definition of randomness has various satisfying properties (Cover and Thomas, 1991).
H(X) 0.
H(X, Y ) H(X) + H(Y ), with equality if and only if X and Y are independent.
Therefore H(Y |X) H(Y ).
If X can only take values in some finite set S then H(X) log2 |S|, with equality if
and only if X is uniformly distributed over S.
Suppose a collection of m data points is obtained from some arbitrary target distri-
bution D over (X1 , . . . , Xn ), and a probabilistic net is sought which fits this data well. The
log-likelihood of a particular model is
m
1 X
LL() = log (Probability that assigns to the ith data point).
m
i=1
Let i be the parents that chooses for Xi . The maximum-likelihood choice is to set the
conditional probabilities P(Xi = xi |i = i ) to their empirically observed values. The
log-likelihood then becomes
n
X
LL() = H(Xi |i ),
j=1
where H(Xi |i ) is computed with respect to the empirical distribution consisting of the m
samples.
The question how many samples are needed so that a good fit to the data reflects
a good fit to D? has been considered elsewhere (Dasgupta, 1997; Hoffgen, 1993), and so
we will not dwell upon sample complexity issues here. Henceforth we blur the distinction
between D and the empirically observed distribution.
Probabilistic nets 61
SL(k): Assume the random variables Xi are drawn from some finite set .
Given the conditional entropy H(X|) for each variable X and each set of k
parents , find a directed acyclic graph G which minimizes cost(G) subject to
the constraint that all of its indegrees are at most k.
The machine learning literature would describe SL as addressing the unknown struc-
ture, fully-observable case. This means that the structure G needs to be learned and that
there are no hidden variables, that is to say the samples contain values for all n attributes.
The use of hidden variables can lead to significantly more compact probabilistic nets.
For instance, if n-tuples (X1 , . . . , Xn ) are generated from a mixture of two product distri-
butions over {0, 1}n then a fully-observable probabilistic net with nodes X1 , . . . , Xn will
typically require a large amount of connectivity to accurately model this data. If, however,
a single hidden node Y is introduced then a very simple structure (Figure 1) will suffice.
Here Y functions as a mechanism for distinguishing clusters in the data. A different and
Probabilistic nets 62
X4
X2 X1 X2 X3 X4 X5
versus
X5
X1 Y
X3
less subtle use of hidden nodes, for storing intermediate results to reduce indegree, was
mentioned in the introduction (Figure I.3).
Despite the tremendous importance of hidden variables, almost nothing is known
about when and how to use them given observable data, except in a few of the simple
clustering scenarios that we have mentioned. We too will steer clear of this difficult subject.
In the fixed structure scenario, the underlying graph of the probabilistic net has al-
ready been constructed by experts and cannot be changed even if it poorly matches the
actual data. All that remains is to assign the conditional probabilities. If these take the
form of CPTs, then the maximum-likelihood choice is to set them to their empirically ob-
served values. Setting the parameters of a noisy-OR gate is only slightly more complicated,
since the space of possibilities ranked by likelihood can be shown to be concave.
3 SL(k)
Once again, in the SL(k) problem we assume we know H(X|Z1 , . . . , Zk ) for all choices
of X, Z1 , . . . , Zk , and the goal is to find a directed acyclic graph G which has all indegrees
k, so as to minimize
n
X
cost(G) = H(Xi |G
i ).
i=1
We could pick the best k parents for each node Xi : the parents which convey the most
information about Xi , which leave the least amount of randomness in Xi . The problem
is that the resulting graph might not be acyclic. Thus the two conditions which together
make the optimization task hard are the degree restriction and acyclicity. Remove either of
these, and the problem is trivial.
Example 1 Suppose X1 by itself looks like a random coin flip but its value is determined
Probabilistic nets 63
completely by X2 and X3 . That is, H(X1 ) = 1 but H(X1 |X2 , X3 ) = 0. There is then a
temptation to add edges from X2 and X3 to X1 , but it might ultimately be unwise to do
so because of problematic cycles that this creates.
The cost of a candidate solution can lie anywhere in the range [0, O(n)]. The most in-
teresting situations are those in which the optimal structure has very low cost (low entropy),
since it is in these cases that there is structure to be discovered.
The central positive result in this field was the very first one, when in 1967 Edmonds
demonstrated an algorithm for SL(1). Directed graphs of indegree bounded by one are
called branchings. Edmonds showed that the maximum-likelihood branching can be found
in quadratic time by a simple greedy procedure. In fact, the symmetric relationship
(the amount by which knowing Y decreases the entropy of X equals the amount by which
knowing X decreases the entropy of Y ) makes it possible to reduce SL(1) to a single min-
imum spanning tree computation. This was observed by Chow and Liu (1968), with the
result that branchings are often called Chow-Liu trees by artificial intelligence researchers.
The next candidate, SL(2), was shown to be NP-hard by Chickering (1996). His proof
assumes a Bayesian setting in which there is a prior distribution on graph structures, but
the techniques can be adapted to our maximum-likelihood framework.
The algorithms used to solve SL(k) in practice are generally local search heuristics,
including some based on gradient descent (Russell, Binder, Koller and Kanazawa, 1995)
and EM (Friedman, 1997). These procedures start with some initial guess of a structure,
and then incrementally change it by adding and removing single edges. They will also
accommodate hidden nodes. Example 2 illustrates one inherent limitation of such schemes.
Probabilistic nets 64
The NP-hardness of SL(2) closes one door but opens many others. Over the last three
decades, provably good approximation algorithms have been developed for a whole host
of NP-complete optimization problems. Do such algorithms exist for learning structure?
We now present a further negative result showing that for some constant c > 1, unless
P = NP there is no polynomial-time procedure which c-approximates SL(2), that is, which
consistently returns a graph whose cost is within a multiplicative factor c of optimal. From
a practical viewpoint this is a weak result, since it does not rule out the possibility of, say,
2-approximation, or O(log n)-approximation, either of which would be a tremendous boon.
Its importance lies instead in bringing into sharper focus some of the aspects of SL(2) that
make it a difficult problem, and thereby providing valuable insight into the hurdles that
must be overcome by good algorithms.
Theorem 1 There exists some constant c > 1 for which c-approximating SL(2) is an NP-
hard optimization problem.
This is often described by saying that VC(3) has an approximability gap of 1 + . A similar
result must hold for VC(4) since the graphs produced by can be altered to be four-regular
by repeatedly connecting the gadget in Figure 3 to pairs of distinct vertices u, v. The gadget
itself has a vertex cover of size four, regardless of the status of the two vertices it is hooked
into. We will use the fact that f () is always at least some constant fraction of the number
of vertices in (); call this constant r (0, 1).
The second reduction converts instances G = (V, E) of VC(4) to instances G =
(V , E ) of FVS . This is managed by first replacing each undirected edge {u, v} in E by a
Probabilistic nets 65
z1
u u
u1 u2
w x y z w x y z
w x y z
directed cycle u v in E , and then reducing the indegree of each node u from four to
two by introducing two intermediate nodes u1 and u2 which lead to u and each of which
handles two of us inputs (Figure 4). The number of nodes is tripled, so the optimal solution
to G has size at least 31 r|V |, while the approximability gap remains 1 + .
We can finally start on the main reduction. We have chosen FVS as a starting
point because, like SL(2), it requires finding an acyclic subgraph. Given an n-node graph
G = (V, E) as input to FVS , we will create a very similar directed graph G which specifies
a probability distribution P . This distribution (more specifically, conditional entropy values
generated from it) will be used as input to SL(2).
We have already seen how directed acyclic graphs can generate probability distribu-
tions. However, the graph G will contain many cycles. We will ensure that its induced
distribution P is well-defined by making certain that influences do not propagate very far
in G . That is, all correlations are local, or equivalently, any node (random variable) in G
can only affect the values of nodes very close to it, so there is no fear of influences travelling
around cycles of G .
An algorithm for SL(2) will attempt to model P by some directed acyclic graph G
whose indegrees are all at most two. We will show that G can be used to produce a good
solution for the original FVS instance. In particular, the parents of node u G will be
either the parents of u in G (in which case the FVS solution retains node u), or a special
set of default parents (in which case the FVS solution discards node u). These defaults
are extra random variables whose behavior is coordinated through the gadget shown in
Figure 5.
In this device, A1 and A2 are random B( 21 ) coin flips (B(p) stands for Bernoulli(p),
that is, a {0, 1}-valued random variable with probability p of being one), C1 = (A1 A2 )
B(q), C2 = (A1 A2 ) B(q), and L = C1 C2 , for some specific q (0, 12 ) whose value we
Probabilistic nets 67
A1 C1
A2 C2
z2 z3
z1
z4
u
y4
v w
Figure 6: A node u in G.
shall not dwell upon. The entropy of the gadgets distribution is H0 = 2H( 21 ) + 2H(q) =
2 + 2H(q). Any attempt to model the same distribution by a different degree-two-bounded
structure in which L has less than two inedges will increase the cost to H0 + , where
= min{ 12 21 H(q), H(2q(1 q)) H(q)} > 0 is some small constant. Moreover, this
suboptimal cost can be achieved by an arrangement, or , in which there are no
edges into L.
We can now describe the directed graph G in detail. Pick any node u G, with
inedges from v and w and outedges to z1 , . . . , zl , l 4 (Figure 6). G will contain a gadget
for u, but the L node of this gadget, denoted Lu , will be incorporated into a special random
variable Xu . The remaining gadget nodes, A1u , A2u , C1u , C2u , will exist as Boolean random
variables. In other words, there are five nodes corresponding to u in G : Xu and the four
gadget nodes (Figure 7).
Xu has 64 possible values which can be segregated into six binary-valued fields, Lu ,
(1) (4)
Iu , and Ou , . . . , Ou . The Ou s are output fields containing information to be passed on
(i)
to the Xzi . Each Ou has distribution B(p), for some p (0, 21 ), independent of everything
except Izi , to which it is passed. If u has less than four outedges, then some of these output
Probabilistic nets 68
Xz 3
Iz 3
Xu
A1u C1u
O3u 4
Ou
Lu Iu
1
Ou O2u
A2u C2u
Xv
Xw
O3v 4
Ow
fields contain random values that never get sent anywhere. Iu receives inputs from the
output fields of Xv and Xw , and its value is their exclusive-or. Xv and Xw provide a lot of
information about Xu . The next most informative set of parents for Xu is {C1u , C2u }.
G contains many cycles, of course, but specifies a unique probability distribution
P . To see this, notice that no piece of information can travel very far in this graph it
originates in an O node and moves to an I node, where it is forgotten. To sample from this
distribution, first generate values for all the gadgets, then for all the O nodes, and finally
set each I node based upon the O nodes of its parents.
We now ask an SL(2) algorithm to find the best indegree-two network that models
P . How will this algorithm pick the parents of each vertex? By construction, the only pairs
of nodes which give any information at all about Xu must be chosen from the nodes of us
gadget, and the nodes Xw , Xv , Xzi , Xyi , and Xyi , where yi is the other parent of zi in G,
and yi is a child of yi in G. By considering the various combinations in turn, we find that
there are effectively just two sensible choices for Xu s parents:
{Xv , Xw } (v and w being the parents of u in the original graph), which costs (a total
conditional entropy of) some H1 = H0 + + 4H(p) for the five nodes affiliated with
u and can potentially produce cycles; and
{C1u , C2u }, which entails an entropy cost of H2 = H0 + 4H(p) + H(2p(1 p)), and
can never result in the formation of cycles.
Any other choice of parents incurs cost at least H3 = H0 + + 4H(p) + min{H(2p(1 p))
H(p), 2H(p) H(2p(1 p))}. By setting p and q carefully, for instance p = 0.07, q = 0.25,
we ensure that H3 > H2 = H1 + , for a small constant > 0.
Thus an SL(2) algorithm will always be able to choose two parents for each node
Xu . As far as possible it ought to choose the nodes parents in the original graph G. This
corresponds to retaining that node for FVS purposes. Otherwise, the parents might as well
be chosen from that nodes gadget, which corresponds to removing the node in G. More
precisely, a solution of FVS (G) in which j nodes are removed corresponds to a solution
G G of the SL(2) problem with cost(G ) = nH1 + j. Since and H1 are constants, and
since instances of FVS with optimal solutions of size at least 13 rn are hard to approximate
within a constant, we see that SL(2) is also hard to approximate within a constant.
Probabilistic nets 70
The next question is whether there exists an even more damaging hardness result.
Consider the following generalization of SL.
HT: Given a non-negative weight w(X, ) for each variable X P and each set
of k parents , find a directed acyclic graph G which minimizes i w(Xi , G i )
subject to the constraint that all of its indegrees are at most k.
This problem is hard to approximate within O(log n), by reduction from Set Cover
(Hoffgen, 1993). Therefore, any algorithm which seeks a better performance guarantee for
SL must rely upon the fact that the weights w(X, ) are not arbitrary but are conditional
entropies. The proofs of the next chapter will make heavy use of this.
Open Problem 1 Standard tricks for reducing fan-in and fan-out can be deployed to
show that FVS is not substantially easier to approximate than the general FVS problem,
for which the best known algorithm (Seymour, 1995) achieves an approximation ratio of
O(log n log log n). However, our proof technique can demonstrate at most an O(1) approx-
imability gap for SL because the distribution P created by the reduction has entropy (n).
Reducing the entropy of P would require allowing its dependency graph to contain long-
range (that is, non-local) influences. Can these be introduced in such a way that there are
no inconsistencies around cycles?
Example 3 A useful test distribution for greedy heuristics is one which is produced by
a directed graph (Figure 8) with 2 n sources X1 , . . . , Xn , Y1 , . . . , Yn , each i.i.d. B( 21 ).
The sources generate Xij = Xi Xj and Yij = Yi Yj , for all i, j such that |i j| is odd.
Finally all combinations Zij = Xij Yij are present. There are a total of about 2 n + 34 n
random variables.
The optimal structure shown in the figure has entropy 2 n. There are no correlated
triples of variables other than those of the types mentioned. For instance, there are no
Probabilistic nets 71
Zij zjk
Xj
Xi Yk
Yj
Yi
Xk
Figure 8: Finding the 2 n source variables might not be easy.
relationships Xij Xik = Xjk because at least one of the differences |i j|, |i k|, |j k|
must be even. Thus any greedy scheme which iteratively identifies strong correlations and
then chooses arbitrarily amongst them will end up picking (n) of the Xij or Yij as sources,
giving an overall entropy of (n).
Open Problem 2 In this deterministic setting, SL reduces to finding sources of the optimal
structure. To what extent does this source discovery problem capture the hardness of
SL?
72
Chapter V
Learning polytrees
1 Why polytrees?
The theoretical terrain of the structure learning problem so far contains just two major
landmarks: it is easy to learn branchings and difficult to learn general probabilistic nets
of bounded indegree. With these two endposts in mind, it is natural to ask whether there
is some subclass of nets which is more general than branchings, and for which a provably
good learning algorithm can be found. Even if this learning problem is NP-hard, it may
well be the case that a good approximation algorithm exists. We will now take a small step
in this direction by considering the task of learning polytrees from data. Polytrees have
more expressive power than branchings, and have turned out to be an important class of
directed probabilistic nets, largely because they permit fast exact inference (Pearl, 1988).
In strict analogy with SL, we define
PT: Assume the random variables Xi are drawn from a finite set . Given the
conditional entropy H(X|) for each variable X and each set of parents , find
a polytree G which minimizes cost(G).
PT(k): just like PT, except that each node is allowed at most k parents (we
shall call these k-polytrees).
It is not at all clear that PT is any easier than SL. However, we will see that, first,
the optimal branching constitutes a provably good approximation to the best polytree, and
second, it is not possible to do very much better, because there is some constant c > 1
Learning polytrees 73
for which it is NP-hard to c-approximate PT(2). That is to say, if on input I the cost of
the optimal solution to PT(2) is denoted OP T (I), then it is NP-hard to consistently find
2-polytrees of cost c OP T (I).
There has been some work on reconstructing a polytree given data which actually
comes from a polytree distribution for instance, by Geiger, Paz and Pearl (1990) and by
Acid, de Campos, Gonzalez, Molina and Perez de la Blanca (1991). We, on the other hand,
are trying to approximate an arbitrary distribution by a polytree.
2 An approximation algorithm
We will show that the optimal branching has cost (and therefore log-likelihood) close
to that of the optimal polytree. They are different by at most a multiplicative factor
which is not an universal constant but depends upon the nature of the individual attributes
(nodes). For instance, if the attributes individually resemble fair coin flips, then the cost of
the optimal branching is always at most twice that of the optimal polytree. This is a very
significant performance guarantee, given that this latter cost can be as low as one or as high
as n, and instantly gives us an O(n2 )-time approximation algorithm for learning polytrees.
The following definition will be crucial for the rest of the development.
Definition For any probability distribution over variables X1 , . . . , Xn , let U = maxi H(Xi )
(that is, the highest entropy of an individual node) and L = mini H(Xi ).
Example 1 If all the variables are Boolean and each by itself is a random coin flip, then
U = L = 1. Similarly, if some variables have values in the range {1, 2, . . . , 100} and the rest
are Boolean, and each variable by itself looks pretty uniform-random (within its range),
then U log2 100 < 7 and L 1. It is always true that the cost of the optimal polytree (or
the optimal branching, or the optimal directed probabilistic net) lies in the range [L, nU ].
We warm up with an easy theorem which provides useful intuition about PT.
U
Theorem 1 The cost (negative log-likelihood) of the optimal branching is at most (1 + L)
times than that of the optimal polytree.
Learning polytrees 74
Proof. Let the optimal polytree have total cost H (we have chosen the letter H because
the cost is essentially an entropy rating, and it is helpful to think of it as such). We will use
this polytree to construct a branching of cost H (1 + UL ), and this will prove the theorem.
In the optimal solution, let S denote the vertices with indegree more than one. Since
the total number of edges in the structure is at most n 1, each node of high indegree
is effectively stealing inedges away from other nodes (cf. Example IV.1). In particular
therefore, the polytree must have at least |S| + 1 sources (that is, nodes with no parents),
implying that H |S|L.
If we remove the edges into the vertices of S (so that they become sources in the
graph), we are left with a branching. Each vertex of S has entropy at most U , and so the
U
cost of this branching is H + |S|U H (1 + L ).
Thus in cases where the nodes have approximately equal individual entropies, the
optimal branching is at most about a factor of two worse than the optimal polytree, in
U
terms of log-likelihood. What happens when the ratio L is pretty large? This is especially
a danger when different variables have vastly different ranges. Over the course of the next
U U
few lemmas, we will improve the dependence on L to O(log L ), and we will show that this
is tight. We start with a very crude bound which will be useful to us later.
Sink
Child
Parent X
Source
Tx
Figure 1: Notation
over all nodes is H ) and ignores its original definition in terms of Zs parents in the optimal
polytree. This is because we will be doctoring the optimal polytree while leaving the (Z)
fixed. We start with a few examples to clarify the notation.
Example 2 Suppose that in the optimal polytree, node X has parents Y1 , . . . , Yk . Then
the subtrees TY1 , . . . , TYk are disjoint parts of TX and therefore
k
X
(TYi ) (TX ) H .
i=1
Decompositions of this kind constitute the bread and butter of the proofs that follow.
H(X) (TX ),
since the right-hand term is the total entropy of the subtree TX which includes X. Here is
another way to think of it: denote by S the variables (including X) in the subtree TX . Then
TX is a polytree over the variables S, and so (TX ) is at least the inherent entropy H(D|S )
of the target distribution D restricted to variables S, which in turn is at least H(X).
Given the optimal polytree, we need to somehow construct a branching from it which
approximates it well. Nodes with zero or one parent can remain unchanged, but for each
node Z with parents Y1 , Y2 , . . . , Yl , l 2, we must eliminate all but one inedge. Naturally,
the parent we choose to retain should be the one which contributes the most information to
Learning polytrees 76
Z. The information lost by removing parent Yi is at most its entropy H(Yi ), which in turn
is upper-bounded by (TYi ) (see Example 3). Therefore, we will use the simple rule: retain
the parent with highest (TYi ). Formally, the increase in cost occasioned by removing all
parents of Z except Yj can be bounded by
X X X
H(Z|Yj ) H(Z|Y1 , Y2 , . . . , Yl ) H(Yi ) (TYi ) = (TYi ) max (TYi ).
i
i6=j i6=j i
Definition For any node Z with l 2 parents Y1 , . . . , Yl in the optimal polytree, let the
P
charge C(Z) be i (TYi ) maxi (TYi ). For nodes with less than two parents, C(Z) = 0.
So we apply our rule to each node with more than one parent, and thereby fashion
a branching out of the polytree that we started with. All these edge removals are going to
drive up the cost of the final structure, but by how much? Well, there is no increase for
nodes with less than two parents. For other nodes Z, we have seen that the increase is at
most C(Z).
In this section, we will charge each node Z exactly the amount C(Z), and it will turn
out that the total charges are at most 12 H log n. Most of the work is done in the following
Lemma 2 For any node Z, the total charge for all nodes in subgraph TZ is at most
1
2 (TZ ) log |TZ |.
P
Proof. Let C(TZ ) = XTZ C(X) denote the total charge for nodes in subgraph TZ . We
will use induction on |TZ |. If |TZ | = 1, Z is a source and trivially C(TZ ) = 0. So, assume
the claim is true for all subtrees of size less than |TZ |. If Z has just one parent, say Y , then
C(TZ ) = C(TY ), and we are done. Assume, then, that Z has parents Y1 , . . . , Yl , l 2, and
let i = |TYi |/|TZ |. Then 1 + + l 1, and
X
C(TZ ) = C(TYi ) + C(Z)
i
X X
1
2 (TYi )log i |TZ | + (TYi ) max (TYi )
i
i i
X
1
2 (TZ ) log |TZ | + (TYi )(1 + 21 log i ) max (TYi )
i
i
1
2 (TZ ) log |TZ |,
where the second line applies the inductive hypothesis to subtrees TY1 , . . . , TYl , the third
P
line follows from (TZ ) i (TYi ) (because these l subtrees are disjoint), and the fourth
Learning polytrees 77
line is the result of (1) ignoring all i smaller than 41 , (2) applying Jensens inequality, and
(3) using the fact that maxi ai dominates any convex combination of {ai }.
Now we apply this to selected subgraphs of the optimal polytree. Our aim, again, is
to bound the additional cost incurred by removing edges necessary to convert the optimal
polytree into a branching. Specifically, we want to show
n
X n
X
1
C(Xi ) 2 H log n = (Xi ) 21 log n.
i=1 i=1
Theorem 3 The cost of the optimal branching is at most (1 + 12 log n) times the cost of the
optimal polytree.
Proof. Suppose the optimal polytree were the union of a few disjoint subtrees TZ1 , . . . , TZl
(equivalently, suppose each connected component had just one sink). Then we would be in
business, because we could apply the previous lemma to each of these subtrees and sum the
results. We would get
n
X l
X
C(Xi ) = C(TZi )
i=1 i=1
Xl
1
2 (TZi ) log |TZi |
i=1
n
X
1
2 (Xi ) log n
i=1
1
= 2 H log n,
where the second line uses Lemma 2 and the third line uses |TZi | n.
If the optimal polytree is not of this convenient form, then it must have some con-
nected component which contains two sinks X and Y such that TX and TY share a common
subgraph TU (Figure 2).
Say that TU is connected to the rest of TY through the edge (U, V ), V TY TU .
Remove this edge, and instead add the edge (X, V ). This has the effect of moving TY TU
all the way up TX . After this change, X is no longer a sink, and for every node Z in the
graph, (TZ ) (and thus C(Z)) has either stayed the same or increased, because each node
has all the ancestors it had before and maybe a few more (recall that we are thinking of
each (W ) as a fixed value associated with node W and that (TZ ) is the sum of these
values over W TZ ). It is also the case, and this will be useful later, that the indegree of
Learning polytrees 78
X Y V
V X
each node remains unchanged. In this manner we alter the structure so that it is a disjoint
union of single-sink components, and then apply the previous argument.
In the optimal polytree, let n0 denote the number of nodes with no parents (that is,
the number of sources), and n2 the number of nodes with at least 2 parents. By examining
what happens to nodes of different indegree in the above proof, we notice that the theorem
remains true if we replace n by n0 + n2 , that is, if we ignore nodes with exactly one inedge.
In the argument so far, we have bounded the entropy of a node X by H(X) (TX ).
This is a fairly tight bound in general, except when these entropy values start getting close
to the maximum entropy U . We factor this in by using a slightly more refined upper bound.
Definition For any node Z, let C (Z) = min{C(Z), U }. Then C (Z) is an upper bound on
the extra cost incurred at node Z due to the removal of all but one parent.
This gives us the final improvement. Once again, we start by considering a single subtree.
Lemma 4 For any node Z, the total cost incurred by nodes in the subtree TZ is at most
C (TZ ) ( 25 + 1
2 log UL )(TZ ).
Proof. Let T TZ denote the polytree induced by all nodes X TZ such that (TX ) > U ;
that is, T consists of these nodes, and any edges between them in TZ . Note that T must be
Learning polytrees 79
In T2 Z
In T1
In T0
T
where the final inequality follows by noticing that the preceding summation is over disjoint
subtrees of TZ (cf. Example 2).
(2) Nodes X T2 , T3 , . . . have at least two parents in T , and so get charged exactly U ;
however, since T is a tree we know that |T0 | 1 + |T2 | + |T3 | + , and thus
X X
C (X) U |T0 | (TX ) (TZ ),
XTi ,i2 XT0
U 2U
we can conclude that TV has at most L sources and therefore at most L nodes of indegree
6= 1. The charge C (TV ) is therefore, by Lemma 2, at most 12 (TV ) log 2U
L . We sum this
over the various disjoint {TV : V B} and find that the total charge for TZ T is at most
1 2U
2 (TZ ) log L , whereupon C (TZ ) ( 25 + 1
2 log UL )(TZ ), as promised.
Proof. It can be verified as in Theorem 3 that for charging purposes we may assume the
optimal solution is a disjoint union of single-sink subgraphs. Apply the previous lemma to
each of these subtrees in turn, and sum the results.
Example 4 The bound on the ratio between the best branching and the best polytree has
to depend upon log UL . To see this, consider a polytree which is structured as a complete
1
binary tree (with 2n + 1 leaves) and with edges pointing towards the single sink. All
nodes are binary-valued, each internal node is the exclusive-or of its two parents, and each
source has some small entropy = (1/n). The nodes on the next level up each have
approximately double this entropy, and so on, so that the sink has entropy about (1). The
optimal polytree has cost ( 12 n + 1) = (1) whereas the best branching loses (n) entropy
on each level and therefore has cost about (n log n) = (log n) = (log UL ).
3 A hardness result
Theorem 6 There is some constant c > 1 for which the problem of finding a 2-polytree
whose cost is within c times optimal is an NP-hard optimization problem.
Proof. We shall reduce from the following canonical problem ( > 0 is some universal
constant).
Learning polytrees 81
C1 C2 C3 Cm
R1 X1 R2 X2 ...... Rn Xn
Max3SAT(3)
Input: A Boolean formula in conjunctive normal form, in which each clause
contains at most three literals and each variable appears exactly three times. It
is guaranteed that either there is an assignment which satisfies all the clauses
(that is, is satisfiable) or no assignment satisfies more than a (1 ) fraction
of the clauses.
Question: Is satisfiable?
1. The top layer consists of i.i.d. B(p) random variables, one per clause, for some p
(0, 1). This entire layer has entropy mH(p).
2. The middle layer contains two nodes per variable xi : a principal node Xi which is
Learning polytrees 82
Xi
C1 C2
Ri C3 B(p) Ri
Pi Ni
correlated with the nodes for the three Cj s in which xi appears, and a B( 12 ) satellite
node called Ri . Let us focus on a particular variable xi which appears in clauses
C1 , C2 with one polarity and C3 with the opposite polarity this is the general case
since each variable appears exactly three times. For instance, C1 and C2 might contain
xi while C3 contains xi . Then the nodes Ri , Xi will have the structure shown in
Figure 5. Here Pi and Ni are both B( 21 ), and stand for previous and next. Each
Ri is Boolean with entropy one and each Xi is a four-tuple of Boolean values, with
entropy H(2p(1 p)) + 3. Thus the sum of the entropies of the second layer nodes is
n(H(2p(1 p)) + 4).
3. The third layer (of n1 nodes) joins together consecutive nodes Xi , Xi+1 of the second
layer. The value of the ith node in this level, 1 i n 1, is Ni Pi+1 , and the
entire level has entropy n 1.
If is satisfiable then there is a 2-polytree whose cost is some value H which can
easily be computed from .
If is not satisfiable (and so at most m(1 ) clauses can be satisfied by any assign-
ment) then there is no 2-polytree of cost less than cH .
Thus, any algorithm which can find a 2-polytree within factor c of optimal, can also decide
whether is satisfiable.
Learning polytrees 83
B( 12 ) Ri
A1 B1 ,...,Ak Bk
A B
A1 ,A2 ,...,Ak B1 ,B2 ,...,Bk
How can we prove this? A good polytree must somehow tell us a satisfying assignment
for . If the polytree that is learned has an edge from Cj to Xi , we will say Cj chooses Xi
and will take this as meaning that clause Cj is satisfied on account of variable xi . There are
several steps towards showing this assignment is well-defined.
First we will force the square nodes to have no inedges. This is quite easy to achieve;
for instance, R1 , ostensibly a single coin flip, can be rigged to avoid inedges by giving
it two default inedges via the little gadget of Figure 6. Here A and B are two extra
nodes, independent of everything but R1 , which each contain k i.i.d. copies of B(q), called
1
A1 , A2 , . . . , Ak and B1 , . . . , Bk , for some constants q < 2 and k > 1. R1 has now expanded
to include A1 B1 , . . . , Ak Bk . On account of the polytree constraint, there can be at
most two edges between A, B, and R1 . The optimal choice is to pick edges from A and B
to R1 . Any other configuration will increase the cost by at least k(H(2q(1 q)) H(q)) > 1
(the constant k is chosen to satisfy this). These suboptimal configurations might permit
inedges from outside nodes which reveal something about the B( 21 ) part of R1 ; however
such edges can provide at most one bit of information and will therefore never be chosen
(they lose more than they gain). In short, these gadgets ensure that square nodes will have
no inedges from the nodes in the overall graph (Figure 4).
Next we will show that any half-decent polytree must possess all 2(n 1) connections
between the second and third levels. Any missing connection between consecutive Xi , Xi+1
causes the cost to increase by one and in exchange permits at most one extra edge among
the Xi s and Cj s, on account of the polytree constraint. However we will see that none of
these edges can decrease the cost by one.
Therefore, all the consecutive Xi s are connected via the third layer, and each clause-
node Cj in the first level can choose at most one variable-node Xi in the second level
Learning polytrees 84
1
C1 , C3 or C2 , C3 : the entropy decrease is 2 .
Thus C1 and C3 will not both choose variable Xi , and more generally, the assignment
embodied in the edges from clause-nodes to variable-nodes will be well-defined. Each node
Xi can earn an entropy reduction of by merely having an inedge from Ri . In addition,
1
each satisfied clause will reduce entropy further by 2 . Suppose m clauses are satisfiable
(either m = m or m m(1 )). Then the edges into the Xi s will decrease the cost of
the second layer by m ( 21 ) + n.
In this way, if is satisfiable, then the optimal structure has some cost H = (m),
whereas if only (1 )m of s clauses are satisfiable, then the optimal cost is H + (m)
cH for some constant c > 1 which depends only upon and not upon . Thus PT(2) is
hard to approximate within some fixed constant factor.
Open Problem 1 This proof relies heavily upon the degree restriction as a source of
hardness. It should be possible to produce similar effects using just the polytree constraint.
Learning polytrees 85
Open Problem 2 We have seen that the optimal branching, in which each node has at
most one parent, is a good approximation to the optimal polytree, in terms of log-likelihood.
Any algorithm with a better performance guarantee must be able to give nodes two or more
parents when necessary. What are good heuristics for doing this?
Open Problem 3 Polytrees are currently the most natural class of directed graphs for
which efficient inference algorithms are known. But in fact the junction tree algorithm
works quickly for any directed graph whose moralized, triangulated, undirected version has
very small cliques. Is there a larger class of directed graphs with simple characterization
which permits fast inference and for which efficient, provably good learning algorithms can
be constructed?
86
Chapter VI
1 Mixture models
The most important contribution of this thesis to the theory of mixture models is its
suggestion that random projection be used to reduce the dimension of data sets. This has
a wide range of benefits, the first of which is that arbitrarily high-dimensional data from
a mixture of k well-separated distributions can be mapped into just O(log k) dimensions.
Another especially helpful effect is a reduction in the eccentricity of the Gaussians.
Open Problem 1 Lemma II.4 bounds the eccentricity of a projected Gaussian in terms
of its original eccentricity and the initial and reduced dimensions. Is this bound tight?
Mixtures of identical spherical Gaussians in high dimension are easy to learn. How-
ever, real clusters are unlikely to be spherical-Gaussian, if only because different attributes
are typically measured in different units. Our approach has been to exploit the fact that
random projection makes a variety of different distributions, including very ellipsoidal Gaus-
sians, look more spherical-Gaussian, in the sense of the weak assumption of Chapter III.
This beneficial effect deserves much more detailed treatment than it has received here.
There is one question that is particularly beguiling.
Open Problem 2 Let X be a random vector chosen uniformly from a subset S of the
1
hypercube {0, 1}n . If S is large, say |S| = 2 2n , is it the case that for most unit directions
, the distribution of X is close to normal? How close?
which are adept at learning mixtures of well-separated spherical Gaussians in relatively low
dimension. We have analyzed one such algorithm which has the unfortunate requirement
that the Gaussians all have about the same radius.
Open Problem 3 Is there a simple algorithm which will learn a mixture of spherical
Gaussians with widely differing radii?
We have identified SL as the core combinatorial problem in the task of learning the
maximum-likelihood directed probabilistic net from data. The central proof of Chapter IV,
showing that SL(2) is hard to c-approximate for some constant c > 1, requires an unusual
reduction in which a probability distribution is explicitly constructed from some instance
of a graph problem. Can these techniques be extended to show stronger hardness results?
Open Problem 5 Can SL(2) be shown to be as hard as Set Cover or FVS? Conversely,
is there any algorithm for SL(2) with a nontrivial performance guarantee?
Learning the maximum-likelihood polytree might not be any easier than SL. However,
we have seen that the representational power of polytrees is not vastly superior to that of
branchings. It would be a significant theoretical advance to develop a PT(2) algorithm
whose performance guarantees did not rely solely upon this fact, an algorithm which could
make principled decisions about when to assign a node more than one parent.
Appendix A
1 Chernoff-Hoeffding bounds
These are reproduced from the book by Kearns and Vazirani (1991).
The restriction 1 makes these bounds useful only for modest deviations; for the
> 1 regime the following, from Motwani and Raghavan (1995), can be applied.
The next few results are straightforward and presumably well-known. Proofs are
included for the sake of completeness.
Large deviation bounds 89
2
!n/2
Y1 + + Ys
es 1
P
n .
s s2
d
Proof. Let Zi = Yi = N (0, In ). The mean 1s (Z1 + + Zs ) has distribution N (0, 1s In ),
and its squared L2 norm has moment-generating function (t) = (1 2t
s)
n/2 for t s . By
2
Markovs inequality,
Z1 + + Zs
(t)
P
n t2 n ;
s
e
the lemma follows by choosing t = 2s (1 1
2 s
).
Lemma 4 Let X1 , . . . , Xn be i.i.d. N (0, 1) random variables. Pick any sequence of positive
p
coefficients 1 2 n , and define = n1 (1 + + n ) and r = n /1 . Notice
P
that i i Xi2 has expectation n. Then for any 0 < 1,
X n
2 n2
P i Xi n > n 2 exp .
24r 2
i=1
1
Proof. Each Xi2 has moment-generating function (1 2t)1/2 , for t 2. Therefore,
applying Markovs inequality,
X
n P
P i Xi2 > (1 + )n = P exp( i ti Xi2 ) > exp(t(1 + )n)
i=1
Q ti Xi2
i Ee
etn(1+)
!1/2
1 Y 1
=
etn(1+) i
1 2ti
1 1 1 n 2
for 0 < t 2n . Choose t = 2n 1+/2 ln(1 + /2) to bound this expression by exp( 24n
)
n 2
exp( 24r 2 ). Similarly, for t 0,
X P
P i Xi2 < (1 )n = P exp( i ti Xi2 ) > exp(t(1 + )n)
i
Q ti Xi2
i Ee
etn(1)
!1/2
Y 1
= etn(1) .
1 + 2ti
i
2
This is at most exp( n
8r 2
) if you pick t = 1 1
2n 1/2
1
ln 1/2 .
Large deviation bounds 90
Proof. Let X = (X1 , . . . , Xn ) be a vector of n i.i.d. N (0, 1) random variables. Then U has
1
the same distribution as kXk (X1 , . . . , Xn ), and
n
X d 1 X12 + + n Xn2
i Ui2 = .
i=1
X12 + + Xn2
The numerator has expectation n while the denominator has expectation n. The previous
lemma provides large deviation bounds for each.
Lemma 6 (Johnson-Lindenstrauss, 1984) Fix some 0 < < 1 and 0 < < 1. Pick
any m data points x1 , . . . , xm Rn . If these points are projected into a randomly chosen
12 m2
subspace of dimension d 2 log then with probability > 1 , the projected points
x1 , . . . , xm satisfy
d kxi xj k2 d
(1 ) (1 + ) for all i 6= j.
n kxi xj k2 n
The original proof of Johnson and Lindenstrauss was simplified by Frankl and Mae-
hara (1988), using geometric insights and refined approximation techniques. The proof
given here, joint work with Gupta (1999), uses elementary probabilistic techniques to ob-
tain essentially the same results.
We start by estimating the length of a unit vector in Rn when it is projected into
a randomly chosen d-dimensional subspace. This length has the same distribution as the
length of a random unit vector projected into a fixed d-dimensional subspace. We take this
fixed subspace to be that spanned by the first d coordinate vectors, for simplicity.
Let X1 , . . . , Xn be i.i.d. N (0, 1) random variables. Then the distribution of Y =
1
||X|| (X1 , . . . , Xn ) is uniform over the surface of the n-dimensional unit sphere. Let the
vector Y Rd be the projection of Y onto its first d coordinates, so EkY k2 = nd . We will
show that kY k2 is tightly concentrated around this mean value.
Large deviation bounds 91
Lemma 7 Choose any d < n. Let X1 , . . . , Xn be i.i.d. N (0, 1) random variables and let
1
Y denote the first d coordinates of Y = ||X|| (X1 , . . . , Xn ).
(a) If < 1 then
(nd)/2
2 d/2 (1 )d d
P(kY k d/n) 1+ exp (1 + ln ) .
(n d) 2
(b) If > 1 then
(nd)/2
2 d/2 (1 )d d
P(kY k d/n) 1+ exp (1 + ln ) .
(n d) 2
Proof. The proofs are similar to those for Chernoff-Hoeffding bounds. We use the fact
d
that if X = N (0, 1), then X 2 has moment-generating function (1 2t)1/2 , for t < 12 .
(1)
for t 0, td < 12 , and t(d n) < 21 . Choosing t = t0 = 2(nd) then completes the proof
of (a). The proof of (b) is almost exactly the same. By the same calculations,
Lemma 8 Fix any , (0, 1). Let u, v be any two points in Rn . Suppose these points are
projected into a randomly chosen subspace of dimension d < n, where
2 2
d ln .
2 /2 3 /3
If u , v denote the projected points, then with probability at least 1 over the choice of
projection,
d d
(1 ) ku vk2 ku v k2 (1 + ) ku vk2 .
n n
Large deviation bounds 92
Literature cited
Acid, S., de Campos, L.M., Gonzalez, A., Molina, R. & Perez de la Blanca, N. (1991)
Learning with CASTLE. In Symbolic and Quantitative Approaches to Uncertainty, Lec-
ture Notes in Computer Science 548, 99-106. Berlin: Springer-Verlag.
Alimonti, P. & Kann, V. (1997) Hardness of approximating problems on cubic graphs. Third
Italian Conference on Algorithms and Complexity, Lecture Notes in Computer Science
1203, Berlin: Springer.
Arora, S. & Lund, C. (1996) Hardness of approximations. In Approximation algorithms for
NP-hard problems, ed. D. Hochbaum. Boston: PWS.
Bellman, R.E. (1961) Adaptive control processes: a guided tour. Princeton: Princeton
University Press.
Basu, S. & Micchelli, C.A. (1998) Parametric density estimation for the classification of
acoustic feature vectors in speech recognition. Nonlinear Modeling, eds. J. Suykens and
J. Vandewalle. Boston: Kluwer.
Cheeseman, P. & Stutz, J. (1995) Bayesian Classification (AutoClass): Theory and Results.
Advances in Knowledge Discovery and Data Mining, eds. U. Fayyad, G. Piatetsky-
Shapiro, P. Smyth and & R. Uthurusamy. Menlo Park, CA: AAAI Press.
Chickering, D.M. (1996) Learning Bayesian networks is NP-complete. Learning from data:
AI and Statistics V, 121-130. New York: Springer.
Chow, C.K. & Liu, C.N. (1968) Approximating discrete probability distributions with de-
pendence trees. IEEE Transactions on Information Theory, 14:462-467.
Cooper, G. (1990) The computational complexity of probabilistic inference using Bayesian
belief networks. Artificial Intelligence, 42:393-405.
Cover, T.M. & Thomas, J.A. (1991) Elements of information theory. New York: John
Wiley.
Literature cited 94
Dagum, P. & Luby, M. (1993) Approximate probabilistic reasoning in Bayesian belief net-
works is NP-hard. Artificial Intelligence, 60:141-153.
Dasgupta, S. (1997) The sample complexity of learning fixed-structure Bayesian networks.
Machine Learning, 29:165-180. Boston: Kluwer.
Dasgupta, S. & Gupta, A. (1999) An elementary proof of the Johnson-Lindenstrauss lemma.
Technical Report 99-006, International Computer Science Institute, Berkeley.
Dempster, A.P., Laird, N.M. & Rubin, D.B. (1977) Maximum-likelihood from incomplete
data via the EM algorithm. J. Royal Statist. Soc. Ser. B, 39:1-38.
Diaconis, P. & Freedman, D. (1984) Asymptotics of graphical projection pursuit. Annals
of Statistics, 12:793-815.
Duda, R.O. & Hart, P.E. (1973) Pattern Classification and Scene Analysis. New York:
John Wiley.
Dudley, R.M. (1979) Balls in Rk do not cut all subsets of k + 2 points. Advances in
Mathematics, 31:306-308.
Durrett, R. (1996) Probability: Theory and Examples. Belmont, CA: Duxbury.
Edmonds, J. (1967) Optimum branchings. Journal of Research of the National Bureau of
Standards, 71B(4):233-240.
Feder, T. & Greene, D. (1988) Optimal algorithms for approximate clustering. ACM Sym-
posium on Theory of Computing.
Feller, W. (1966) An Introduction to Probability Theory and its Applications, vol. II. New
York: Wiley.
Frankl, P. & Maehara, H. (1988) The Johnson-Lindenstrauss lemma and the sphericity of
some graphs. Journal of Combinatorial Theory Ser. B, 44:355-365.
Friedman, N. (1997) Learning Bayesian networks in the presence of missing values and
hidden variables. International Conference on Machine Learning.
Freund, Y. & Mansour, Y. (1999) Estimating a mixture of two product distributions. ACM
Conference on Computational Learning Theory.
Geiger, D., Paz, A. & Pearl, J. (1990) Learning causal trees fron dependence informa-
tion. Proceedings of the Eighth National Conference on Artificial Intelligence, 770776.
Cambridge: AAAI Press.
Gonzalez, T.F. (1985) Clustering to minimize the maximum intercluster distance. Theoret-
ical Computer Science, 38:293-306. North-Holland.
Gupta, A. (1999) Embedding tree metrics into low dimensional Euclidean spaces. ACM
Literature cited 95