A Novel Generative Encoding For Exploiti
A Novel Generative Encoding For Exploiti
...
value y
y
f at x,y
...
are protected and have time to optimize their structure be- 0.5,1 1,1
fore competing with other niches in the population. NEAT 1,1 0,1 1,1 ... 3) Output is weight
uses the historical markings on genes to determine to which Substrate between (x1,y1) and (x2,y2)
species different individuals belong. Figure 3: Hypercube-based Geometric Connectivity
Third, NEAT begins with a uniform population of simple Pattern Interpretation. A grid of nodes, called the sub-
networks with no hidden nodes, differing only in their ini- strate, is assigned coordinates such that the center node
tial random weights. Speciation protects new innovations, is at the origin. 1. Every potential connection in the sub-
allowing diverse topologies to gradually complexify over evo- strate is queried to determine its presence and weight;
lution. Thus, NEAT can start minimally, and grow the nec- the dark directed lines in the substrate in the figure rep-
essary structure over generations. A similar process of grad- resent a sample of connections that are queried. 2. For
ually adding new genes has been confirmed in natural evolu- each query, the CPPN takes as input the positions of the
tion [10]. Through complexification, high-level features can two endpoints and (3) outputs the weight of the connec-
be established early in evolution and then elaborated and tion between them. Thus, connective CPPNs can produce
refined as new genes are added. regular patterns of connections in space.
For these reasons, in this paper the NEAT method is used returns a weight for every connection between every node
to evolve increasingly complex CPPNs. CPPN-generated in the grid, including recurrent connections. By conven-
patterns evolved with NEAT exhibit several essential mo- tion, a connection is not expressed if the magnitude of its
tifs and properties of natural phenotypes [14, 16]. If such weight, which may be positive or negative, is below a min-
properties could be transferred to evolved connectivity pat- imal threshold wmin . Magnitudes above this threshold are
terns, the representational power of CPPNs could poten- scaled to be between zero and a maximum magnitude in the
tially evolve large-scale ANNs and other graph structures, substrate. That way, the pattern produced by the CPPN
as explained in the next section. can represent any network topology (figure 3).
3. HyperNEAT The connectivity pattern produced by a CPPN in this way
is called the substrate so that it can be verbally distinguished
The spatial patterns in Section 2.1 present a challenge: from the CPPN itself, which has its own internal topology.
How can such spatial patterns describe connectivity? This Furthermore, in the remainder of this paper, CPPNs that
section explains how CPPN output is effectively interpreted are interpreted to produce connectivity patterns are called
as a connectivity pattern rather than a spatial pattern. Fur- connective CPPNs while CPPNs that generate spatial pat-
thermore, this novel interpretation allows neurons, sensors, terns are called spatial CPPNs. This paper focuses on neural
and effectors to exploit meaningful geometric relationships. substrates produced by connective CPPNs.
The next section introduces the key insight, which is to as- Because the CPPN is a function of four dimensions, the
sign connectivity a geometric interpretation. two-dimensional connectivity pattern expressed by the CPPN
3.1 Geometric Connectivity Patterns is isomorphic to a spatial pattern embedded in a four-di-
The main idea is to input into the CPPN the coordinates mensional hypercube. Thus, because CPPNs generate reg-
of the two points that define a connection rather than in- ular spatial patterns (Section 2.1), by extension they can be
putting only the position of a single point as in Section 2.1. expected to produce geometric connectivity patterns with
The output is then interpreted as the weight of the connec- corresponding regularities. The next section demonstrates
tion rather than the intensity of a point. This way, connec- this capability.
tions are defined in terms of the locations that they connect,
thereby taking into account the network’s geometry. 3.2 Producing Regular Connectivity Patterns
For example, consider a 5×5 grid of nodes. The nodes are Simple, easily-discovered substructures in the connective
assigned coordinates corresponding to their positions within CPPN produce connective motifs in the substrate. The key
the grid (labeled substrate in figure 3), where (0, 0) is the difference between connectivity patterns and spatial pat-
center of the grid. Assuming that these nodes and their po- terns is that each discrete unit in a connectivity pattern has
sitions are given a priori, a geometric connectivity pattern is two x values and two y values. Thus, for example, symmetry
produced by a CPPN that takes any two coordinates (source along x can be discovered simply by applying a symmetric
and target) as input, and outputs the weight of their connec- function (e.g. Gaussian) to either x1 or x2 (figure 4a).
tion. The CPPN is queried in this way for every potential The human brain is roughly symmetric at a gross resolu-
connection on the grid. Because the connection weights are tion, but its symmetry is imperfect. Thus, imperfect sym-
thereby a function of the positions of their source and target metry is an important structural motif in ANNs. Connec-
nodes, the distribution of weights on connections through- tive CPPNs can produce imperfect symmetry by composing
out the grid will exhibit a pattern that is a function of the symmetric functions of one axis along with asymmetric coor-
geometry of the coordinate system. dinate frames such as the axis itself. In this way, the CPPN
Such a CPPN in effect computes a four-dimensional func- produces varying degrees of imperfect symmetry (figure 4b).
tion CP P N (x1 , y1 , x2 , y2 ) = w, where the first node is at Another important motif in biological brains is repeti-
(x1 , y1 ) and the second node is at (x2 , y2 ). This formalism tion, particularly repetition with variation. Just as sym-
II II II II II II II II O 1 2
O
1 2 3 4 5 6 7 8 I I
O O
8 3
1 2 I I
8 3
7 4 I I
7 4
6 5 O O
I I
6 5
O O O O O O O O O O
(a) Sym. (b) Imperf. (c) Repet. (d) Var.
(a) Robot (b) Parallel (c) Concentric
Figure 4: Connectivity Patterns Produced by Connec-
tive CPPNs. These patterns, produced through inter- Figure 6: Placing Inputs and Outputs. Unlike tradi-
active evolution, exhibit important connectivity motifs: tion ANN representations, neurons on the substrate ex-
(a) bilateral symmetry, (b) imperfect symmetry, (c) rep- ist at defined locations in space that are accessible to the
etition, and (d) repetition with variation. That these learning algorithm. The figure depicts a robot (a) with
fundamental motifs are compactly represented and eas- eight radar sensors along its circumference and eight mo-
ily produced suggests the power of this type of encoding. tion effectors set at the same angle. In (b), the inputs
(labeled I ) and outputs (labeled O) are placed such at
their location along x determines whether they represent
a corresponding direction. In (c), the inputs and outputs
are laid out literally according to the eight directions in
(a) Grid (b) Circular space. Both arrangements create a geometric relation-
ship between each input and its corresponding output.
Figure 5: Alternative Substrate Configurations. There
Sigmoid
exist many potential substrate configurations. This fig-
ure shows (a) a grid and (b) a concentric configuration.
Different configurations are likely suited to problems Sigmoid Gaussian
metric functions produce symmetry, periodic functions such (a) 5 × 5 Concept (b) 7 × 7 Concept (c) CPPN
as sine produce repetition (figure 4c). Patterns with vari-
Figure 7: Equivalent Connectivity Concept at Differ-
ation are then produced by composing a periodic function
ent Substrate Resolutions. A connectivity concepts is
with a coordinate frame that does not repeat, such as the
depicted that was evolved through interactive evolution.
axis itself (figure 4d). Repetitive patterns can also be pro-
The CPPN that generates the concept at (a) 5 × 5 and
duced in connectivity as functions of invariant properties
(b) 7 × 7 is shown in (c). This illustration shows that
between the two nodes, such as connection length. Thus,
CPPNs represent a mathematical concept rather than a
symmetry, imperfect symmetry, repetition, and repetition
single structure. Thus, the same connective CPPN can
with variation, key structural motifs in biological brains,
produce patterns with the same underlying concept at
are compactly represented and therefore easily discovered
different substrate resolutions (i.e. node densities).
by connective CPPNs.
puts’ and outputs’ geometry, they can use this information
3.3 Substrate Configuration to their advantage.
CPPNs produce connectivity patterns among nodes on There is room to be creative and try different configu-
the substrate by querying the CPPN for each pair of points rations with different geometric advantages. For example,
in the substrate to determine the weight of the connection figure 6 depicts two methods in which the inputs and out-
between them. The layout of these nodes can take forms puts of a circular robot can be configured, both of which
other than the planar grid (figure 5a) discussed thus far. create an opportunity to exploit a different kind of geomet-
Different such substrate configurations are likely suited to ric relationship.
different kinds of problems.
For example, the nodes need not be distributed in a grid. 3.5 Substrate Resolution
The nodes within a substrate that controls a radial entity As opposed to encoding a specific pattern of connections
such as a starfish might be best laid out with radial geometry among a specific set of nodes, connective CPPNs in effect
(figure 5b) so that the connectivity pattern can be situated encode a general connectivity concept, i.e. the underlying
with perfect polar coordinates. mathematical relationships that produce a particular pat-
tern. The consequence is that same connective CPPN can
3.4 Input and Output Placement represent an equivalent concept at different resolutions (i.e.
Part of substrate configuration is determining which nodes node densities). Figure 7 shows a connectivity concept at
are inputs, which are outputs, and which are hidden. The different resolutions.
flexibility to assign inputs and outputs to specific coordi- For neural substrates, a significant implication is that the
nates in the substrate creates a powerful opportunity to ex- same ANN can be generated at different resolutions. With-
ploit geometric relationships advantageously. out further evolution, previously-evolved connective CPPNs
In many ANN applications, the inputs are drawn from a can be re-queried to specify the connectivity of the substrate
set of sensors that exist in a geometric arrangement in space. at a new, higher resolution, thereby producing a working
Because connective CPPN substrates are aware of their in- solution to the same problem at a higher resolution! This
operation, i.e. increasing substrate resolution, introduces a orientation at all times, that is, its north-facing side always
powerful new kind of complexification to ANN evolution. It faces north and never rotates. Internally, the robot also
is an interesting question whether, at a high level of abstrac- contains a set of n effectors. Each effector, when activated,
tion, the evolution of brains in biology in effect included causes the robot to move in one of n directions. Thus, there
several such increases in density on the same connectivity is one effector for each sensor that points in the same direc-
concept. Not only can such an increase improve the imme- tion. The robot’s objective is to go to the food.
diate resolution of sensation and action, but it can provide The interpretation of effector outputs constrains the prob-
additional substrate for increasingly intricate local relation- lem and the potential solutions. For the experiments in this
ships to be discovered through further evolution. paper, the motion vector resulting from effector output is
interpreted to incentivize HyperNEAT to find holistic solu-
3.6 Computational Complexity tions, i.e. solutions that do not only require a single connec-
Querying every potential connection in the substrate is tion. The robot moves in the direction corresponding to the
realistic for modern computers. For example, a CPPN gen- largest effector output. In the case of a tie, the robot moves
erating a substrate with 250,000 potential connections is in the direction of the first tied output in sampling order.
queried 250,000 times, which can be computed e.g. in 4.64 The robot’s speed s is determined by
omax
seconds on a 3.19 Ghz Pentium 4. Note that this substrate s = (smax omax )( ), (1)
otot
is an enormous ANN with up to a quarter-million connec-
tions. Connective CPPNs present an opportunity to evolve where smax is the maximum possible speed, omax is the
structures of a complexity and functional sophistication gen- maximum output, and otot is the sum of all outputs. The
uinely commensurate with available processing power. first term correlates speed with output, such that to go the
maximum speed, the robot must maximize the output cor-
3.7 Evolving Connective CPPNs responding to the direction of the food. The second term
The approach in this paper is to evolve connective CPPNs encourages the robot to excite a single output by penalizing
with NEAT. This approach is called HyperNEAT because it for activating more than one at a time. Furthermore, out-
NEAT evolves CPPNs that represent spatial patterns in hy- puts have sigmoidal activation, which means that if their
perspace. Each point in the pattern, within a hypercube, is input is zero, they will output 0.5. Thus, the robot also
interpreted as a connection in a lower-dimensional connected needs to inhibit effectors that point in the wrong direction
graph. NEAT is the natural choice for evolving CPPNs because they will otherwise slow down motion in the cho-
because it evolves increasingly complex network topologies. sen direction. Thus, while diverse solutions still work in
Therefore, as CPPNs complexify, so do the regularities and this domain, many are not optimal in terms of speed. The
elaborations (i.e. the global dimensions of variation) that best solutions require a correct pattern connecting to all the
they represent in their corresponding connectivity pattern. outputs from all the sensors.
Thus, HyperNEAT is a powerful new approach to evolv- Each robot attempts r trials, where r is twice the reso-
ing large-scale connectivity patterns and ANNs. The next lution; thus higher resolutions are evaluated on more trials.
section describes initial experiments that demonstrate how For each trial a single piece of food is placed 100 units away
HyperNEAT can exploit input and output geometry. from the robot at either the center of a sensor or the bor-
der between two sensors. Each trial tests a different such
4. FOOD GATHERING EXPERIMENT location. If a robot is not able to get food for a particular
If sensors and outputs are placed such that they respect trial after 1000 ticks, its trial ends. Individuals are evalu-
regularities in the outside world, HyperNEAT can discover ated based on their amount of food collected and the average
those regularities through connective CPPNs and exploit speed at which they obtain each item:
fc ttot
them to solve the problem. For example, HyperNEAT can f itness = (10000 ) + ( ), (2)
discover that the way one reacts to stimulus on the left r 1000r
is related to the way one react to similar stimulus on the where fc is the total number of food collected and ttot is the
right. In this way, HyperNEAT can solve problems with total time spent on all trials.
high-dimensional input because it does not need to learn This task is a good proof of concept because it requires
the meaning of each sensor independently. discovering the underlying regularity of the domain: Two
The experiments in this paper demonstrate this capabil- nodes at the same angle (i.e. the sensor and effector) should
ity and its implications through a food gathering task. This be connected and the others inhibited. If this concept is
task was chosen for its simplicity as a proof-of-concept; it discovered, the task is effectively trivial. However, direct
effectively isolates the issue of sensor and output placement, encodings would need to discover the connection between
which is the primary topic of this paper. Furthermore, al- each pair independently. That is, they cannot discover the
though the task is simple, at high resolutions (i.e. with large underlying concept that describes the solution. In contrast,
numbers of sensors and effectors) it becomes a difficult op- HyperNEAT can discover the general concept. Demonstrat-
timization problem because of its increasing dimensionality. ing this fact helps to establish the promise of HyperNEAT
In the experiments, two different sensor placement arrange- in more complex domains.
ments are compared that both present a chance to exploit
regularity in different ways. 4.1 Sensor Placement
The food gathering domain works as follows. A single HyperNEAT makes it possible to decide how the sensors
piece of food is placed within a square room with a robot at and effectors should be placed in the substrate to allow the
the center (figure 6a). A set of n rangefinder sensors, placed concept to be discovered. In general, the geometry of the
at regular angular intervals, encircle the robot’s perimeter. placement scheme should reflect the correlation between sen-
The robot has a compass that allows it to maintain the same sors and effectors. Two different configurations are tested:
(1)Two parallel lines of nodes (figure 6b). The top faults and prior reported settings for NEAT [18, 20, 17].
row of sensors are placed in clockwise order starting from They were found to be robust to moderate variation through
the sensor closest to 45◦ above west. In the bottom row, preliminary experimentation.
effectors are placed in the same order. In this way, the key
regularity is geometrically expressed as two nodes being in 5. RESULTS
the same column. All sensor configurations were able to collect food at all
(2)Two concentric circles of nodes (figure 6c). The positions within the first few generations except unbiased
inner circle (radius .5) is the sensors and the outer circle concentric, which took on average 33 generations to learn
(radius 1) is the effectors. The nodes are placed at the same how to get food at every position. Thus, for most configura-
angle as they exist in the robot. In this layout, the key tions, the main challenge was to learn to get food efficiently.
regularity is captured by shared angle. It is also interesting The performance measure in this section is thus the average
because the sensors and effectors are placed exactly in the time (i.e. number of ticks) it takes the robot to get a piece
shape of the robot, an intuitive scheme that would not be of food over all its trials. Robots that cannot get the food
meaningful to traditional methods. in a trial are given the maximum time 1,000 for that trial.
Results are averaged over 20 runs.
4.2 Additional Geometric Bias Figure 8a shows how performance improved over 500 gen-
Connective CPPNs already provide a strong geometric erations for both placement schemes, with and without the
bias, but they also make possible injecting additional a pri- length-input geometric bias. Parallel placement on average
ori knowledge about the problem into the search. In biology, evolved significantly faster strategies (p < .05) than concen-
distance is a significant bias simply because in the physical tric. This disparity is explained by a more complex rela-
world it is easier to connect to something closer than farther tionship, in Cartesian coordinates, between corresponding
away. Thus, if a CPPN is given connection length as an in- nodes in the concentric case. However, the geometric bias
put, related concepts placed close together on the substrate significantly increased performance of both methods after
can be treated specially by the CPPN. the fourth generation (p < .01). Furthermore, the geomet-
In both placement schemes, sensors should excite their ric bias is so useful that it erases the difference between
nearest neighbor effector and inhibit those farther away. the two schemes, causing both to perform similarly when
While CPPNs have the capability to discover the concept of present. Thus, parallel placement is easier to exploit for
distance themselves, they may solve tasks more efficiently if HyperNEAT except when the CPPN is provided connection
it is given. To explore this capability, a separate experiment length as input.
is performed on both substrate configurations in which the
CPPN receives an extra input representing the Euclidean 5.1 Scaling Performance
distance between the two points being queried. As described in Section 4.3, after evolution at resolution
eight is completed, generation champions are reevaluated by
4.3 Scaling doubling their resolution repeatedly without further evolu-
An important question is whether HyperNEAT will dis- tion. Individuals generally did retain the ability to collect
cover the key regularity and whether one configuration has food although there is some degradation in performance at
an advantage over another. Furthermore, if HyperNEAT each increment in resolution. To illustrate this degradation
does discover the underlying concept, then solutions should for different configurations, figure 8b shows the average dif-
scale to more sensors and effectors without further evolution ference in efficiency between resolution eight and resolution
simply by increasing the resolution of the substrate. 32; lower numbers imply better ability to scale.
Therefore, the number of inputs and outputs of each gen- Parallel placement scaled significantly more effectively than
eration champion from all runs of both configurations is concentric except in the earliest generations (p < .01). How-
doubled several times starting from the evolved 8 × 8 res- ever, as with efficiency, when concentric placement’s CPPN
olution. Each double-resolution substrate samples twice as was provided length as input, it scaled as well as parallel
many points as its predecessor by decreasing the angular placement did with length input. In both biased cases, scal-
sampling interval around the robot by half. Doubling for ing significantly improved over runs using concentric place-
each champion proceeds from the initial 8 × 8 resolution to ment without the geometric bias (p < .01), but not signifi-
a maximum size of 128×128, a 16-fold increase in resolution. cantly over unbiased parallel placement.
As figure 8b shows, unbiased concentric placement de-
4.4 Experimental Parameters graded significantly between resolution eight and 32; in fact
Because HyperNEAT differs from original NEAT only in individuals could no longer collect food in every position.
its set of activation functions, it uses the same parameters However, it turns out that information about the task was
[18]. Experiments were run using a modified version of the still retained implicitly at the higher resolution: When al-
public domain SharpNEAT package [7]. The size of each lowed to continue evolving at the higher resolution, solutions
population was 150 with 20% elitism. Reproduction had an that collect all the food were always found within five gen-
equal chance of being sexual or asexual. Asexual offspring erations (2.5 on average). On the other hand, when concen-
had .96 probability of link weight mutation, .03 chance of tric evolution is started from scratch at a lower resolution, it
link addition, and .01 chance of node addition. The coef- takes on average 33 generations to learn to get food on every
ficients for species similarity were 1.0 for nodes and con- trial. Thus, even when performance degrades significantly
nections, and 0.1 for weights. CPPN connection weights after scaling, the resultant individual still retains important
ranged between -3.0 and 3.0. Sigmoid, Gaussian, absolute geometric information that can be quickly tweaked to work
value, and sine nodes were created with equal probability. at the higher resolution.
Parameter settings are based on standard SharpNEAT de- Figure 8c shows the average absolute performance at dif-
700
8 Parallel Parallel
8 Parallel Biased 600 Parallel Biased
8 Concentric
400
400
300
300
200 200
100 100
0 0
0 50 100 150 200 250 300 350 400 450 500 0 50 100 150 200 250 300 350 400 450 500
Generation Generation
8 Parallel 8 Parallel
1400 16 Parallel 1400 16 Parallel
32 Parallel 32 Parallel
1200 128 Parallel 1200 64 Parallel
8 Concentric 128 Parallel
Average Number of Ticks
600 600
400 400
200 200
0 0
0 50 100 150 200 250 300 350 400 450 500 0 50 100 150 200 250 300 350 400 450 500
Generation Generation