IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 15, NO.

3, MAY 2004 613

A Novel Neural Network for

Nonlinear Convex Programming
Xing-Bao Gao

Abstract—In this paper, we present a neural network for solving such as application-specific integrated circuits, where the opti-
the nonlinear convex programming problem in real time by means mization procedure is truly parallel and distributed. Thus, the
of the projection method. The main idea is to convert the convex neural network approach can solve optimization problems in
programming problem into a variational inequality problem. Then
a dynamical system and a convex energy function are constructed running times at the orders of magnitude much faster than con-
for resulting variational inequality problem. It is shown that the ventional optimization algorithms executed on general-purpose
proposed neural network is stable in the sense of Lyapunov and digital computers.
can converge to an exact optimal solution of the original problem. Recently, many neural networks have been developed and
Compared with the existing neural networks for solving the non- investigated [3], [4], [6], [7], [9], [14], [17], [19], [21]–[24].
linear convex programming problem, the proposed neural network
has no Lipschitz condition, no adjustable parameter, and its struc- Among them, Kennedy and Chua [14] proposed a neural
ture is simple. The validity and transient behavior of the proposed network, which employs both gradient method and penalty-
neural network are demonstrated by some simulation results. function method for solving a nonlinear programming problem.
Index Terms—Convergence and stability, neural network, non- Because this network contains a penalty parameter and its energy
linear convex programming. function can be viewed as an “inexact” penalty function, it
generates approximate solution only and has an implementation
problem when the penalty parameter is large. To avoid using
I. INTRODUCTION
penalty parameters, Rodríguez-Vázquez et al. [17] proposed

W E consider the following convex programming problem: a class of optimization neural networks, and Glazos et al. [8]
performed a rigorous analysis of these dynamics for the convex
programming with inequality constraints. However, the right
(1)
side of the system in [17] is discontinuous and, to prove its
convergence, they must assume that the feasible region of
where , the problem is bounded and that the interior set of is
, and some (or ) could be nonempty. In order to solve the equilibrium models arising in
, and the fields of economics and transportation science, Friesz et al.
are convex on and continuously [6] proposed the globally projected dynamic systems, which
differentiable on an open convex set including can be viewed as a neural network model suitable for hardware
and . implementation [23]. Even though this system with a monotone
and symmetric mapping is asymptotically stable [7], [23], it
Problem (1) includes linear and convex quadratic program-
is unwise for using it to solve (1) since another nonlinear
ming and arises in a wide variety of scientific and engineering
convex programming problem with the same constraints must
fields [2] including regression analysis, function approxima-
be solved to compute the projection on its feasible region.
tion, signal processing, image restoration, parameter estimation,
Moreover, the asymptotical stability of this system with a
filter design, robot control, etc., a real-time solution is often de-
monotone and asymmetric mapping could not be guaranteed
sired. However, traditional numerical methods [1], [2], [5], [10],
[6], [23], [24]. In addition, to solve the following variational
[11], [12] might not be efficient for digital computers since the
inequality problem VI : find a such that
computing time required for a solution is greatly dependent on
the dimension and the structure of the problem and the com- (2)
plexity of the algorithm used. One promising approach to handle
these optimization problems with high dimensions and dense where is a continuous mapping from to and is
structure is to employ artificial neural network based on circuit a nonempty closed convex subset of , Xia and Wang [22]
implementation. Because of the dynamic nature for optimiza- proposed the following neural network:
tion and the potential of electronic implementation, neural net-
works can be implemented physically by designated hardware
(3)
Manuscript received June 12, 2002; revised April 22, 2003. This work was
supported in part by NSFC Grant 10071048. where is a scale parameter and is a projec-
The author is with the College of Mathematics and Information Science, tion operator defined by
Shaanxi Normal University, Xi’an, Shaanxi 710062, P.R. China (e-mail: xin-
[email protected]).
Digital Object Identifier 10.1109/TNN.2004.824425

1045-9227/04$20.00 © 2004 IEEE

614 IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 15, NO. 3, MAY 2004

This network has two-layer structure and is asymptotically II. A NEURAL NETWORK MODEL
stable when is monotone and , where In this section, we shall construct a neural network for solving
(4) (1) and show that the new model is simpler than the existing
ones. First, we give a sufficient and necessary condition for the
It is easy to see that there could not exist such an that the optimal solution of (1), which is the theoretical foundation for
condition (4) is satisfied for a nonlinear mapping , defined on us to design the network for it.
a unbounded set, so this condition may limit the applicability Theorem 1: is an optimal solution of (1) if and only if
of the model (3) to practical optimization problems and it is there exist and such that (7), shown at the
sometimes difficult to find . In particular, (1) can be solved bottom of the page.
by this model since it is equivalent to the variational inequality Proof: See the Appendix.
problem defined in (2) (see Remark 1 below). However, their Remark 1: Let
model needs to estimate the constant in (4) and its structure
is quite complex.
Based on the above considerations and in order to avoid using
and
the condition (4) and simplify the model’s structure, in this
paper we: 1) propose a new neural network for solving (1) by
means of the necessary and sufficient conditions of its optimal
solution; 2) define a convex energy function by introducing a then Theorem 1 indicates that (1) is equivalent to the variational
convex function; and 3) prove that the proposing neural net- inequality problem VI defined in (2).
work is stable in the sense of Lyapunov and can converge to an From (6) and (7), we can easily verify the following result.
exact optimal solution of the (1). The new model has a good sta- Lemma 1: is an optimal solution of (1) if and only if there
bility performance without condition (4), does not contain any exist and such that
adjustable parameter, and its structure is simple for (1), thus,
it avoids the difficulty to choose networks parameter and de- (8)
creases the network complexity in implementation.
Throughout this paper, we assume that (1) has a finite-optimal
solution and satisfies the Slater condition (see [1, p. 97]), i.e., where and
there exists a such that .
Lemma 1 indicates that the optimal solution of (1) can be
obtained by solving the system (8). For simplicity, we denote
(5)
A neural network is said to be stable in the sense of Lyapunov solves (8) , and
and asymptotically stable if the corresponding dynamic system
is so.
The following notations will be used in the later
Then the above result suggests the following dynamic system
discussion. We let denote the Euclidean norm,
for a neural network model to solve (1):
denote the
gradient vector of the differentiable function (9)
at ;
, and ,
where . A basic property of the projection mapping on a The architecture of neural network (9) is shown in Fig. 1,
closed convex set [15] is where vectors and are neural network’s outputs, vector
is the external input and the other parame-
(6) ters are defined by and . The pro-
jection operators and could be easily implemented
The rest of the paper is organized as follows. In Section II, a by using piecewise-activation functions [3], [4]. According to
neural network for solving (1) is constructed. The stability and Fig. 1, the circuit realizing the proposing neural network (9)
convergence of the proposing neural network are analyzed in consists of integrators, activation functions
Section III. Numerical simulations can be found in Section IV. for and processors for
Finally, we summarize the paper in Section V. and analog multipliers for , amplifiers, and summers.

(7)
GAO: NOVEL NEURAL NETWORK FOR NONLINEAR CONVEX PROGRAMMING 615

Fig. 1. Architecture of neural network (9).

Thus, the complexity of the proposing neural network depends where and
only on and of the original problem. , the model (9) has no parameter and does not contain
In contrast to model (3) for (1), which can be written in details the additional terms , and
as . Moreover, in terms of model complexity, we can
easily see that the complexity of (9) is about the same as the
first term of the right side of (10). Thus, the model (9) is simpler.
To clarify this issue further, we consider the following quadratic
programming problem:

(10)
(11)
616 IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 15, NO. 3, MAY 2004

where is symmetric and positive semidefinite, Lemma 3: Let be the function in (15). Then the
, and . Then, (9) and (10) for this following items are true:
problem can be simplified as i) is convex on and continuously differentiable
on ;
ii) for all ;
iii)
for all .
Proof: See the Appendix.
(12) The results in Lemma 3 pave a way for us to study the dy-
and namics of network (9). In particular, the neural network (9) has
the following basic property.
Theorem 2: If and are locally Lipschitz contin-
uous on , then for any , there exists a unique contin-
(13) uous solution of (9) with for all .
Proof: See the Appendix.
respectively, where is the identity matrix of order . In terms The results of Corollary 1 and Theorem 2 indicate that neural
of model complexity, it is easy to see that the total numbers network (9) is well defined. In the proof of Theorem 2, we see
of multiplications/divisions and additions/subtractions per iter- that if , then the solution of (9) will eventually move
ation for (12) are and , into and stays in from then on. Now we are ready to state
respectively. But for the model (13), the total numbers of multi- and prove the following stability result for (9).
plications/divisions and additions/subtractions per iteration are Theorem 3: Assume that and are locally Lips-
and , respectively. chitz continuous on an open convex set including ,
Thus, the asymptotic complexity of model (12) is about half of then neural network (9) is stable in the sense of Lyapunov and,
model (13) for (11). for any , the trajectory corresponding to (9) will con-
As a result of Lemma 1, we have the following result, which verge to a point in . In particular, neural network (9) with
describes the relationship between an equilibrium point of (9) is asymptotically stable when .
and a solution of (8). Proof: See the Appendix.
Corollary 1: if and only if is an equilibrium point Remark 2: If and are continuously differen-
of neural network (9). tiable on an open convex set including , then
and are locally Lipschitz continuous on and Theorems
III. STABILITY ANALYSIS 2 and 3 hold without the condition (4).
In this section, we shall study some stability and convergence Remark 3: From the above analysis, we see that
properties for (9). First, we prove the following lemma, which when and ( )
will be very useful in our later discussion. are convex and continuously differentiable on .
Lemma 2: The function is convex on and con- Remark 4: Let and
tinuously differentiable on . ; then the dual problem of (1), shown
Proof: Obviously, and for in the first equation at the bottom of the next page, can be
solved by neural network (9). In fact, if , then it is easy
to prove that
if is an optimal solution of
(14)
. the above dual problem.
Then, according to [1], the result is obtained from the differen-
tiable convexity of . This completes IV. ILLUSTRATIVE EXAMPLES
the proof. In this section, we give four examples to illustrate the theo-
Now, we can define the following function: retical results achieved in Section III and the effectiveness of
neural network (9). The simulation is conducted in Borland
C++ and the Euler method is used to solve the ordinary differ-
(15) ential equation. In all figures, the time unit is 1/18.2 s.
Example 1: Consider the following optimization problem
where is finite (since (1) [13]:
has a finite-optimal solution, Theorem 1 and Lemma 1 ensure
that there exists a finite ) and

(16)

Then and we have the following result which This problem satisfies Slater condition (5), has a unique optimal
explores some properties for . solution ,
GAO: NOVEL NEURAL NETWORK FOR NONLINEAR CONVEX PROGRAMMING 617

Fig. 2. Transient behavior of (9) in Example 1. Fig. 3. Transient behavior of (9) in Example 2.

, and , and
. .
It is easy to verify that the above problem is convex on Obviously, the above problem is convex on and does
and does not satisfy (4) since is nonlinear. not satisfy the condition (4) since is nonlinear.
From the analysis in Section III, we use neural network (9) to Let ; we use neural network (9) to solve
solve this problem on . All simula- this problem. All simulation results show that neural
tion results show that neural network (9) is asymptotically stable network (9) is asymptotically stable at , where
at , where and . As an example,
Fig. 2 shows the trajectories of neural network (9), with the ini- . For example, Fig. 3 shows that the trajec-
tial point converge to . tories of neural network (9) with the initial point
Example 2: Consider the following optimization problem
with linear constraints [13]: converge to .
Example 3: Consider the following optimization problem:

where This problem satisfies Slater condition (5), has infinite

number of optimal solutions
(see the equation at the
bottom of the page.) This problem has a unique optimal solution
618 IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 15, NO. 3, MAY 2004

Fig. 4. Transient behavior of (9) in Example 3. (a) The first initial point (n = 3). (b) The second initial point (n = 3). (c) n = 500. (d) n = 1000.

and for
.
Obviously, the above problem is convex on and does
not satisfy the condition (4) since is nonlinear. From
Remark 3, we use neural network (9) to solve this problem on
. All simulation results show that neural network (9)
converges to one of its equilibrium points , where
for . For example, Fig. 4(a) and (b)
shows the trajectories of neural network (9) with different
initial points
and converge to
and

, respectively, when , and Fig. 4(c) and (d) Fig. 5. Transient behavior of (9) in Example 4.
shows the first 14 trajectories of neural network (9) with
a random initial point This problem satisfies Slater condition (5) and has unique op-
and , when and timal solutions
, respectively. , and
To illustrate further the efficiency of neural network (9), we .
make the comparisons with the existing models in [6]and [14] It is easy to see that the above problem is convex on and
through the following example. does not satisfy the condition (4) since is nonlinear on
Example 4: Consider the following optimization problem: . Let ; we still use neural network (9) to solve this
problem. All simulation results show that neural network (9) is
asymptotically stable at , where . As an example,
Fig. 5 shows the trajectories of neural network (9) with the initial
point converge to .
GAO: NOVEL NEURAL NETWORK FOR NONLINEAR CONVEX PROGRAMMING 619

Fig. 6. Transient behavior of (17) in Example 4.

To make a comparison, we solve the above problem by using

the globally projected dynamic systems [6] given by

(17)

But Fig. 6 shows this model with the initial point

is not stable. Moreover, we solve the above problem by using
Kennedy and Chua’s neural network [14] given by

(18)
Fig. 7. Transient behavior of (18) in Example 4. (a)  = 5000. (b)  =
where is a penalty parameter, and 10 000.
. Even
though the simulation results show that neural network (18) APPENDIX
converges to its equilibrium point ,
Proof of Theorem 1: Consider the following Lagrange
which can be viewed as the approximate solution of the above
function of (1):
problem when is large, this equilibrium point is not feasible.
For example, Fig. 7(a) and (b) shows the trajectories of neural
network (18) with the initial point converge to
and when and which is defined on . Since (1) is convex
, respectively. Thus, we can conclude that the proposing and satisfies (5), by the Kuhn-Tucker theorem [1], we know that
network is feasible and has a good stability performance from is an optimal solution of (1) if and only if there exist
the above simulation results. and such that is a saddle point of
on . That is
V. CONCLUSION
In this paper, we have presented a neural network for solving (19)
the nonlinear convex programming problem in real time by con-
verting it into a variational-inequality problem. In contrast to By the left side of (19), we have
the existing neural networks, the proposed neural network does
not require any adjustable parameter and its structure is simple.
Thus, the new model is more suitable to be implemented in hard-
ware. Furthermore, we have shown that the proposed neural net-
Using the fact that this result must hold for all
work is stable in the sense of Lyapunov and can converge to an
, it follows that:
exact solution of the original problem. Our stability condition
has no condition (4) and, thus, the obtained results improve the and
existing ones. Since nonlinear convex programming problem
has wide applications, the new model and the obtained results and , we know that for all
in this paper are significant in both theory and application. The . Thus, by the right side of (19) we have
simulation results clearly demonstrate the convergence behavior
and the characteristics of the proposed network.
620 IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 15, NO. 3, MAY 2004

It follows from setting at the above inequality that where the last step follows by (7) since

Therefore (7) is obtained. This completes the proof.

Proof of Lemma 3: i) Since is differentially from substituting
convex on defined in (16) is also convex on and and in (6). From the differentiable convexities of
continuously differentiable on from Lemma 2 and and , we have
the differentiable convexity of . Thus, i) is obtained by (15).
ii) According to [1] and the above analysis of i), we have (21)

Thus, , for all by (15). Therefore, iii) is obtained by (20), (21),

iii) From Lemma 2 and (14), we have and . This completes the proof.
Proof of Theorem 2: From (6), we know that the projec-
tion operators and are nonexpansive. Since
and are locally Lipschitz continuous on , it is easy
to prove that is also locally Lipschitz continuous on
. Thus, from Theorem 1 in [9], for any , there
exists a unique continuous solution for (9) with
From i), (15), and (16), it follows that: and is its maximal interval of existence.
Substituting and in (6), we have

Thus by (9) and Lemma 4.7 in [19], , we have

Thus, by (8), (9), and ,

there is

It follows from that:

That is to say for all .

From Lemma 3 ii) and iii), it is easy to see that the set
is bounded [16], and

This implies that is bounded and,

thus, . This completes the proof.
Proof of Theorem 3: From the assumptions and Theorem
2, , system (9) has a unique continuous solution
with for all .
From Lemma 3 iii) and (9), , we have

(22)
This inequality and Lemma 3 ii) indicate that is a Lya-
punov function of (9). From Theorem 3.2 in [18], (9) is Lya-
(20) punov stable.
GAO: NOVEL NEURAL NETWORK FOR NONLINEAR CONVEX PROGRAMMING 621

From the proof of Theorem 2, is bounded. [7] X. B. Gao, “Exponential stability of globally projected dynamic sys-
Thus, there exist a limit point and a sequence with tems,” IEEE Trans. Neural Networks, vol. 14, pp. 426–431, Mar. 2003.
[8] M. P. Glazos, S. Hui, and S. H. Źak, “Sliding modes in solving convex
and as programming problems,” SIAM J. Contr. Optimizat., vol. 36, no. 2, pp.
such that 680–697, Mar. 1998.
[9] Q. M. Han, L.-Z. Liao, H. D. Qi, and L. Q. Qi, “Stability analysis of gra-
(23) dient-based neural networks for optimization problem,” J. Global Opti-
mizat., vol. 19, no. 1, pp. 363–381, 2001.
[10] P. T. Harker and J. S. Pang, “Finite-dimensional variational inequality
Equation (23) indicates that is an -limit point of and nonlinear complementarity problems: A survey of theory, algo-
. rithms, and applications,” Math. Programming, vol. 48B, pp. 161–220,
1990.
From the LaSalle Invariant Set Theorem [18, Theorem 3.4], [11] B. S. He and L. Z. Liao, “Improvements of some projection methods
we know that as , where is the largest for monotone nonlinear variational inequalities,” J. Optimizat. Theory
invariant set in . From (9) Applicat., vol. 112, no. 1, pp. 111–128, 2002.
[12] B. S. He and J. Zhou, “A modified alternating direction method for
and (22), it follows that . convex minimization problems,” Appl. Math. Lett., vol. 13, pp. 123–130,
From this result and Corollary 1, we know that . Thus, 2000.
by and (23). [13] W. Hock and K. Schittkowski, Test Examples for Nonlinear Program-
ming Codes. New York: Springer-Verlag, 1981.
Substituting in (15), similar to the analysis above, we [14] M. P. Kennedy and L. O. Chua, “Neural networks for nonlinear program-
can prove that the function is monotonically nonin- ming,” IEEE Trans. Circuits Syst. , vol. 35, pp. 554–562, May 1988.
creasing in , and for all . [15] D. Kinderlehrer and G. Stampacchia, An Introduction to Variational In-
equalities and Their Applications. New York: Academic, 1980.
From the continuity of and (23), , there exists a [16] T. M. Ortega and W. C. Rheinboldt, Iterative Solution of Nonlinear
such that Equation in Several Variables. New York: Academic, 1970.
[17] A. Rodríguez-Vázquez, R. Domínguez-Castro, J. L. Huertas, and E.
when Sánchez-Sinencio, “Nonlinear switched-capacitor ‘neural networks’
for optimization problems,” IEEE Trans. Circuits Syst. , vol. 37, pp.
Thus 384–397, Mar. 1990.
[18] J.-J. E. Slotine and W. Li, Applied Nonlinear Control. Englewood
Cliffs, NJ: Prentice-Hall, 1991.
[19] T. E. Smith, T. L. Friesz, D. H. Bernstein, and Z. G. Suo, “A compara-
when tive analysis of two minimum-norm projective dynamics and their rela-
tionship to variational inequalities,” in Complementarity and Variational
That is, . Problems: State of Art, M. C. Ferris and J. S. Pang, Eds. Philadelphia,
In particular, if , then for each , the so- PA: SIAM, 1997, pp. 405–424.
lution with will approach to by the analysis [20] M. V. Solodov and P. Tseng, “Modified projection-type methods for
monotone variational inequalities,” SIAM J. Contr. Optimizat., vol. 34,
above. So neural network (9) with is asymptotically pp. 1814–1830, 1996.
stable. This completes the proof. [21] A. Wu and P. K. S. Tam, “A neural network methodology and strategy
of quadratic optimization,” Neural Computing Applicat., vol. 8, pp.
283–289, 1999.
ACKNOWLEDGMENT [22] Y. S. Xia and J. Wang, “A general methodology for designing glob-
ally convergent optimization neural networks,” IEEE Trans. Neural Net-
The author would like to thank Prof. L. Z. Liao for many works, vol. 9, pp. 1331–1343, Nov. 1998.
helpful discussions. The author is also very grateful to the Asso- [23] Y. S. Xia and J. Wang, “On the stability of globally projected dynamical
ciate Editor and four anonymous reviewers for their comments systems,” J. Optimizat. Theory Applicat., vol. 106, no. 1, pp. 129–160,
2000.
and constructive suggestions on an earlier version of this paper. [24] Y. S. Xia and J. Wang, “Global asymptotic and exponential stability of
a dynamic neural system with asymmetric connection weights,” IEEE
REFERENCES Trans. Automat. Contr., vol. 46, pp. 635–638, Apr. 2001.

[1] M.Avriel, NonlinearProgramming: AnalysisandMethods. Englewood

Cliffs, NJ: Prentice-Hall, 1976.
[2] M. S. Bazaraa, H. D. Sherali, and C. M. Shetty, Nonlinear Programming-
Theory and Algorithms, 2nd ed. New York: Wiley, 1993.
[3] A. Bouzerdorm and T. R. Pattison, “Neural network for quadratic opti- Xing-Bao Gao received the B.S. and M.S. degrees
mization with bound constraints,” IEEE Trans. Neural Networks, vol. 4, in mathematics from Shaanxi Normal University,
pp. 293–304, Mar. 1993. Shaanxi, China, in 1988 and 1991, respectively,
[4] A. Cichocki and R. Unbehauen, Neural Networks for Optimization and and the Ph.D. degree in applied mathematics from
Signal Processing. New York: Wiley, 1993. Xidian University, Shaanxi, China, in 2000.
[5] R. Fletcher, Practical Methods of Optimization. New York: Wiley, His current research interests include optimization
1981. theory and applications, neural networks, and evolu-
[6] T. L. Friesz, D. H. Bernstein, N. J. Mehta, R. L. Tobin, and S. Ganjl- tionary computations.
izadeh, “Day-to-day dynamic network disequilibria and idealized trav-
eler information systems,” Operat. Res., vol. 42, pp. 1120–1136, 1994.