|
In a great variety of neuron models neural inputs are combined using
the summing operation. We introduce the concept of multiplicative
neural networks which contain units that multiply their inputs instead
of summing them and, thus, allow inputs to interact nonlinearly. The
class of multiplicative networks comprises such widely known and well
studied network types as higher-order networks and product unit
networks.
We investigate the complexity of computing and learning for
multiplicative neural networks. In particular, we derive upper and
lower bounds on the Vapnik-Chervonenkis (VC) dimension and the pseudo
dimension for various types of networks with multiplicative units. As
the most general case, we consider feedforward networks consisting of
product and sigmoidal units, showing that their pseudo dimension is
bounded from above by a polynomial with the same order of magnitude as
the currently best known bound for purely sigmoidal networks.
Moreover, we show that this bound holds even in the case when the unit
type, product or sigmoidal, may be learned. Crucial for these results
are calculations of solution set components bounds for new network
classes. As to lower bounds we construct product unit networks of
fixed depth with superlinear VC dimension.
For higher-order sigmoidal networks we establish polynomial bounds
that, in contrast to previous results, do not involve any restriction
of the network order. We further consider various classes of
higher-order units, also known as sigma-pi units, characterized by
connectivity constraints. In terms of these we derive some
asymptotically tight bounds.
Multiplication plays an important role both in neural modeling of
biological behavior and in applications of artificial neural
networks. We also briefly survey research in biology and in
applications where multiplication is considered an essential
computational element. The results we present here provide new tools
for assessing the impact of multiplication on the computational power
and the learning capabilities of neural networks.
|
