Complex-Valued Neural Networks with Multi-Valued Neurons
Complex-Valued Neural Networks with Multi-Valued Neurons
Igor Aizenberg, Springer, June 2011
Springer Book Series "Studies in Computational Intelligence", vol. 353
Complex-Valued Neural Networks have higher functionality, learn faster and generalize better than their real-valued counterparts.
This book is devoted to the Multi-Valued Neuron (MVN) and MVN-based neural networks. It contains a comprehensive observation of MVN theory, its learning, and applications. MVN is a complex-valued neuron whose inputs and output are located on the unit circle. Its activation function is a function only of argument (phase) of the weighted sum. MVN derivative-free learning is based on the error-correction rule. A single MVN can learn those input/output mappings that are non-linearly separable in the real domain. Such classical non-linearly separable problems as XOR and Parity n are the simplest that can be learned by a single MVN. Another important advantage of MVN is a proper treatment of the phase information.
These properties of MVN become even more remarkable when this neuron is used as a basic one in neural networks. The Multilayer Neural Network based on Multi-Valued Neurons (MLMVN) is an MVN-based feedforward neural network. Its backpropagation learning algorithm is derivative-free and based on the error-correction rule. It does not suffer from the local minima phenomenon. MLMVN outperforms many other machine learning techniques in terms of learning speed, network complexity and generalization capability when solving both benchmark and real-world classification and prediction problems. Another interesting application of MVN is its use as a basic neuron in multi-state associative memories.
The book contains a comprehensive observation of MVN, MLMVN, mechanisms of their learning, and their important advantages over real-valued neurons and neural networks.
The book also contains many examples of using MVN and MLMVN for solving classification and prediction problems. Along with a number of classical benchmark problems, some interesting real-world problems are considered, for example, identification of blur for blurred image restoration and financial time series prediction.
The book is addressed to those readers who develop theoretical fundamentals of neural networks and use neural networks for solving various real-world problems. It should also be very suitable for Ph.D. and graduate students pursuing their degrees in computational intelligence.
· Order the book 2011, Hardcover, 264 p., 258 illus. ISBN 978-3-642-20352-7
The book preview is available from
Table of Contents:
Chapter 1 Why We Need Complex-Valued Neural Networks?. 12
1.1 Neurons and Neural Networks: Basic Foundations and Historical View.. 12
What is a Neural Network?. 12
The Neuron. 14
Linear Separability and Non-Linear Separability. XOR Problem... 17
1.2 Learning: Basic Fundamentals. 19
Hebbian Learning. 19
Perceptron and Error-Correction Learning. 23
Learning Algorithm... 27
Examples of Application of the Learning Algorithm Based on the Error-Correction Rule 29
Limitation of the Perceptron. Minsky’s and Papert’s work. 32
1.3 Neural Networks: Popular Topologies. 33
XOR Problem: Solution using a Feedforward Neural Network. 33
Popular Real-Valued Activation Functions. 35
Multilayer Feedforward Neural Network (MLF) and its Backpropagation Learning 37
Hopfield Neural Network. 46
Cellular Neural Network. 48
1.4 Introduction to Complex-Valued Neurons and Neural Networks. 50
Why we need them?. 50
Higher Functionality. 51
Importance of Phase and its Proper Treatment 54
Complex-Valued Neural Networks: Brief Historical Observation and State of the Art 60
1.5 Concluding Remarks to Chapter 1. 64
Chapter 2 The Multi-Valued Neuron. 66
2.1 Multiple-Valued Threshold Logic over the Field of Complex Numbers. 66
Multiple-Valued Logic over the Field of Complex Numbers. 67
Multiple-Valued Threshold Functions over the Field of Complex Numbers. 69
2.2 Multi-Valued Neuron (MVN) 76
Discrete MVN.. 76
Continuous MVN.. 78
2.3 Edged Separation of n-Dimensional Space. 82
Important Background. 82
Separation of an n-Dimensional Space. 85
k-Edge. 93
Properties of the k-Edge. 95
Edged Separation and MVN: Summary. 100
2.4 MVN and a Biological Neuron. 103
2.5 Concluding Remarks to Chapter 2. 105
Chapter 3 MVN Learning. 106
3.1 MVN Learning Algorithm... 106
Mechanism of MVN Learning. 107
Learning Strategy. 111
3.2 MVN Learning Rule Based on the Closeness to the Desired Output in terms of Angular Distance 113
Basic Fundamentals. 113
Convergence Theorem... 118
3.3 MVN Error-Correction Learning Rule. 122
Basic Fundamentals. 122
Convergence Theorem... 127
Example of Error-Correction Learning. 133
3.4 Hebbian Learning and MVN.. 136
3.5 Concluding Remarks to Chapter 3. 143
Chapter 4 Multilayer Feedforward Neural Network based on Multi-Valued Neurons (MLMVN) 144
4.1 Introduction to Derivative-Free Backpropagation Learning. 144
4.2 MVN-based Multilayer Neural Network and Error Backpropagation. 149
Simple MLMVN: a Single Hidden Layer and a Single Output Neuron. 149
MLMVN and Backpropagation Learning: the General Case. 154
4.3 MLMVN Learning Algorithm and The Convergence Theorem... 163
Basic Fundamentals. 163
MLMVN Learning Algorithm... 166
Convergence Theorem... 169
Important Advantages of the MLMVN Learning Algorithm... 174
4.4 Examples of MLMVN Applications. 175
Simple Example: 3-Class Classification Problem... 175
Mackey-Glass Time Series Prediction. 179
4.5 Concluding Remarks to Chapter 4. 183
Chapter 5 Multi-Valued Neuron with a Periodic Activation Function. 184
5.1 Universal Binary Neuron (UBN): Two-Valued MVN with a Periodic Activation Function 184
A Periodic Activation Function for k=2. 184
Implementation of the Prity n function using a single neuron. 187
Projection of a two-valued non-linearly separable function into an m-valued threshold function 188
UBN Learning. 192
5.2 k-valued MVN with a Periodic Activation Function. 194
Some Important Fundamentals. 194
Periodic Activation Function for Discrete MVN.. 195
Learning Algorithm for MVN-P. 198
5.3 Simulation Results for k-valued MVN with a Periodic Activation Function. 203
Iris. 203
Two Spirals. 206
Breast Cancer Wisconsin (Diagnostic) 207
Sonar 208
Parity N problem... 209
mod k Addition of n k-valued Variables. 210
5.4 Concluding Remarks to Chapter 5. 218
Chapter 6 Applications of MVN and MLMVN.. 219
6.1 Identification of Blur and its Parameters using MLMVN.. 219
Importance of Blur and its Paramters Identification. 220
Specification of the PSF Recognition Problem... 222
Choice of the Features. 223
Output Definition. “Winner Takes it all Rule” for MLMVN.. 226
Simulation Results. 229
6.2 Financial Time Series Prediction using MLMVN.. 233
Mathematical Model 233
Implementation and Simulation Results. 236
6.3 MVN-Based Associative Memories. 239
A CNN-MVN-based Associative Memory. 239
A Hopfield MVN-based Associative Memory. 244
MVN-based Associative Memory with Random Connections. 249
An MVN-based Associative Memory with Rotation Invariant Association. 253
6.4 Some Other Applications of MVN-based Neural Networks and Concluding Remarks to Chapter 6 258