|
Extreme
Learning Machines |
|
|
|
Kindly Feedback and
Comment to: Guang-Bin Huang Hands-on Workshop on Machine
Learning for BioMedical Informatics 2006 and the
Inaugural Meeting of the International Society for Computational Biology
Student Council (ISCB-SC) Regional Students Group RSG of Singapore. PhD Positions available |
||
|
|
||
It is clear that the learning speed of feedforward neural
networks is in general far slower than required and it has been a major bottleneck
in their applications for past decades. Two key reasons behind may be: 1) the
slow gradient-based learning algorithms are extensively used to train neural
networks, and 2) all the parameters of the networks are tuned iteratively by
using such learning algorithms.
Unlike these traditional
implementations, recently we proposes a new learning algorithm called Extreme
Learning Machine (ELM) which randomly all the hidden nodes parameters of generalized Single-hidden
Layer Feedforward Networks (SLFNs) and analytically determines
the output weights of SLFNs. Many types of hidden nodes including additive/RBF
hidden nodes, and multiplicative nodes, and non neural alike nodes, can be used
as long as they are piecewise nonlinear. Strictly speaking, given a type of
piecewise computational hidden nodes (possibly not neural alike nodes), if SLFNs can work as universal
approximators with adjustable hidden parameters, from a function approximation
point of view the hidden node parameters of SLFNs can actually be randomly
generated according to any continuous sampling distribution. All
the hidden node parameters are independent from the target functions or the
training datasets. All the parameters of ELMs can be
analytically determined instead of being tuned. In theory, this algorithm tends
to provide the good generalization performance at extremely fast learning
speed. The experimental results based on a few artificial and real benchmarking
function approximation and classification problems
including large complex applications show that the new algorithm can produce
better generalization performance in many cases and can learn thousands of
times faster than traditional popular learning algorithms for feedforward
neural networks.
Unlike
traditional implementations and learning theory, from function approximation
point of view, ELM theory shows that the hidden node parameters can be
completely independent from the training data.
1)
In conventional learning theory and implementations, one has to
see the training data before generating the hidden node parameters.
2) In ELM learning theory and implementations, one can generate the hidden node parameters before seeing the training data.
Compared to popular Backpropagation (BP) Algorithm and Support Vector Machine (SVM), ELM has several salient features:
- Ease of use. No parameters need to be manually tuned except predefined network architecture. Users neednt spend several hours or days tuning and training learning machines.
- Faster learning speed. Most training can be completed in milliseconds, seconds, and minutes (for large-scale complex applications). Such a fast learning speed might not be easily obtained using conventional learning methods although it is highly expected.
- Higher generalization performance. It could obtain better generalization performance than BP in most cases, and reach generalization performance similar to or better than SVM.
- Suitable for almost all nonlinear activation functions. Almost all piecewise continuous (including discontinuous, differential, non-differential functions) can be used as activation functions in ELM.
- Suitable for fully complex activation functions. Fully complex functions can also be used as activation functions in ELM.
Difference Between Extreme Learning Machine (ELM) and Random Vector Function Link (RVFL) Net
Misunderstandings on ELM, RVFL, and RBF Networks
(Comments and responses)
Presentations Slides in WCCI2008, Hong Kong
1. G.-B. Huang, L. Chen and C.-K. Siew, Universal Approximation Using Incremental Constructive Feedforward Networks with Random Hidden Nodes, IEEE Transactions on Neural Networks, vol. 17, no. 4, pp. 879-892, 2006. (Technical Report ICIS/46/2003) (Manuscript submitted on Oct 29, 2003, revised on May 8, 2005)
2.
N.-Y. Liang, G.-B. Huang, P. Saratchandran, and
3.
G.-B. Huang, Q.-Y. Zhu, and C.-K. Siew, Extreme Learning
Machine: A New Learning Scheme of Feedforward Neural Networks, 2004 International Joint Conference on
Neural Networks (IJCNN'2004), (
4.
G.-B. Huang, Q.-Y. Zhu and C.-K. Siew, Extreme Learning Machine: Theory
and Applications, Neurocomputing, vol. 70, pp. 489-501,
2006. (Technical Report ICIS/03/2004)
(the extension of [3]) Most cited paper published in Neurocomputing in the past five
years (SCOPU on Nov 7 2009)
5.
H.-J. Rong, G.-B. Huang, P. Saratchandran, and
N. Sundararajan, On-Line Sequential Fuzzy Extreme Learning Machine for
Function Approximation and Classification Problems, IEEE Transactions on Systems, Man, and Cybernetics: Part B,
vol. 39, no. 4, pp. 1067-1072, 2009.
6.
G. Feng, G.-B. Huang, Q. Lin, and R. Gay, Error
Minimized Extreme Learning Machine with Growth of Hidden Nodes and Incremental
Learning, IEEE Transactions on
Neural Networks, vol. 20, no. 8, pp. 1352-1357, 2009.
7.
Y. Lan, Y. C. Soh, and G.-B. Huang, Ensemble
of Online Sequential Extreme Learning Machine, Neurocomputing,
vol. 72, pp. 3391-3395, 2009.
8.
M.-B.
Li, G.-B. Huang, P. Saratchandran, and N. Sundararajan, Fully
Complex Extreme Learning Machine, Neurocomputing,
vol. 68, pp. 306-314, 2005.
9.
G.-B.
Huang and L. Chen, Convex Incremental Extreme Learning
Machine, Neurocomputing,
vol. 70, pp. 3056-3062, 2007. (available for fuzzy inference system, etc)
10.
G.-B.
Huang and L. Chen, Enhanced Random Search Based
Incremental Extreme Learning Machine, Neurocomputing,
vol. 71, pp. 3460-3468, 2008. (available for fuzzy inference system, etc), (higher prediction
accuracy, fast learning rate
and compact network achieved)
11.
G.-B.
Huang, M.-B. Li, L. Chen and C.-K. Siew, Incremental
Extreme Learning Machine With Fully Complex Hidden Nodes, Neurocomputing, vol. 71, pp.
576-583, 2008. (also
briefing the differences between RVFL and RBF)
12. Q.-Y. Zhu, A. K. Qin, P. N. Suganthan, and G.-B. Huang, Evolutionary Extreme Learning Machine, Pattern Recognition, vol. 38, pp. 1759-1763, 2005. (Source-Codes of E-ELM) (A higher prediction accuracy and more compact network required?)
13. G.-B.
Huang, N.-Y. Liang, H.-J. Rong, P. Saratchandran, and N. Sundararajan, On-Line
Sequential Extreme Learning Machine, the IASTED International
Conference on Computational Intelligence (CI 2005), Calgary, Canada, July 4-6,
2005. (available for
fuzzy inference system, etc)
14. G.-B. Huang, Q.-Y. Zhu, K. Z. Mao, C.-K.
Siew, P. Saratchandran, and
15. N.-Y.
Liang, P. Saratchandran, G.-B. Huang, and
16. G.-B. Huang, Q.-Y. Zhu and C.-K. Siew, Real-Time Learning Capability of Neural Networks, IEEE Transactions on Neural Networks, vol. 17, no. 4, pp. 863-878, 2006. (Technical Report ICIS/45/2003)
17. C.-W. T. Yeu , M.-H. Lim, G.-B. Huang, A. Agarwal, and Y. S. Ong, A New Machine Learning Paradigm for Terrain Reconstruction, IEEE Geoscience and Remote Sensing Letters, vol. 3, no. 3, pp. 382-386, 2006.
18. R.
Zhang, G.-B. Huang,
19. G.-B.
Huang and C.-K. Siew, Extreme Learning Machine: RBF
Network Case, Proceedings of the Eighth
International Conference on Control, Automation, Robotics and Vision
(ICARCV2004), Dec 6-9,
20. G.-B. Huang and C.-K. Siew, Extreme Learning Machine with Randomly Assigned RBF Kernels, International Journal of Information Technology, vol. 11, no. 1, pp. 1624, 2005.
21. D.
Wang and G.-B. Huang, Protein Sequence
Classification Using Extreme Learning Machine, Proceedings of
International Joint Conference on Neural Networks (IJCNN2005), (
Free counter Last update : 16-03-06