LP-SVR Model Selection Using an Inexact Globalized Quasi-Newton Strategy

Pablo Rivas-Perea, Juan Cota-Ruiz, J. A. Perez Venzor, David Garcia Chaparro, Jose-Gerardo Rosiles

Department of Computer Science, Baylor University, Waco, TX, USA.
Department of Electrical and Computer Engineering, Auto- nomous University of Ciudad Juarez, Ciudad Juárez, Mexico.
Science Applications International Corporation, El Paso, TX, USA..

DOI: 10.4236/jilsa.2013.51003                

Abstract

In this paper we study the problem of model selection for a linear programming-based support vector machine for regression. We propose generalized method that is based on a quasi-Newton method that uses a globalization strategy and an inexact computation of first order information. We explore the case of two-class, multi-class, and regression problems. Simulation results among standard datasets suggest that the algorithm achieves insignificant variability when measuring residual statistical properties.

Keywords

Hyper-Parameter Estimation; Support Vector Regression; Machine Learning; Data Mining

Conflicts of Interest

The authors declare no conflicts of interest.

References

[1]	A. J. Smola and B. Scholkopf, “A Tutorial on Support Vector Regression,” Statistics and Computing, Vol. 14, No. 3, 2004, pp. 199-222. doi:10.1023/B:STCO.0000035301.49549.88
[2]	D. Anguita, A. Boni, S. Ridella, F. Rivieccio and D. Sterpi, “Theoretical and Practical Model Selection Methods for Support Vector Classifiers,” Support Vector Machines: Theory and Applications, Vol. 177, 2005, pp. 159-179. doi:10.1007/10984697_7
[3]	K. Duan, S. Keerthi and A. Poo, “Evaluation of Simple Performance Measures for Tuning SVM Hyperparameters,” Neurocomputing, Vol. 51, 2003, pp. 41-59. doi:10.1016/S0925-2312(02)00601-X
[4]	Z. Hui-ren and P. Zheng, “Method for Selecting Parameters of Least Squares Support Vector Machines Based on GA and Bootstrap,” Journal of System Simulation, Vol. 12, 2008.
[5]	D. Anguita, S. Ridella, F. Rivieccio and R. Zunino, “Hyperparameter Design Criteria for Support Vector Classifiers,” Neurocomputing, Vol. 55, No. 1-2, 2003, pp. 109-134. doi:10.1016/S0925-2312(03)00430-2
[6]	L. Wang and S. O. Service, “Support Vector Machines: Theory and Applications,” Studies in Fuzziness and Soft Computing, Springer-Verlag, Berlin, 2005.
[7]	G. Cawley, “Leave-One-Out Cross-Validation Based Model Selection Criteria for Weighted Ls-Svms,” IEEE International Conference on Neural Networks, 16-21 July 2006. doi:10.1109/IJCNN.2006.246634
[8]	P. R. Perea, “Algorithms for Training Large-Scale Linear Programming Support Vector Regression and Classification,” Ph.D. Thesis, The University of Texas, El Paso, 2011.
[9]	J. Dennis and R. Schnabel, “Numerical Methods for Unconstrained Optimization and Nonlinear Equations,” Society for Industrial Mathematics, 1996. doi:10.1137/1.9781611971200
[10]	M. Argaez and L. Velazquez, “A New Infeasible InteriorPoint Algorithm for Linear Programming,” Proceedings of the 2003 Conference on Diversity in Computing, ACM, New York, 2003, pp. 12-14. http://doi.acm.org/10.1145/948542.948545
[11]	J. Mercer, “Functions of Positive and Negative Type, and Their Connection with the Theory of Integral Equations,” Philosophical Transactions of the Royal Society of London. Series A, Containing Papers of a Mathematical or Physical Character, Vol. 209, No. 441-458, 1909, pp. 415-446. doi:10.1098/rsta.1909.0016
[12]	R. Courant and D. Hilbert, “Methods of Mathematical Physics,” Interscience, New York, 1966.
[13]	Z. Lu, J. Sun and K. R. Butts, “Linear Programming Support Vector Regression with Wavelet Kernel: A New Approach to Nonlinear Dynamical Systems Identification,” Mathematics and Computers in Simulation, Vol. 79, No. 7, 2009, pp. 2051-2063. doi:10.1016/j.matcom.2008.10.011
[14]	Y. Torii and S. Abe, “Decomposition Techniques for Training Linear Programming Support Vector Machines,” Neurocomputing, Vol. 72, No. 4-6, 2009, pp. 973-984. doi:10.1016/j.neucom.2008.04.008
[15]	L. Zhang and W. Zhou, “On the Sparseness of 1-Norm Support Vector Machines,” Neural Networks, Vol. 23, No. 3, 2010, pp. 373-385. http://www.sciencedirect.com/science/article/B6T08-4XVBP5J-1/2/b032646ea72f40e7025a40b499134a21
[16]	T. Fawcett, “Roc Graphs: Notes and Practical Considerations for Researchers,” Machine Learning, Vol. 31, 2004, pp. 1-38.
[17]	J. Nocedal and S. Wright, “Numerical Optimization,” Springer Verlag, New York, 1999. doi:10.1007/b98874
[18]	M. Hestenes, “Pseudoinversus and Conjugate Gradients,” Communications of the ACM, Vol. 18, No. 1, 1975, pp. 40-43. doi:10.1145/360569.360658