Luận án Adaptive learning solution based on deep learning for traffic object recognition

MINISTRY OF EDUCATION AND TRAINING 
DUY TAN UNIVERSITY 
ADAPTIVE LEARNING SOLUTION BASED ON DEEP 
LEARNING FOR TRAFFIC OBJECT RECOGNITION 
DOCTOR OF PHILOSOPHY OF COMPUTER SCIENCE 
Da Nang, 2022 
MINISTRY OF EDUCATION AND TRAINING 
DUY TAN UNIVERSITY 
ADAPTIVE LEARNING SOLUTION BASED ON DEEP 
LEARNING FOR TRAFFIC OBJECT RECOGNITION 
Major: Computer Science 
Code: 9480101 
Da Nang, 2022 
 i 
COMMITMENT 
 To the best of my knowledge, I hereby certify that all the content in the 
thesis entitled "Adaptive learning solution based on deep learning for traffic object 
recognition" is my own research. The figures and results of the thesis are honest, 
fully quoted and have not been previously published by another. 
 The author's signature 
 ii 
ACKNOWLEDGEMENTS 
First of all, I would like to express my endless thanks to my instructors. Their 
kindly support and advices went through the completion process of my PhD thesis. 
Their companion encouraged me to improve my work. Their instructions and 
motivation helped me to grow as a research scientist. 
I would also like to thank my council reviewers, members and independent 
scientists for giving me contribution and brilliant comments to my thesis. 
I would like to express my sincere thanks to the Board of Trustees and Board 
of Rector of Duy Tan University, the teachers and officers of Duy Tan University's 
Graduate School, for helping me in the process of learning and researching at 
University. 
I also acknowledge my thankfulness to the Board of Directors of the Quang 
Binh provincial Department of Information and Communications for kind 
assistances and support in my work and learning so that I can achieve the results 
today. 
Many thanks come to the research group’s members for their participation in 
the published works and allowing me to use the research results for this thesis. 
Finally, my deeply thanks come to my loved people and friends who were 
always beside me to help me when I need for the last time. A special thanks to my 
family where I got the most assistances and motivation for the whole of my life. 
In spite of the fact that many efforts are made during the working process, the 
thesis may remain shortcomings due to limited time and research conditions. All 
valuable comments and suggestions for the thesis completion will be highly 
appreciated. 
 The author 
 iii 
TABLE OF CONTENTS 
LIST OF FIGURES .............................................................................................................. vi 
LIST OF TABLES .............................................................................................................. viii 
LIST OF ABBREVIATIONS ................................................................................................ x 
INTRODUCTION ................................................................................................................. 1 
1. Introduction .................................................................................................................... 1 
2. Research goal ................................................................................................................. 3 
3. Research method ............................................................................................................ 3 
4. Research subject and scope ............................................................................................ 4 
5. The structure of the thesis .............................................................................................. 5 
CHAPTER 1. OVERVIEW OF ARTIFICIAL INTELLIGENCE ........................................ 7 
1.1 Overview of artificial intelligence ............................................................................... 7 
1.1.1. Definition of artificial intelligence ........................................................................... 7 
1.1.2 History of artificial intelligence ................................................................................ 7 
1.2. Machine learning and identification techniques .......................................................... 8 
1.2.1 Machine learning applications .................................................................................. 8 
1.2.1.1 Image processing .................................................................................................... 8 
1.2.1.2 Text analysis ........................................................................................................... 9 
1.2.1.3 Data mining ............................................................................................................ 9 
1.2.1.4. Video games and robotics ................................................................................... 10 
1.2.2 Basic recognition techniques in machine learning .................................................. 10 
1.2.2.1 Decision tree ......................................................................................................... 10 
1.2.2.2 Random forests..................................................................................................... 11 
1.2.2.3 Boosting technique ............................................................................................... 11 
1.2.2.4 Support vector machine ........................................................................................ 12 
1.2.2.5 Artificial neural network ...................................................................................... 13 
1.3 Deep Learning and Adaptive Learning ...................................................................... 15 
1.3.1 Overview of Deep Learning and Adaptive Learning .............................................. 15 
1.4.1.1 Deep Learning ...................................................................................................... 15 
1.3.1.2 Adaptive learning ................................................................................................. 15 
1.3.2 Deep neural network (DNN) ................................................................................... 16 
1.3.3 Convolution neural network (CNN) ........................................................................ 17 
 iv 
1.4 Domestic and international research .......................................................................... 18 
1.4.1 Domestic research ................................................................................................... 18 
1.4.2 International research .............................................................................................. 19 
1.4.1.1 Overview .............................................................................................................. 19 
CHAPTER 2. RECOGNIZING OBJECTS BY DEEP LEARNING .................................. 27 
2.1 Object recognition problems ...................................................................................... 27 
2.1.1 Problem: Pedestrian action prediction .................................................................... 27 
2.1.2 Problem: Vehicle recognition ................................................................................. 29 
2.2 Suggested solution ..................................................................................................... 30 
2.2.1 Solution to pedestrian recognition .......................................................................... 31 
2.2.1.1 Extracting features and training classifier model ................................................. 31 
2.2.1.2 Pedestrian action prediction ................................................................................. 32 
2.2.2 Solution to vehicle recognition ............................................................................... 35 
2.2.2.1 Sequential Deep Learning architecture ................................................................ 35 
2.2.2.2 Data augmentation ............................................................................................... 36 
2.3. Experimental evaluation............................................................................................ 37 
2.3.1 Pedestrian detection ................................................................................................ 37 
2.3.1.1 Extracting features and training classifier model ................................................. 37 
2.3.1.2 Pedestrian detection and action prediction ........................................................... 37 
2.3.2 Vehicle recognition ................................................................................................. 38 
2.3.2.1 Experimental data ................................................................................................. 38 
2.3.2.2 Training CNN....................................................................................................... 39 
2.3.2.3 Categorical vehicle recognition............................................................................ 41 
2.4 Conclusion.................................................................................................................. 43 
CHAPTER 3: DEVELOPMENT OF ADAPTIVE LEARNING TECHNIQUE IN OBJECT 
RECOGNITION .................................................................................................................. 45 
3.1 Adaptive learning problem in object recognition....................................................... 45 
3.2 Suggested solutions .................................................................................................... 45 
3.2.1 Overview of solutions ............................................................................................. 45 
3.2.2. Analysis .................................................................................................................. 46 
3.2.2.1 Concept Definitions of System Components ....................................................... 46 
3.2.2.2 General Structure of the System .......................................................................... 48 
3.2.2.3 Details of the Proposed Architecture ................................................................... 50 
 v 
3.3. Experimental evaluation............................................................................................ 54 
3.3.1 Training CNN Model .............................................................................................. 54 
3.3.1.1 IONet model ......................................................................................................... 55 
3.3.1.2 PDNet model ........................................................................................................ 56 
3.3.2 Retraining and updating model ............................................................................... 60 
3.3.3 Compared results ............................. ... X. Shubham Mittal, Suraj Saurabh, Twisha Prasad, Hyunchul Shin, "Pedestrian Detection 
and Tracking Using Deformable Part Models and Kalman Filtering," Journal of Computer-
Mediated Communication, vol. 10, pp. 960-966, 2013. 
[56] A. A. Yu Xiang, Silvio Savarese, "Learning to Track: Online Multi-object Tracking by 
Decision Making," Computer Vision (ICCV), 2015 IEEE International Conference on, pp. 
4705-4713, 2015. 
[57] B. T. Navneet Dalal, "Histograms of Oriented Gradients for Human Detection," 
Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and 
Pattern Recognition (CVPR'05), vol. 1, pp. 886-893, 2005. 
[58] M.-H. L. Van-Dung Hoang, Kang-Hyun Jo, "Robust Human Detection Using Multiple Scale 
of Cell Based Histogram of Oriented Gradients and AdaBoost Learning," Computational 
Collective Intelligence. Technologies and Applications, vol. 7653, pp. 61-71, 2012. 
 104 
[59] R. G. Pedro F Felzenszwalb, David McAllester, Deva Ramanan, "Object Detection with 
Discriminatively Trained Part-Based Models," IEEE Transactions on pattern analysis and 
machine intelligence, vol. 32, pp. 1627-1645, 2010. 
[60] R. J. Mstafa and K. M. Elleithy, "A video steganography algorithm based on Kanade-Lucas-
Tomasi tracking algorithm and error correcting codes," Multimedia Tools and 
Applications, vol. 75, pp. 10311-10333, 2016. 
[61] V.-D. Hoang, "Multiple classifier-based spatiotemporal features for living activity 
prediction," Journal of Information and Telecommunication, vol. 1, pp. 100-112, 2017. 
[62] K.-H. J. Joko Hariyono, "Detection of Pedestrian Crossing Road A Study on Pedestrian Pose 
Recognition," Neurocomputing, vol. 234, pp. 144-153, 2016. 
[63] M. A. Russell Stewart, Andrew Y. Ng, "End-to-end people detection in crowded scenes," 
2016 IEEE Conference on Computer Vision and Pattern Recognition, pp. 2325-2333, 2015. 
[64] A. L. a. M. V. D. S. Piérard, "A probabilistic pixel-based approach to detect humans in 
video streams," IEEE International Conference on Acoustics, Speech and Signal Processing 
(ICASSP), pp. 921-924, 2011. 
[65] C. R. Dow, H. H. Ngo, L. H. Lee, P. Y. Lai, K. C. Wang, and V. T. Bui, "A crosswalk pedestrian 
recognition system by using deep learning and zebra‐crossing recognition techniques," 
Software: Practice and Experience, vol. 50, pp. 630-644, 2020. 
[66] I. Amirullah, R. Yusliana Bakti, I. Areni, and A. A. Alimuddin, Vehicle detection and tracking 
using Gaussian Mixture Model and Kalman Filter, 2016. 
[67] Y. Chen and Q. Wu, Moving vehicle detection based on optical flow estimation of edge, 
2015. 
[68] J.-y. Choi, K.-S. Sung, and Y. Yang, Multiple Vehicles Detection and Tracking based on 
Scale-Invariant Feature Transform, 2007. 
[69] G. Yan, Y. Ming, Y. Yu, and L. Fan, Real-time vehicle detection using histograms of oriented 
gradients and AdaBoost classification vol. 127, 2016. 
[70] S. G. d. S. Filho, R. Z. Freire, and L. d. S. Coelho, "Feature Extraction for On-Road Vehicle 
Detection Based on Support Vector Machine," Conference Proceedings, 2017. 
[71] Z. Moutakki, M. I. Ouloul, A. Karim, and A. Abdellah, Real-Time System Based on Feature 
Extraction for Vehicle Detection and Classification vol. 19, 2018. 
[72] A. A. Yilmaz, M. S. Guzel, I. Askerbeyli, and E. Bostanci, "A vehicle detection approach 
using deep learning methodologies," arXiv preprint arXiv:1804.00429, 2018. 
[73] J. Espinosa Oviedo, S. Velastin, and J. W. Branch, Vehicle Detection Using Alex Net and 
Faster R-CNN Deep Learning Models: A Comparative Study, 2017. 
[74] X. Chen, S. Xiang, C.-L. Liu, and C.-H. Pan, Vehicle Detection in Satellite Images by Hybrid 
Deep Convolutional Neural Networks vol. 11, 2014. 
[75] S. Qu, Y. Wang, G. Meng, and C. Pan, Vehicle Detection in Satellite Images by 
Incorporating Objectness and Convolutional Neural Network, 2016. 
[76] Y. Koga, H. Miyazaki, and R. Shibasaki, "Counting vehicles by deep neural network in high 
resolution satellite images." 
[77] C. Migel Bautista, C. Austin Dy, M. Inigo Manalac, R. Angelo Orbe, and M. Cordel, II, 
Convolutional neural network for vehicle detection in low resolution traffic videos, 2016. 
[78] M. S. I. Harbas, "Detection of roadside vegetation using features from the visible 
spectrum," 2014 37th International Convention on Information and Communication 
Technology, Electronics and Microelectronics (MIPRO), pp. 1204-1209, 26-30 May 2014 
2014. 
[79] M. Aly, "Real time Detection of Lane Markers in Urban Streets," IEEE Intelligent Vehicles 
Symposium, p. 6, 2014. 
 105 
[80] M. M. T. R. K. Satzoda, "Vision-Based Lane Analysis: Exploration of Issues and Approaches 
for Embedded Realization," 2013 IEEE Conference on Computer Vision and Pattern 
Recognition Workshops, pp. 604-609, 23-28 June 2013 2013. 
[81] Q. W. Wang Hua, Wang Y, R Miller Gregory, "Dual Roadside Seismic Sensor for Moving 
Road Vehicle Detection and Characterization," Sensors, vol. 14, pp. 2892-910, 2014. 
[82] J. Z. Q. Wang, H. Xu, B. Xu, R. Chen, "Roadside Magnetic Sensor System for Vehicle 
Detection in Urban Environments," IEEE Transactions on Intelligent Transportation 
Systems, vol. 19, pp. 1365-1374, 2018. 
[83] F. J. J. Brostow Gabriel, Cipolla Roberto, "Semantic object classes in video: A high-
definition ground truth database," Pattern Recognition Letters, vol. 30, pp. 88-97, 2009. 
[84] A. K. V. Badrinarayanan, R. Cipolla, "SegNet: A Deep Convolutional Encoder-Decoder 
Architecture for Image Segmentation," IEEE Transactions on Pattern Analysis and Machine 
Intelligence, vol. 39, pp. 2481-2495, 2017. 
[85] X. Li, M. Ye, Y. Liu, and C. Zhu, "Adaptive deep convolutional neural networks for scene-
specific object detection," IEEE Transactions on Circuits and Systems for Video Technology, 
vol. 29, pp. 2538-2551, 2017. 
[86] T.-K. Lin, "Adaptive learning method for multiple-object detection in manufacturing," 
Advances in Mechanical Engineering, vol. 7, p. 1687814015618906, 2015. 
[87] L. Cheng, X. Liu, L. Li, L. Jiao, and X. Tang, "Deep Adaptive Proposal Network for Object 
Detection in Optical Remote Sensing Images," arXiv preprint arXiv:1807.07327, 2018. 
[88] K. Blix and T. Eltoft, "Machine learning automatic model selection algorithm for oceanic 
chlorophyll-a content retrieval," Remote Sensing, vol. 10, p. 775, 2018. 
[89] S. Raschka, "Model evaluation, model selection, and algorithm selection in machine 
learning," arXiv preprint arXiv:1811.12808, 2018. 
[90] L. Li and A. Talwalkar, "Random search and reproducibility for neural architecture search," 
arXiv preprint arXiv:1902.07638, 2019. 
[91] H. Bertrand, R. Ardon, M. Perrot, and I. Bloch, "Hyperparameter optimization of deep 
neural networks: Combining hyperband with Bayesian model selection," in Conférence sur 
l’Apprentissage Automatique, 2017. 
[92] T. Domhan, J. T. Springenberg, and F. Hutter, "Speeding up automatic hyperparameter 
optimization of deep neural networks by extrapolation of learning curves," in Twenty-
fourth international joint conference on artificial intelligence, 2015. 
[93] S. Kamada and T. Ichimura, "An object detection by using adaptive structural learning of 
deep belief network," in 2019 international joint conference on neural networks (IJCNN), 
2019, pp. 1-8. 
[94] C. Huang, S. Lucey, and D. Ramanan, "Learning policies for adaptive tracking with deep 
feature cascades," in Proceedings of the IEEE International Conference on Computer 
Vision, 2017, pp. 105-114. 
[95] M. Long, Y. Cao, Z. Cao, J. Wang, and M. I. Jordan, "Transferable representation learning 
with deep adaptation networks," IEEE transactions on pattern analysis and machine 
intelligence, vol. 41, pp. 3071-3085, 2018. 
[96] N. Q. K. Le, T.-T. Huynh, E. K. Y. Yapp, and H.-Y. Yeh, "Identification of clathrin proteins by 
incorporating hyperparameter optimization in deep learning and PSSM profiles," 
Computer methods and programs in biomedicine, vol. 177, pp. 81-88, 2019. 
[97] A. Klein, S. Falkner, S. Bartels, P. Hennig, and F. Hutter, "Fast bayesian optimization of 
machine learning hyperparameters on large datasets," arXiv preprint arXiv:1605.07079, 
2016. 
 106 
[98] J. Snoek, O. Rippel, K. Swersky, R. Kiros, N. Satish, N. Sundaram, et al., "Scalable bayesian 
optimization using deep neural networks," in International conference on machine 
learning, 2015, pp. 2171-2180. 
[99] M. Claesen and B. De Moor, "Hyperparameter search in machine learning," arXiv preprint 
arXiv:1502.02127, 2015. 
[100] S. C. Smithson, G. Yang, W. J. Gross, and B. H. Meyer, "Neural networks designing neural 
networks: multi-objective hyper-parameter optimization," in Proceedings of the 35th 
International Conference on Computer-Aided Design, 2016, pp. 1-8. 
[101] E. Bochinski, T. Senst, and T. Sikora, "Hyper-parameter optimization for convolutional 
neural network committees based on evolutionary algorithms," in 2017 IEEE International 
Conference on Image Processing (ICIP), 2017, pp. 3924-3928. 
[102] W.-Y. Lee, K.-E. Ko, Z.-W. Geem, and K.-B. Sim, "Method that determining the 
Hyperparameter of CNN using HS algorithm," Journal of Korean institute of intelligent 
systems, vol. 27, pp. 22-28, 2017. 
[103] A.-C. Florea and R. Andonie, "Weighted random search for hyperparameter optimization," 
arXiv preprint arXiv:2004.01628, 2020. 
[104] L. Kotthoff, C. Thornton, H. H. Hoos, F. Hutter, and K. Leyton-Brown, "Auto-WEKA 2.0: 
Automatic model selection and hyperparameter optimization in WEKA," The Journal of 
Machine Learning Research, vol. 18, pp. 826-830, 2017. 
[105] X. Zeng and G. Luo, "Progressive sampling-based Bayesian optimization for efficient and 
automatic machine learning model selection," Health information science and systems, 
vol. 5, p. 2, 2017. 
[106] G. Dikov, P. van der Smagt, and J. Bayer, "Bayesian learning of neural network 
architectures," arXiv preprint arXiv:1901.04436, 2019. 
[107] P. Dollár, R. Appel, S. Belongie, and P. Perona, "Fast feature pyramids for object 
detection," IEEE transactions on pattern analysis and machine intelligence, vol. 36, pp. 
1532-1545, 2014. 
[108] K. He, X. Zhang, S. Ren, and J. Sun, Delving Deep into Rectifiers: Surpassing Human-Level 
Performance on ImageNet Classification vol. 1502, 2015. 
[109] E. Brochu, V. M. Cora, and N. De Freitas, "A tutorial on Bayesian optimization of expensive 
cost functions, with application to active user modeling and hierarchical reinforcement 
learning," arXiv preprint arXiv:1012.2599, 2010. 
[110] B. Shahriari, K. Swersky, Z. Wang, R. P. Adams, and N. De Freitas, "Taking the human out 
of the loop: A review of Bayesian optimization," Proceedings of the IEEE, vol. 104, pp. 148-
175, 2015. 
[111] J. Bergstra, R. Bardenet, Y. Bengio, and B. Kégl, "Algorithms for hyper-parameter 
optimization," in 25th annual conference on neural information processing systems (NIPS 
2011), 2011.