Super-resolved Thermal Imagery for High-accuracy Facial Areas Detection and Analysis - Publication - Bridge of Knowledge

Your account

login

Search

Super-resolved Thermal Imagery for High-accuracy Facial Areas Detection and Analysis

Abstract

In this study, we evaluate various Convolutional Neural Networks based Super-Resolution (SR) models to improve facial areas detection in thermal images. In particular, we analyze the influence of selected spatiotemporal properties of thermal image sequences on detection accuracy. For this purpose, a thermal face database was acquired for 40 volunteers. Contrary to most of existing thermal databases of faces, we publish our dataset in a raw, original format (14-bit depth) to preserve all important details. In our experiments, we utilize two metrics usually used for image enhancement evaluation: Peak-Signal-to-Noise Ratio (PSNR) and Structural Similarity Index Metric (SSIM). In addition, we present how to design a SR network with a widened receptive field to mitigate the problem of contextual information being spread over larger image regions due to the heat flow in thermal images. Finally, we determine whether there is a relation between achieved PSNR and accuracy of facial areas detection that can be analyzed for vital signs extraction (e.g. nostril region). The performed evaluation showed that PSNR can be improved even by 60\% if full bit depth resolution data is used instead of 8 bits. Also, we showed that the application of image enhancement solution is necessary for low resolution images to achieve a satisfactory accuracy of object detection.

Citations

  • 1 8

    CrossRef

  • 0

    Web of Science

  • 1 9

    Scopus

Authors (4)

Cite as

Full text

download paper
downloaded 51 times
Publication version
Accepted or Published Version
License
Creative Commons: CC-BY-NC-ND open in new tab

Keywords

Details

Category:
Articles
Type:
artykuły w czasopismach
Published in:
ENGINEERING APPLICATIONS OF ARTIFICIAL INTELLIGENCE no. 87,
ISSN: 0952-1976
Language:
English
Publication year:
2020
Bibliographic description:
Kwaśniewska A., Rumiński J., Szankin M., Kaczmarek M.: Super-resolved Thermal Imagery for High-accuracy Facial Areas Detection and Analysis// ENGINEERING APPLICATIONS OF ARTIFICIAL INTELLIGENCE -Vol. 87, (2020), s.103263-
DOI:
Digital Object Identifier (open in new tab) 10.1016/j.engappai.2019.103263
Bibliography: test
  1. R. Keys, Cubic convolution interpolation for digital image processing, IEEE transactions on acoustics, speech, and signal processing 29 (6) (1981) 1153-1160. open in new tab
  2. L. Zhang, X. Wu, An edge-guided image interpolation algorithm via directional filtering and data fusion, IEEE transactions on Image Processing 15 (8) (2006) 2226-2238.
  3. Y. Romano, M. Protter, M. Elad, Single image interpolation via adaptive nonlocal sparsity-based modeling, IEEE Transactions on Image Processing 23 (7) (2014) 3085-3098. open in new tab
  4. D. Glasner, S. Bagon, M. Irani, Super-resolution from a single image, in: Com- puter Vision, 2009 IEEE 12th International Conference on, IEEE, 2009, pp. 349- 455 356. open in new tab
  5. G. Freedman, R. Fattal, Image and video upscaling from local self-examples, ACM Transactions on Graphics (TOG) 30 (2) (2011) 12. open in new tab
  6. Z. Cui, H. Chang, S. Shan, B. Zhong, X. Chen, Deep network cascade for image super-resolution, in: European Conference on Computer Vision, Springer, 2014, 460 pp. 49-64. open in new tab
  7. H. Chang, D.-Y. Yeung, Y. Xiong, Super-resolution through neighbor embedding, in: Computer Vision and Pattern Recognition, 2004. CVPR 2004. Proceedings of the 2004 IEEE Computer Society Conference on, Vol. 1, IEEE, 2004, pp. I-I.
  8. K. I. Kim, Y. Kwon, Single-image super-resolution using sparse regression and 465 natural image prior, IEEE transactions on pattern analysis & machine intelli- gence (6) (2010) 1127-1133.
  9. M. Bevilacqua, A. Roumy, C. Guillemot, M. L. Alberi-Morel, Low-complexity single-image super-resolution based on nonnegative neighbor embedding. open in new tab
  10. K. Jia, X. Wang, X. Tang, Image transformation based on learning dictionaries 470 across image spaces, IEEE transactions on pattern analysis and machine intelli- gence 35 (2) (2013) 367-380. open in new tab
  11. C. Dong, C. C. Loy, K. He, X. Tang, Image super-resolution using deep convolu- tional networks, IEEE transactions on pattern analysis and machine intelligence 38 (2) (2016) 295-307. open in new tab
  12. J. Kim, J. Kwon Lee, K. Mu Lee, Accurate image super-resolution using very deep convolutional networks, in: Proceedings of the IEEE conference on com- puter vision and pattern recognition, 2016, pp. 1646-1654. open in new tab
  13. J. Kim, J. Kwon Lee, K. Mu Lee, Deeply-recursive convolutional network for image super-resolution, in: Proceedings of the IEEE conference on computer 480 vision and pattern recognition, 2016, pp. 1637-1645. open in new tab
  14. Y. Tai, J. Yang, X. Liu, Image super-resolution via deep recursive residual net- work, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Vol. 1, 2017, p. 5. open in new tab
  15. J. Liu, W. Yang, X. Zhang, Z. Guo, Retrieval compensated group structured spar- 485 sity for image super-resolution, IEEE Transactions on Multimedia 19 (2) (2017) 302-316. open in new tab
  16. X. Li, M. T. Orchard, New edge-directed interpolation, IEEE transactions on im- age processing 10 (10) (2001) 1521-1527.
  17. H. Chen, X. He, L. Qing, Q. Teng, Single image super-resolution via adaptive 490 transform-based nonlocal self-similarity modeling and learning-based gradient regularization, IEEE Transactions on Multimedia 19 (8) (2017) 1702-1717. open in new tab
  18. V. Jain, S. Seung, Natural image denoising with convolutional networks, in: Ad- vances in Neural Information Processing Systems, 2009, pp. 769-776.
  19. C. Ledig, L. Theis, F. Huszár, J. Caballero, A. Cunningham, A. Acosta, A. Aitken, 495 open in new tab
  20. A. Tejani, J. Totz, Z. Wang, et al., Photo-realistic single image super-resolution using a generative adversarial network, in: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, 2017, pp. 105-114.
  21. M. S. Sajjadi, B. Schölkopf, M. Hirsch, Enhancenet: Single image super- resolution through automated texture synthesis, in: Computer Vision (ICCV), 500 2017 IEEE International Conference on, IEEE, 2017, pp. 4501-4510. open in new tab
  22. S. C. Park, M. K. Park, M. G. Kang, Super-resolution image reconstruction: a technical overview, IEEE signal processing magazine 20 (3) (2003) 21-36.
  23. L. G. Villanueva, G. M. Callicó, F. Tobajas, S. López, V. De Armas, J. F. López, R. Sarmiento, Medical diagnosis improvement through image quality enhance- 505 ment based on super-resolution, in: Digital System Design: Architectures, Meth- ods and Tools (DSD), 2010 13th Euromicro Conference on, IEEE, 2010, pp. 259- 262. open in new tab
  24. M. Abdel-Nasser, J. Melendez, A. Moreno, O. A. Omer, D. Puig, Breast tumor classification in ultrasound images using texture analysis and super-resolution 510 methods, Engineering Applications of Artificial Intelligence 59 (2017) 84-92. open in new tab
  25. Y. Gao, H. Li, J. Dong, G. Feng, A deep convolutional network for medical image super-resolution, in: Chinese Automation Congress (CAC), 2017, IEEE, 2017, pp. 5310-5315. open in new tab
  26. A. Kwaśniewska, A. Giczewska, J. Rumiński, Big data significance in remote 515 medical diagnostics based on deep learning techniques, Task Quarterly 21 (2017) 309-319.
  27. M. Lewandowska, J. Rumiński, T. Kocejko, J. Nowak, Measuring pulse rate with a webcama non-contact method for evaluating cardiac activity, in: Computer Sci- ence and Information Systems (FedCSIS), 2011 Federated Conference on, IEEE, 520 2011, pp. 405-410.
  28. J. Ruminski, A. Kwasniewska, Evaluation of respiration rate using thermal imag- ing in mobile conditions, in: Application of Infrared to Biomedical Sciences, Springer, 2017, pp. 311-346. open in new tab
  29. M. Hanmandlu, et al., A new entropy function and a classifier for thermal face 525 recognition, Engineering Applications of Artificial Intelligence 36 (2014) 269- 286.
  30. A. Kwaśniewska, J. Rumiński, Face detection in image sequences using a portable thermal camera, in: Proceedings of the 13th Quantitative Infrared Ther- mography Conference, 2016. open in new tab
  31. A. Kwaśniewska, J. Rumiński, K. Czuszyński, M. Szankin, Real-time facial fea- tures detection from low resolution thermal images with deep classification mod- els, Journal of Medical Imaging and Health Informatics 8 (5) (2018) 979-987. open in new tab
  32. M. Szankin, A. Kwasniewska, T. Sirlapu, M. Wang, J. Ruminski, R. Nicolas, M. Bartscherer, Long distance vital signs monitoring with person identification 535 for smart home solutions, in: 2018 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), IEEE, 2018, pp. 1558-1561. open in new tab
  33. I. Goodfellow, Y. Bengio, A. Courville, Y. Bengio, Deep learning, Vol. 1, MIT press Cambridge, 2016.
  34. R. Gade, T. B. Moeslund, Thermal cameras and applications: a survey, Machine vision and applications 25 (1) (2014) 245-262. open in new tab
  35. Flir lepton camera modules, https://www.flir.com/products/ lepton/, accessed: 2018-11-10. open in new tab
  36. J. Rumiński, Analysis of the parameters of respiration patterns extracted from 545 thermal image sequences, in: Biocybernetics and Biomedical Engineering, Vol. 36, 2016, pp. 732-741. open in new tab
  37. B. Qi, V. John, Z. Liu, S. Mita, Pedestrian detection from thermal images: A sparse representation based approach, Infrared Physics & Technology 76 (2016) open in new tab
  38. K. A. R. J. N. R. Szankin, Maciej, Road condition evaluation using fusion of multiple deep models on always-on vision processor, in: Proc. Of the 44th Annual Conference of the IEEE Industrial Electronics Society, in print, 2018. open in new tab
  39. W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C.-Y. Fu, A. C. Berg, Ssd: Single shot multibox detector, in: European conference on computer vision, 555 open in new tab
  40. K. He, X. Zhang, S. Ren, J. Sun, Deep residual learning for image recognition, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770-778. open in new tab
  41. V. Nair, G. E. Hinton, Rectified linear units improve restricted boltzmann ma- 560 chines, in: Proceedings of the 27th international conference on machine learning (ICML-10), 2010, pp. 807-814.
  42. K. He, X. Zhang, S. Ren, J. Sun, Delving deep into rectifiers: Surpassing human- level performance on imagenet classification, in: Proceedings of the IEEE inter- national conference on computer vision, 2015, pp. 1026-1034. open in new tab
  43. D. P. Kingma, J. Ba, Adam: A method for stochastic optimization, arXiv preprint arXiv:1412.6980. open in new tab
  44. Tensorflow implementation of sr models, https://github.com/ LoSealL/VideoSuperResolution, accessed: 2018-10-10. open in new tab
  45. Drrn repository, https://github.com/tyshiwo/DRRN_CVPR17, ac- 570 cessed: 2018-10-10. open in new tab
  46. L. Torrey, J. Shavlik, Transfer learning, in: Handbook of Research on Machine Learning Applications and Trends: Algorithms, Methods, and Techniques, IGI Global, 2010, pp. 242-264. open in new tab
  47. Tensorflow detection model zoo, https://github.com/tensorflow/ 575 models/blob/master/research/object_detection/g3doc/ detection_model_zoo.md, accessed: 2018-11-10. open in new tab
  48. J. Bergstra, Y. Bengio, Random search for hyper-parameter optimization, Journal of Machine Learning Research 13 (Feb) (2012) 281-305.
  49. A. Hore, D. Ziou, Image quality metrics: Psnr vs. ssim, in: 2010 20th Interna- 580 tional Conference on Pattern Recognition, IEEE, 2010, pp. 2366-2369. open in new tab
Verified by:
Gdańsk University of Technology

seen 123 times

Recommended for you

Improving Accuracy of Contactless Respiratory Rate Estimation by Enhancing Thermal Sequences with Deep Neural Networks

2019

Evaluating Accuracy of Respiratory Rate Estimation from Super Resolved Thermal Imagery

2019

Thermal Images Analysis Methods using Deep Learning Techniques for the Needs of Remote Medical Diagnostics

2020

Influence of Thermal Imagery Resolution on Accuracy of Deep Learning based Face Recognition

2019
Meta Tags