Abstract
Deep neural networks have achieved great success in many domains. However, successful deployment of such systems is determined by proper manual selection of the neural architecture. This is a tedious and time-consuming process that requires expert knowledge. Different tasks need very different architectures to obtain satisfactory results. The group of methods called the neural architecture search (NAS) helps to find effective architecture in an automated manner. In this paper, we present the use of an architecture search framework to solve the medical task of malignant melanoma detection. Unlike many other methods tested on benchmark datasets, we tested it on practical problem, which differs greatly in terms of difficulty in distinguishing between classes, resolution of images, data balance within the classes, and the number of data available. In order to find a suitable network structure, the hill-climbing search strategy was employed along with network morphism operations to explore the search space. The network morphism operations allow for incremental increases in the network size with the use of the previously trained network. This kind of knowledge reusing allows significantly reducing the computational cost. The proposed approach produces structures that achieve similar results to those provided by manually designed structures, at the same time making use of almost 20 times fewer parameters. What is more, the search process lasts on average only 18h on single GPU.
Citations
-
4 4
CrossRef
-
0
Web of Science
-
4 5
Scopus
Authors (3)
Cite as
Full text
- Publication version
- Accepted or Published Version
- License
- open in new tab
Keywords
Details
- Category:
- Articles
- Type:
- artykuły w czasopismach
- Published in:
-
IEEE Access
no. 8,
pages 9061 - 9071,
ISSN: 2169-3536 - Language:
- English
- Publication year:
- 2020
- Bibliographic description:
- Kwasigroch A., Grochowski M., Mikołajczyk A.: Neural Architecture Search for Skin Lesion Classification// IEEE Access -Vol. 8, (2020), s.9061-9071
- DOI:
- Digital Object Identifier (open in new tab) 10.1109/access.2020.2964424
- Bibliography: test
-
- Y. LeCun, Y. Bengio, and G. Hinton, ''Deep learning,'' Nature, vol. 521, no. 7553, pp. 436-444, May 2015. open in new tab
- A. Krizhevsky, I. Sutskever, and G. E. Hinton, ''ImageNet classification with deep convolutional neural networks,'' in Proc. Adv. Neural Inf. Pro- cess. Syst., Lake Tahoe, NV, USA, Dec. 2012, pp. 1106-1114. open in new tab
- J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, ''ImageNet: A large-scale hierarchical image database,'' in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Jun. 2009. open in new tab
- J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, ''You only look once: Unified, real-time object detection,'' in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Las Vegas, NV, USA, Jun. 2016, pp. 779-788. open in new tab
- S. Ren, K. He, R. B. Girshick, and J. Sun, ''Faster R-CNN: Towards real- time object detection with region proposal networks,'' IEEE Trans. Pattern Anal. Mach. Intell., vol. 39, no. 6, pp. 1137-1149, Jun. 2017. open in new tab
- Y. Zhu, K. Sapra, F. A. Reda, K. J. Shih, S. Newsam, A. Tao, and B. Catanzaro, ''Improving semantic segmentation via video propagation and label relaxation,'' Dec. 2018, arXiv:1812.01593. [Online]. Available: https://arxiv.org/abs/1812.01593 open in new tab
- T. Zhou, M. Brown, N. Snavely, and D. G. Lowe, ''Unsupervised learning of depth and ego-motion from video,'' in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Jul. 2017, pp. 6612-6619. open in new tab
- Z. Chen and X. Huang, ''End-to-end learning for lane keeping of self- driving cars,'' in Proc. IEEE Intell. Vehicles Symp. (IV), Jun. 2017, pp. 1856-1860. open in new tab
- C. Barata, M. E. Celebi, and J. S. Marques, ''A survey of feature extraction in dermoscopy image analysis of skin cancer,'' IEEE J. Biomed. Health Inform., vol. 23, no. 3, pp. 1096-1109, May 2019. open in new tab
- X. Liu, W. Liu, T. Mei, and H. Ma, ''A deep learning-based approach to progressive vehicle re-identification for urban surveillance,'' in Proc. ECCV, 2016, pp. 869-884. open in new tab
- A. Mikolajczyk and M. Grochowski, ''Data augmentation for improving deep learning in image classification problem,'' in Proc. Int. Interdiscipl. PhD Workshop (IIPhDW), May 2018, pp. 117-122. open in new tab
- L. Gatys, A. Ecker, and M. Bethge, ''A neural algorithm of artistic style,'' J. Vis., vol. 16, no. 12, p. 326, Sep. 2016. open in new tab
- X. Yi, E. Walia, and P. Babyn, ''Generative adversarial network in medical imaging: A review,'' Med. Image Anal., vol. 58, Dec. 2019, Art. no. 101552. open in new tab
- M. Pietikäinen, A. Hadid, G. Zhao, and T. Ahonen, Computer Vision Using Local Binary Patterns. London, U.K.: Springer-Verlag, 2011. open in new tab
- K. Simonyan and A. Zisserman, ''Very deep convolutional networks for large-scale image recognition,'' in Proc. 3rd Int. Conf. Learn. Repre- sent. (ICLR), San Diego, CA, USA, May 2015, pp. 1-10.
- G. Huang, Z. Liu, L. V. D. Maaten, and K. Q. Weinberger, ''Densely connected convolutional networks,'' in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Jul. 2017, pp. 2261-2269. open in new tab
- K. He, X. Zhang, S. Ren, and J. Sun, ''Deep residual learning for image recognition,'' in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2016, pp. 770-778. open in new tab
- S. Zagoruyko and N. Komodakis, ''Wide residual networks,'' in Proc. Brit. Mach. Vis. Conf. (BMVC), York, U.K., Sep. 2016, pp. 1-15. open in new tab
- M. Tan and Q. V. Le, ''EfficientNet: Rethinking model scaling for convo- lutional neural networks,'' in Proc. 36th Int. Conf. Mach. Learn. (ICML), Long Beach, CA, USA, 2019, pp. 6105-6114.
- D. P. Kingma and J. Ba, ''Adam: A method for stochastic optimization,'' in Proc. 3rd Int. Conf. Learn. Represent. (ICLR), San Diego, CA, USA, May 2015, pp. 1-15.
- I. Loshchilov and F. Hutter, ''SGDR: Stochastic gradient descent with warm restarts,'' in Proc. 5th Int. Conf. Learn. Represent. (ICLR), Toulon, France, Apr. 2017.
- S. Ioffe and C. Szegedy, ''Batch normalization: Accelerating deep net- work training by reducing internal covariate shift,'' in Proc. 32nd
- Int. Conf. Mach. Learn. (ICML), Lille, France, vol. 37, Jul. 2015, pp. 448-456.
- N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdi- nov, ''Dropout: A simple way to prevent neural networks from overfitting,'' J. Mach. Learn. Res., vol. 15, pp. 1929-1958, Jan. 2014.
- A. L. Maas, ''Rectifier nonlinearities improve neural network acoustic models,'' in Proc. ICML, 2013, vol. 30, no. 1, p. 3.
- G. Zhang and H. Li, ''Effectiveness of scaled exponentially-regularized linear units (SERLUs),'' Jul. 2018, arXiv:1807.10117. [Online]. Available: https://arxiv.org/abs/1807.10117
- T. DeVries and G. W. Taylor, ''Improved regularization of convolutional neural networks with cutout,'' Aug. 2017, arXiv:1708.04552. [Online].
- A. Mikolajczyk and M. Grochowski, ''Style transfer-based image syn- thesis as an efficient regularization technique in deep learning,'' in Proc. 24th Int. Conf. Methods Models Autom. Robot. (MMAR), Aug. 2019, pp. 42-47. open in new tab
- C. Tan, F. Sun, T. Kong, W. Zhang, C. Yang, and C. Liu, ''A survey on deep transfer learning,'' in Proc. 27th Int. Conf. Artif. Neural Netw. (ICANN), Rhodes, Greece, vol. 11141, Oct. 2018, pp. 270-279. open in new tab
- T. Elsken, J. H. Metzen, and F. Hutter, ''Neural architecture search: A sur- vey,'' J. Mach. Learn. Res., vol. 20, pp. 55:1-55:21, Mar. 2019. open in new tab
- B. Zoph and Q. V. Le, ''Neural architecture search with reinforcement learning,'' in Proc. 5th Int. Conf. Learn. Represent. (ICLR), Toulon, France, Apr. 2017, pp. 1-37.
- E. Real, ''Large-scale evolution of image classifiers,'' in Proc. 34th Int. Conf. Mach. Learn. (ICML), Sydney, NSW, Australia, vol. 70, Aug. 2017, pp. 2902-2911.
- T. Chen, I. J. Goodfellow, and J. Shlens, ''Net2Net: Accelerating learning via knowledge transfer,'' in Proc. 4th Int. Conf. Learn. Represent. (ICLR), San Juan, Puerto Rico, May 2016, pp. 1-12. open in new tab
- H. Liu, K. Simonyan, and Y. Yang, ''DARTS: Differentiable architecture search,'' in Proc. 7th Int. Conf. Learn. Represent. (ICLR), New Orleans, LA, USA, May 2019, pp. 1-13.
- ISIC Archive. Accessed: Sep. 24, 2019. [Online]. Available: https:// www.isic-archive.com open in new tab
- J. Bergstra and Y. Bengio, ''Random search for hyper-parameter optimiza- tion,'' J. Mach. Learn. Res., vol. 13, pp. 281-305, Feb. 2012.
- K. O. Stanley and R. Miikkulainen, ''Evolving neural networks through augmenting topologies,'' Evol. Comput., vol. 10, no. 2, pp. 99-127, Jun. 2002. open in new tab
- D. Floreano, P. Dürr, and C. Mattiussi, ''Neuroevolution: From architec- tures to learning,'' Evol. Intel., vol. 1, no. 1, pp. 47-62, Mar. 2008. open in new tab
- B. Zoph, V. Vasudevan, J. Shlens, and Q. V. Le, ''Learning transferable architectures for scalable image recognition,'' in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., Salt Lake City, UT, USA, Jun. 2018, pp. 8697-8710. open in new tab
- E. Real, A. Aggarwal, Y. Huang, and Q. V. Le, ''Regularized evolu- tion for image classifier architecture search,'' in Proc. AAAI, vol. 33, pp. 4780-4789, Sep. 2019. open in new tab
- H. Mendoza, A. Klein, M. Feurer, J. T. Springenberg, and F. Hutter, ''Towards automatically-tuned neural networks,'' in Proc. 33rd Workshop Autom. Mach. Learn. (AutoML) 2016, Int. Conf. Mach. Learn. (ICML), New York, NY, USA, Jun. 2016, vol. 64, pp. 58-65. open in new tab
- J. Bergstra, D. Yamins, and D. D. Cox, ''Making a science of model search: Hyperparameter optimization in hundreds of dimensions for vision architectures,'' in Proc. 30th Int. Conf. Mach. Learn. (ICML), Atlanta, GA, USA, Jun. 2013, vol. 28, pp. 115-123. open in new tab
- H. Cai, L. Zhu, and S. Han, ''ProxylessNAS: Direct neural architecture search on target task and hardware,'' in Proc. 7th Int. Conf. Learn. Repre- sent. (ICLR), New Orleans, LA, USA, May 2019, pp. 1-13.
- Y. Weng, T. Zhou, L. Liu, and C. Xia, ''Automatic convolutional neural architecture search for image classification under different scenes,'' IEEE Access, vol. 7, pp. 38495-38506, 2019. open in new tab
- Y. Weng, T. Zhou, Y. Li, and X. Qiu, ''NAS-Unet: Neural architec- ture search for medical image segmentation,'' IEEE Access, vol. 7, pp. 44247-44257, 2019. open in new tab
- A. Zela, A. Klein, S. Falkner, and F. Hutter, ''Towards automated deep learning: Efficient joint neural architecture and hyperparame- ter search,'' Jul. 2018, arXiv:1807.06906. [Online]. Available: https:// arxiv.org/abs/1807.06906
- A. Klein, S. Falkner, S. Bartels, P. Hennig, and F. Hutter, ''Fast Bayesian optimization of machine learning hyperparameters on large datasets,'' in Proc. 20th Int. Conf. Artif. Intell. Statist. (AISTATS), Fort Lauderdale, FL, USA, vol. 54, Apr. 2017, pp. 528-536. open in new tab
- P. Chrabaszcz, I. Loshchilov, and F. Hutter, ''A downsampled vari- ant of ImageNet as an alternative to the CIFAR datasets,'' Jul. 2017, arXiv:1707.08819. [Online]. Available: https://arxiv.org/abs/1707.08819
- B. Baker, O. Gupta, R. Raskar, and N. Naik, ''Accelerating neural architec- ture search using performance prediction,'' in Proc. 6th Int. Conf. Learn. Represent. (ICLR), Vancouver, BC, Canada, Apr./May 2018, pp. 1-14.
- S. Hochreiter and J. Schmidhuber, ''Long short-term memory,'' Neural Comput., vol. 9, no. 8, pp. 1735-1780, Nov. 1997. open in new tab
- R. J. Williams, ''Simple statistical gradient-following algorithms for connectionist reinforcement learning,'' Mach. Learn., vol. 8, nos. 3-4, pp. 229-256, May 1992. open in new tab
- T. Wei, C. Wang, Y. Rui, and C. W. Chen, ''Network morphism,'' in Proc. 33nd Int. Conf. Mach. Learn. (ICML), New York City, NY, USA, vol. 48, Jun. 2016, pp. 564-572. open in new tab
- H. Cai, T. Chen, W. Zhang, Y. Yu, and J. Wang, ''Efficient architec- ture search by network transformation,'' in Proc. 32nd AAAI Conf. Artif. Intell., (AAAI), 30th Innov. Appl. Artif. Intell. (IAAI), 8th AAAI Symp. Educ. Adv. Artif. Intell. (EAAI), New Orleans, LA, USA, Feb. 2018, pp. 2787-2794. open in new tab
- H. Cai, J. Yang, W. Zhang, S. Han, and Y. Yu, ''Path-level network trans- formation for efficient architecture search,'' in Proc. 35th Int. Conf. Mach. Learn. (ICML), Stockholm, Sweden, vol. 80, Jul. 2018, pp. 677-686.
- T. Elsken, J. H. Metzen, and F. Hutter, ''Simple and efficient architecture search for convolutional neural networks,'' in Proc. 6th Int. Conf. Learn. Represent. (ICLR), Vancouver, BC, Canada, Apr./May 2018, pp. 1-14. open in new tab
- F. Nachbar, W. Stolz, T. Merkle, A. B. Cognetta, T. Vogt, M. Landthaler, P. Bilek, O. Braun-Falco, and G. Plewig, ''The ABCD rule of der- matoscopy: High prospective value in the diagnosis of doubtful melanocytic skin lesions,'' J. Amer. Acad. Dermatol., vol. 30, no. 4, pp. 551-559, Apr. 1994. open in new tab
- A. Mikołajczyk, A. Kwasigroch, and M. Grochowski, ''Intelligent system supporting diagnosis of malignant melanoma,'' in Proc. Polish Control Conf., 2017, pp. 828-837. open in new tab
- M. Grochowski, A. Kwasigroch, and A. Mikołajczyk, ''Selected technical issues of deep neural networks for image classification purposes,'' Bull. Polish Acad. Sci., Tech. Sci., vol. 67, no. no. 2, pp. 363-376, 2019.
- I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning. Cambridge, MA, USA: MIT Press, 2016. VOLUME 8, 2020
- Verified by:
- Gdańsk University of Technology
seen 191 times