融合类别数量自适应深度数据增强和迁移学习的造礁珊瑚识别方法研究

王岚; 魏皓; 车亚辰; 张翠翠

doi:10.12284/hyxb2024096

融合类别数量自适应深度数据增强和迁移学习的造礁珊瑚识别方法研究

doi: 10.12284/hyxb2024096

王岚^1,,
魏皓¹,
车亚辰²,
张翠翠^{1, 3, ,}

1.
天津大学海洋科学与技术学院天津 300072
2.
国家海洋技术中心自然资源部海洋观测技术重点实验室天津 300112
3.
中国科学院计算技术研究所北京 100190

基金项目: 国家重点研发计划项目（2022YFC3104600）；海南省重点研发计划项目（ZDYF2024SHFZ051）；国家自然科学基金面上项目（42076007）；自然资源部海洋观测技术重点实验室定向基金（klootB06）。

详细信息

作者简介:
王岚（1997—），女，四川省绵阳市人，主要从事图像识别研究。E-mail：17721910924@163.com

通讯作者:
张翠翠（1986—），女，山东省滨州市人，副教授，主要从事智能海洋计算、模式识别研究。E-mail：cuicui.zhang@tju.edu.cn

中图分类号: P714⁺.5
计量
- 文章访问数: 100
- HTML全文浏览量: 50
- PDF下载量: 5
- 被引次数: 0
出版历程
- 收稿日期: 2023-08-02
- 修回日期: 2024-05-08
- 网络出版日期: 2024-08-15
- 刊出日期: 2024-09-01

Integration of category-quantity adaptive deep data augmentation and transfer learning for reef-building coral recognition

Wang Lan^1
,,
Wei Hao¹,
Che Yachen²,
Zhang Cuicui^{1, 3
, ,}

1.
School of Marine Science and Technology, Tianjin University, Tianjin 300072
2.
Key Laboratory of Ocean Observation Technology of Ministry of National Resources, National Ocean Technology Center, Tianjin 300112
3.
Institute of Computing Technology, Chinese Academy of Sciences, Beijing, 100190

摘要

摘要: 造礁珊瑚识别对于珊瑚礁生态系统的保护与监测具有重要意义。深度学习作为图像识别的前沿技术，在珊瑚识别领域逐渐得到应用。然而，其识别性能仍然面临挑战。其中，数据集中类别间样本数量不平衡和数据多样性欠缺是两个主要问题。前者使得深度学习模型在特征提取过程中更偏向于样本数较多的类，对少数类（尤其是濒危珊瑚）的学习能力不足进而影响其识别准确度。后者因为数据缺乏多样性使得模型无法充分学习各种珊瑚特征，进而限制了特征提取的能力。鉴于此，本文提出了一种融合类别数量自适应深度数据增强和迁移学习的造礁珊瑚类型识别方法。针对第一个问题，本文利用识别结果评价指标F₁-score定义的数据生成量化公式对原始深度数据增强方法DeepSMOTE进行改进，提出了类别数量自适应的深度数据增强方法DeepSMOTE-F₁。该方法根据每类珊瑚的识别结果自适应地增强其样本数量，确保模型充分学习各类珊瑚特征。针对第二个问题，利用迁移学习强化了模型的提取能力。实验结果表明，在RSMAS、EILAT和EILAT2这3个代表性珊瑚识别数据集上，相较于原始DeepSMOTE，本文提出的DeepSMOTE-F₁识别准确率分别提升了2.88%、0.39%和1.54%；与现有的珊瑚智能识别方法相比，准确率分别提升了0.76%、1.40%和1.30%。
- 珊瑚识别 /
- 深度学习 /
- 数据集不平衡 /
- 数据增强 /
- 迁移学习
Abstract: Recognition of reef-building corals is important for protecting and monitoring coral reef ecosystems. Deep learning, as an advanced technology in image recognition, has been increasingly applied in coral recognition. However, its performance is still challenged by several issues, such as the imbalance of samples among different coral categories within a dataset and the limitation of data diversity. The former makes the deep learning model more likely to extract features from classes with a large number of samples and, therefore, decreases its ability to recognize small-sample-size corals, which often refer to endangered ones needing to be protected. The latter further reduces the performance of deep learning in recognizing corals with different appearances and are captured in variant environments. To solve these two problems, this study develops a reef-building coral recognition method by integrating a category-quantity adaptive deep data augmentation algorithm and transfer learning. To address the first problem, a category-quantity adaptive deep data augmentation algorithm named DeepSMOTE-F₁ is proposed. This algorithm improves the existing DeepSMOTE by introducing a sample-size determination stagey using an F1-score based evaluation metric. It can adaptively augment the number of samples of each category of corals according to its recognition performance so that the deep learning model can fully learn features from each class of corals. For the second problem, transfer learning is used to further enhance the model's ability to extract features. The experimental results on three widely used public coral recognition datasets, RSMAS, EILAT, and EILAT2 show that the recognition accuracy of the proposed DeepSMOTE-F₁ is improved by 2.88%, 0.39%, and 1.54%, respectively, compared with the traditional DeepSMOTE; and the accuracy of the integrated method is improved by 0.76%, 1.40% and 1.30% compared with the existing deep learning methods for coral recognition.
- coral recognition /
- deep learning /
- imbalanced dataset /
- data augmentation /
- transfer learning.

HTML全文

图 1 更改后的ResNet-50网络结构

Fig. 1 The network structure of the modified ResNet-50

下载: 全尺寸图片幻灯片

图 2 SMOTE采样示意图

Fig. 2 The diagram of SMOTE sampling

下载: 全尺寸图片幻灯片

图 3 DeepSMOTE深度数据增强结果示例：a中展示了RSMAS数据集^[20]中3个珊瑚类：Colpophyllia natans, Acropora cervicornis和Meandrina meandrites，b中展示了EILAT数据集^[20]中3个珊瑚类：Branches TypeⅡ, Brain Coral和Favid Coral。每个类有3个示例，从左到右依次为原始图像、近邻图像、新图像。右上角的数值是由SMOTE算法随机确定的比例因子，介于0～1，表示新的图像与原始图像和近邻图像的接近程度。值越大则代表越接近原始图像，值越小则代表越接近近邻图像

Fig. 3 The example results of deep data augmentation using DeepSMOTE: a. displays three classes of coral in the RSMAS dataset^[20]: Colpophyllia Natans, Acropora Cervicornis and Meandrina Meandrites, b. displays three classes of coral in the EILAT dataset^[20]: Branches TypeⅡ, Brain Coral和Favid Coral. There are three examples provided for each class. The 1st−3rd column shows the original image, the nearest neighbor image, and the generated image, respectively. The value in the top right corner is the scale factor randomly determined by the SMOTE, which ranges from 0 to 1. It indicates the degree of similarity between the generated image and the original image, as well as the nearest neighbor image. A higher value indicates greater similarity with the original image, while a lower value indicates greater similarity with the nearest neighbor image

下载: 全尺寸图片幻灯片

图 4 DeepSMOTE-F₁流程图

Fig. 4 Flow chart of the DeepSMOTE-F₁

下载: 全尺寸图片幻灯片

图 5 整体方法框架

Fig. 5 The framework of the holistic method

下载: 全尺寸图片幻灯片

图 6 每个敏感性实验在3个数据集（a. EILAT2; b. EILAT; c. RSMAS）上的各类别F₁-score

Fig. 6 The F₁-score for each class on the three datasets (a. EILAT2; b. EILAT; c. RSMAS)

下载: 全尺寸图片幻灯片

图 7 采用DeepSMOTE-F₁前后各类珊瑚图像数量对比（a. EILAT2; b. EILAT; c. RSMAS）

Fig. 7 The comparison of the number of samples before and after data augmentation using DeepSMOTE-F₁(a. EILAT2; b. EILAT; c. RSMAS)

下载: 全尺寸图片幻灯片

表 1 RSMAS数据集基本信息^[12]

Tab. 1 The information of the RSMAS dataset

数据集	类	数量
RSMAS	Acropora cervicornis	109
	Acropora palmata	77
	Colpophyllia natans	57
	Diadema antillarum	63
	Diploria strigosa	24
	Gorgonians	60
	Millepora alcicornis	22
	Montastraea cavernosa	79
	Meandrina meandrites	54
	Montipora spp.	28
	Palythoas palythoa	32
	Sponge fungus	88
	Siderastrea siderea	37
	Tunicates	36
总数		766

下载: 导出CSV

表 2 EILAT数据集基本信息^[12]

Tab. 2 The basic information of the EILAT dataset

数据集	类	数量
EILAT	Sand	87
	Urchin	80
	Dead Coral	280
	Brain Coral	160
	Favid Coral	200
	Branches TypeⅠ	23
	Branches TypeⅡ	216
	Branches TypeⅢ	77
总数		1123

下载: 导出CSV

表 3 EILAT2数据集基本信息^[24]

Tab. 3 The basic information of the EILAT2 dataset

数据集	类	数量
EILAT2	Sand	80
	Urchin	14
	Brain Coral	71
	Favid Coral	89
	Branches Type	49
总数		303

下载: 导出CSV

表 4 参数设置

Tab. 4 The setting of parameters

模型	批次大小	学习率	迭代次数
ResNet-50	32	0.001/0.0001/0.00001	300/500/1000
	64	0.001/0.0001/0.00001	300/500/1000
	128	0.001/0.0001/0.00001	300/500/1000

下载: 导出CSV

表 5 每个敏感性实验在RSMAS、EILAT和EILAT2数据集上的识别准确率

Tab. 5 The classification accuracy of each sensitivity experiment on RSMAS, EILAT, and EILAT2 datasets

方法	RSMAS	EILAT	EILAT2
基线	84.19%	74.28%	74.23%
DeepSMOTE	85.00%	79.22%	80.08%
DeepSMOTE-F₁	87.88%	79.61%	81.62%
迁移学习	97.54%	94.84%	97.06%
DeepSMOTE-F₁ + 迁移学习	98.81%	98.02%	99.01%

下载: 导出CSV

表 6 本文方法提供最佳识别准确率的参数设置

Tab. 6 The setting of parameters of the proposed method on achieving the highest classification accuracy

数据集	批次大小	学习率	迭代次数
RSMAS	64	0.001	500
EILAT	32	0.001	300
EILAT2	32	0.001	300

下载: 导出CSV

表 7 与现有珊瑚识别方法在RSMAS、EILAT和EILAT2数据集上的识别准确率对比

Tab. 7 The comparison of classification accuracy with existing coral classification methods on RSMAS, EILAT and EILAT2 datasets

方法	RSMAS	EILAT	EILAT2
ReasFeats^[10]	97.42%	96.00%	96.83%
MDNet^[11]	89.70%	94.70%	91.20%
ResNet+Augmentation^[12]	98.05%	96.62%	97.71%
DeepSMOTE-F₁+迁移学习	98.81%	98.02%	99.01%

下载: 导出CSV

参考文献(28)

[1]	黄晖, 陈竹, 黄林韬. 中国珊瑚礁状况报告(2010−2019)[M]. 北京: 海洋出版社, 2021. Huang Hui, Chen Zhu, Huang Lintao. Status of Coral Reefs in China (2010−2019)[M]. Beijing: China Ocean Press, 2021.
[2]	Elliff C I, Silva I R. Coral reefs as the first line of defense: shoreline protection in face of climate change[J]. Marine Environmental Research, 2017, 127: 148−154. doi: 10.1016/j.marenvres.2017.03.007
[3]	黄林韬, 黄晖, 江雷. 中国造礁石珊瑚分类厘定[J]. 生物多样性, 2020, 28(4): 515−523. doi: 10.17520/biods.2019384 Huang Lintao, Huang Hui, Jiang Lei. A revised taxonomy for Chinese hermatypic corals[J]. Biodiversity Science, 2020, 28(4): 515−523. doi: 10.17520/biods.2019384
[4]	夏荣林, 宁志铭, 余克服, 等. 长棘海星暴发对珊瑚礁区沉积物营养盐动力学的影响研究[J]. 海洋学报, 2022, 44(8): 23−30. doi: 10.12284/j.issn.0253-4193.2022.8.hyxb202208003 Xia Ronglin, Ning Zhiming, Yu Kefu, et al. Study on the impacts of crown-of-thorns starfish on nutrient dynamics in the coral reef sediments[J]. Haiyang Xuebao, 2022, 44(8): 23−30. doi: 10.12284/j.issn.0253-4193.2022.8.hyxb202208003
[5]	International Union for Conservation of Nature. The IUCN red list of threatened species, version 2022−2[R/OL]. [2023−07−06]. https://www.iucnredlist.org.
[6]	Mahmood A, Bennamoun M, An S, et al. Deep learning for coral classification[M]//Samui P, Sekhar Roy S, Balas V E. Handbook of Neural Computation. London: Academic Press, 2017: 383-401.
[7]	Marcos M S A C, Soriano M N, Saloma C A. Classification of coral reef images from underwater video using neural networks[J]. Optics Express, 2005, 13(22): 8766−8771. doi: 10.1364/OPEX.13.008766
[8]	Pizarro O, Rigby P, Johnson-Roberson M, et al. Towards image-based marine habitat classification[C]//Proceedings of the OCEANS 2008. Quebec City, Canada: IEEE, 2008: 1−7.
[9]	Elawady M. Sparse coral classification using deep convolutional neural networks[J]. arXiv: 1511.09067, 2015.
[10]	Mahmood A, Bennamoun M, An S, et al. Resfeats: residual network based features for image classification[C]//Proceedings of 2017 IEEE International Conference on Image Processing. Beijing, China: IEEE, 2017: 1597−1601.
[11]	Modasshir M, Li A Q, Rekleitis I. MDNet: multi-patch dense network for coral classification[C]//Proceedings of the OCEANS 2018 MTS/IEEE Charleston. Charleston, USA: IEEE, 2018: 1−6.
[12]	Gómez-Ríos A, Tabik S, Luengo J, et al. Towards highly accurate coral texture images classification using deep convolutional neural networks and data augmentation[J]. Expert Systems with Applications, 2019, 118: 315−328. doi: 10.1016/j.eswa.2018.10.010
[13]	Lumini A, Nanni L, Maguolo G. Deep learning for plankton and coral classification[J]. Applied Computing and Informatics, 2023, 19(3/4): 265−283. doi: 10.1016/j.aci.2019.11.004
[14]	Khan S H, Hayat M, Bennamoun M, et al. Cost-sensitive learning of deep feature representations from imbalanced data[J]. IEEE Transactions on Neural Networks and Learning Systems, 2018, 29(8): 3573−3587. doi: 10.1109/TNNLS.2017.2732482
[15]	Lee H, Park M, Kim J. Plankton classification on imbalanced large scale database via convolutional neural networks with transfer learning[C]//Proceedings of 2016 IEEE International Conference on Image Processing. Phoenix, USA: IEEE, 2016: 3713−3717.
[16]	Dablain D, Krawczyk B, Chawla N V. DeepSMOTE: fusing deep learning and SMOTE for imbalanced data[J]. IEEE Transactions on Neural Networks and Learning Systems, 2023, 34(9): 6390−6404. doi: 10.1109/TNNLS.2021.3136503
[17]	Chinchor N. MUC-4 evaluation metrics[C]//Proceedings of the 4th Conference on Message Understanding. McLean, USA: Association for Computational Linguistics, 1992: 22−29.
[18]	Alzubaidi L, Bai J, Al-Sabaawi A, et al. A survey on deep learning tools dealing with data scarcity: definitions, challenges, solutions, tips, and applications[J]. Journal of Big Data, 2023, 10(1): 1−82.
[19]	Pan S J, Yang Qiang. A survey on transfer learning[J]. IEEE Transactions on Knowledge and Data Engineering, 2010, 22(10): 1345−1359. doi: 10.1109/TKDE.2009.191
[20]	Shihavuddin A S M. Coral reef dataset[EB/OL]. [2024-01-22]. https://data.mendeley.com/datasets/86y667257h/2.
[21]	WoRMS. An authoritative classification and catalogue of marine names[EB/OL]. [2024-04-05]. https://www.marinespecies.org.
[22]	Nanglu K, Lerosey-Aubril R, Weaver J C, et al. A mid-Cambrian tunicate and the deep origin of the ascidiacean body plan[J]. Nature Communications, 2023, 14(1): 3832. doi: 10.1038/s41467-023-39012-4
[23]	Rodríguez L, López C, Casado-Amezua P, et al. Genetic relationships of the hydrocoral Millepora alcicornis and its symbionts within and between locations across the Atlantic[J]. Coral Reefs, 2019, 38(2): 255−268. doi: 10.1007/s00338-019-01772-1
[24]	Shihavuddin A S M, Gracias N, Garcia R, et al. Image-based coral reef classification and thematic mapping[J]. Remote Sensing, 2013, 5(4): 1809−1841. doi: 10.3390/rs5041809
[25]	Buda M, Maki A, Mazurowski M A. A systematic study of the class imbalance problem in convolutional neural networks[J]. Neural Networks, 2018, 106: 249−259. doi: 10.1016/j.neunet.2018.07.011
[26]	Nair V, Hinton G E. Rectified linear units improve restricted boltzmann machines[C]//Proceedings of the 27th International Conference on International Conference on Machine Learning (ICML-10). Haifa, Israel: Omnipress, 2010: 807−814.
[27]	Chawla N V, Bowyer K W, Hall L O, et al. SMOTE: synthetic minority over-sampling technique[J]. Journal of Artificial Intelligence Research, 2002, 16(1): 321−357.
[28]	Deng Jia, Dong Wei, Socher R, et al. ImageNet: a large-scale hierarchical image database[C]//Proceedings of 2009 IEEE Conference on Computer Vision and Pattern Recognition. Miami, USA: IEEE, 2009: 248−255.