Abstract:Feeding intensity recognition and classification is an important link to realize accurate feeding in aquaculture. Existing feeding methods have problems such as over-reliance on manual experience judgment, imprecise feeding amount, and serious feed waste. Fish feeding degree classification based on multi-modal fusion can synthesize different types of data (e.g., video, sound, and water quality parameters) to provide a more comprehensive and accurate decision basis for fish feeding. Therefore, a multi-modal fusion framework that integrated video and audio data was proposed with the aim of improving the performance of sea bass feeding intensity classification. The preprocessed Mel Spectrogram (Mel) and video frame images were input into the self-attention-depthwise separable convolution-CNN6 (Self-Attention-DSC-CNN6) optimization model for high-level feature extraction, respectively, and the extracted features were further spliced and fused, and finally the spliced features were classified by a classifier. The Self-Attention-DSC-CNN6 optimization model was improved based on the CNN6 algorithm by replacing the traditional convolutional layers with depthwise separable convolution (DSC) to reduce the computational complexity, and the Self-Attention mechanism was introduced to enhance the feature extraction capability. The experimental results showed that the multi-modal fusion framework proposed achieved an accuracy of 90.24% in sea bass feeding intensity classification, and the model can effectively utilize the information from different data sources to improve the understanding of fish behavior in complex environments, enhance the decision-making ability of the model, and ensure the timeliness and accuracy of the feeding strategy, thus effectively reducing the waste of feed. This not only provided strong technical support for the intelligent management of aquaculture, but also laid the foundation for the development of intelligent feeding system.
