基于空间门控与边缘感知的苹果枝条多模态分割方法

Multimodal Instance Segmentation Method for Apple Branches Based on Spatial – Geometric Gating and Edge-awareness

摘要:

苹果树枝条实例分割既是苹果采摘机器人实现机械臂采摘路径规划与无损避障采摘的重要前提,又是信息采集机器人逐树逐枝感知果树表型信息的重要基础。本研究针对矮化密植苹果园非结构化环境下,光照剧变及背景干扰导致的枝条分割边界模糊,且末端细微枝条在深度学习网络的下采样过程中特征易丢失的难题,提出一种面向果园的跨模态注意力与边缘感知的苹果树枝条实例分割模型CAROL – YOLO( Cross-modal attention and refinement for orchard localization YOLO)。首先,设计了一种可见光图像辅助的“特征仿射非刚性变换”图像级联配准流程,构建了宽波段近红外(Visible – near infrared, Vis-NIR)图像深度图像配准的苹果树枝条多模态数据集;在此基础上,以YOLO v8为基础,设计了伪孪生双流空间几何门控骨干网络,通过该骨干网络,设计了空间几何门控模块(Spatial-geometry aware gating encoder, S-Gate)提取深度图像中的前景枝条空间位置,并自适应增强宽波段近红外图像中前景枝条的空间位置特征;同时设计了一种边缘感知模块(Edge-aware module,EAM),显式增强细微枝条高频边界信息,以增强深度学习网络下采样后果树枝条的分割精度。实验结果表明:CAROL – YOLO模型在不同的参数量级(Nano/ Small/ Large)上均取得一致性的性能提升。以CAROL – YOLO-s为例,模型mAP50-95达(39.6±0.1)% ,参数量仅为2.03×10?;与基线模型YOLO v8m seg相比,在参数量减少25. 6%的情况下,mAP50-95仍提升了1.4个百分点;与YOLO 11m-seg及YOLO 26m-seg模型相比,在参数量分别减少8.9%和24.5%的情况下,mAP50-95仍分别提升了0.8个百分点和1.4个百分点。该方法有效解决了果园非结构化复杂环境下的枝条漏检问题,同一模型不同大小下的一致性能提升可为不同类型果园机器人精细化作业提供可靠的感知算法支撑。

Abstract:

Instance segmentation of apple tree branches is essential for enabling robotic manipulators to plan harvesting paths and perform collision-free, damage-free picking. It also provides a fundamental basis for information-acquisition robots to perceive tree- and branch-level phenotypic traits. However, in unstructured high-density dwarf apple orchards, drastic illumination variations and background interference often result in blurred branch boundaries, while features of fine terminal branches are readily lost during downsampling in deep neural networks. To address these challenges, the cross-modal attention and refinement for orchard localization YOLO ( CAROL – YOLO), a cross-modal attention- and edgeaware instance segmentation model was proposed for apple branches in orchard environments. Firstly, a visible-light-assisted cascaded image registration pipeline integrating feature-based affine and nonrigid transformations was developed to construct a multimodal apple branch dataset comprising registered visible – near-infrared ( Vis-NIR) and depth images. Building on YOLO v8, a pseudo-Siamese dual-stream backbone with spatial-geometry-aware gating was then designed. Within this backbone, a spatialgeometry-aware gating encoder ( S-Gate) extracted the spatial locations of foreground branches from depth images and adaptively enhanced the corresponding spatial features in Vis NIR images. In addition, an edge-aware module ( EAM) was introduced to explicitly enhance the high-frequency boundary information of fine branches, thereby improving branch segmentation accuracy after network downsampling. Experimental results demonstrated that CAROL – YOLO consistently improved segmentation performance across different model scales (Nano, Small, and Large). In particular, CAROL – YOLO-s achieved an mAP50-95 of (39.6±0.1)% with only 2.03 × 10? parameters. Compared with the baseline YOLO v8m-seg, it improved mAP50-95 by 1. 4 percentage points while reducing the number of parameters by 25.6% . Compared with YOLO 11m seg and YOLO 26m-seg, it achieved improvements of 0.8 and 1.4 percentage points, respectively, while using 8.9% and 24.5% fewer parameters. The proposed method effectively mitigated missed detections of apple branches in complex, unstructured orchard environments. Its consistent performance gained across different model scales can provide reliable perceptual support for precision operations by various types of orchard robots.