基于单视角RGBD图像的柑橘果实三维重建与表型检测方法

Three-dimensional Reconstruction and Phenotype Detection of Citrus Fruits Based on Single-view RGBD Images

摘要:

水果表型的测量和分析是植物育种和遗传学研究的一个重要领域。单视角RGBD图像的表型检测方法通量高、成本低，但受限于传感器分辨率和视角，通常无法获取果实的表面积和体积等数据。本文提出了一种基于PFNET的点云补全网络改进方法，可使用深度相机获取的类球形果实单视角点云进行高精度三维重建并进行表型无损测量。为解决补全网络输入比例不固定的问题，提出了一种自适应几何补全策略将单视角点云补全为近似的半球。在PFNET网络框架上增加了第4尺度，以充分利用KINECT相机获取的稠密点云，有利于复杂形状和细节丰富的结构补全。通过引入四头自注意力模块，能更好地捕捉点云中各点间的相互依赖和空间关系，提升网络特征提取能力。增添了果实点云优化模块，解决原网络生成点云存在局部扩散的问题并提升点云质量，模拟人工测量方式设计了针对性的表型检测方法。实验结果表明，该方法与结构光三维扫描仪获取的柑橘果实点云质量接近，三维重建还原度高。对于横径、纵径、表面积和体积4种表型检测的R2均大于0.96，平均测量精度均超过93.24%。与RGBD图像法相比，单果检测时间增加17.97 s，但横纵径检测精度大幅提高，且能一次测量4项表型参数。与三维扫描仪方法相比，检测精度差值在4个百分点以内，但速度超过48倍，硬件成本只有后者的1/10，且易于实现自动化。本文方法在检测精度、运行速度、硬件成本和自动化程度上具有较好的平衡，是一种低成本、综合性能高的三维重建技术，有广泛应用于类球形果实表型无损测量的潜力。

Abstract:

The measurement and analysis of fruit phenotypes are crucial aspects of plant breeding and genetics research. Methods for phenotype detection using single-view RGBD images offer high throughput and low cost but are limited by sensor resolution and perspective, often failing to obtain data such as the surface area and volume of fruits. An improved method was proposed based on PFNET that used a depth camera to capture single-view point clouds of spherical-like fruits for high-precision 3D reconstruction and non-invasive phenotype measurements. To address the issue of varying input scales in the completion network, an adaptive geometric completion strategy was introduced to transform single-view point clouds into approximate hemispheres. The addition of a fourth scale to the PFNET framework enhanced the utilization of dense point clouds acquired by KINECT cameras, facilitating the completion of complex shapes and structures rich in detail. By incorporating a four head self-attention module, the network’s ability to capture interdependencies and spatial relationships among points in the point cloud was improved, enhancing feature extraction capabilities. An optimized fruit point cloud module was added to resolve issues with local diffusion in the original network generated point clouds and to improve their quality. A targeted phenotype detection method, designed to mimic manual measurements, was also proposed. Experimental results showed that this method achieved point cloud quality comparable to that obtained by structured light 3D scanners for citrus fruits, with high fidelity in 3D reconstruction. For the detection of four phenotypes—transverse diameter, longitudinal diameter, surface area, and volume—the R2 values exceeded 0.96, with average measurement accuracy above 93.24%, approaching that of 3D scanners but at 50 times of the efficiency and one-tenth of the cost. Compared with RGBD image methods, the single fruit detection time was increased by 17.97 s, but there was a significant improvement in transverse and longitudinal diameter detection accuracy, allowing for the simultaneous measurement of four phenotypic parameters. Compared with the 3D scanner method, the difference in detection accuracy was within 4 percentage points, but the speed was more than 48 times faster, with hardware costs reduced to only one-tenth of the latter’s and easier implementation of automation. This method struck a good balance between detection accuracy, operational speed, hardware cost, and level of automation, offering a cost-effective, high-performance 3D reconstruction technology with great potential for non-invasive measurement of spherical-like fruit phenotypes.