Abstract:The rapid development of high-throughput crop phenotyping acquisition equipment has provided modern data collection means for breeding and cultivation research, while spawning massive multi-modal and unstructured phenotypic data. Traditional structured data storage models can no longer meet the efficient access requirements of such data.A hybrid access framework was proposed based on distributed technology, which used HBase and HDFS to build a structured and unstructured fusion storage engine, integrated client-side cache and Redis cache to design an efficient retrieval mechanism, and optimized core issues: aiming at the inherent defects of native HDFS in storing phenotypic data, a modal aggregation-based MCH storage framework was designed. By classifying and merging phenotypic data according to modalities and constructing local indexes by using double-layer hashing technology, it effectively reduced NameNode memory pressure while improving access efficiency and storage space utilization of single-modal data. For high-concurrency data reading scenarios, a double-layer cache mechanism based on data popularity was constructed. It optimized hot data reading efficiency through metadata hierarchical caching and innovatively proposed a data popularity evaluation model combining access frequency and time characteristics, which effectively improved cache hit rate. Experimental results showed that when the data scale was 1.0×105, the proposed distributed access method reduced the NameNode memory occupancy rate by 31.2% compared with the optimal native solution (SequenceFile), and the retrieval time by 25.4% compared with the optimal native solution (MapFile), providing technical support for the storage and retrieval of massive multi-modal phenotypic data.
