Thumbnail Image

Time-of-flight perception pipeline for selective green asparagus harvesting: Theory and application

Generally declining labour markets, coupled with a significant increase in the projected global population have raised concerns over a potential food scarcity crisis. In response to this, the agricultural industry is currently undergoing a technological revolution, especially with respect to robotic harvesting. Green asparagus, due to its unique physiology and labour intensive harvesting requirements, has been a long-time crop of focus in this field. Despite the relatively large body of work surrounding the development of selective robotic asparagus harvesting solutions, no such machine has yet ascended to the level of being commercially competitive. A critical component of a prospective selective robotic asparagus harvester, that is missing from contemporary machines, is a robust vision system, capable of detecting, and localising, harvest-eligible spears in real-time from a commercial asparagus field. This thesis presents a novel perception pipeline for time-of-flight images that can achieve this task, and describes the implementation of the proposed perception pipeline into two functional robotic harvesters, AHR-1 and AHR2, capable of harvesting green asparagus spears in a real-world commercial setting. The proposed perception pipeline achieves spear detection by firstly, segmenting the soil plane from an input pointcloud of an asparagus row, generated with a time-of-flight camera. Two plane detection methods, namely RANSAC and a modified version Hyun’s method (MHM) were investigated for this purpose. A detailed evaluation of the performance of each method on cluttered scenes revealed both RANSAC and MHM to be suitable for soil plane segmentation of asparagus beds, with both methods demonstrating similar RMSE error across a variety of scenes. The stability of model predictions made by RANSAC was generally found to be lower than that achieved by MHM, particularly for high-clutter scenes. This was determined to be due to the non-deterministic nature of RANSAC coupled with high degrees of soil plane occlusion in high clutter scenes. Following soil plane removal, non-asparagus points pertaining to rocks, weeds, and other field debris are then filtered from the scene. This is achieved by coarse filtering input points based on the output of a FRCNN model. The various FRCNN models utilised in this work were trained with a novel dataset of labelled images, collected from various asparagus farms throughout New Zealand and California, USA. Evaluation of these models revealed a typical maximum F1 score of 0.73, providing reasonable frame-by-frame identification of asparagus features. The remaining point clusters, representing each asparagus spear in the scene, are then filtered to remove flying pixels; a typical artefact of time-of-flight imaging. A novel geometric filter, named the closest point filter (CP filter), was developed for this task. Based on laboratory testing, it was found that this filter achieved a 68% reduction in the mean standard deviation of intra-cluster distance with respect to ground-truth positions, resulting in a significant improvement in the accuracy of base point predictions. The proposed perception pipeline is implemented inside a ROS framework, and deployed on two robotic harvesting platforms. The first platform, AHR-1, was developed as a proof-of-concept system. This system provided a wealth of knowledge which was utilised to develop a more complete prototype asparagus harvester, AHR-2. The design of AHR-1 and AHR-2 was largely informed by the shortcomings of existing asparagus harvesting robots from the literature, particularly with respect to their harvesting, and detection strategies. The literature surrounding robotic asparagus harvesters does not provide an adequate method for objectively evaluating the performance of such systems. Consequentially, performance metrics pertaining to existing robotic harvesters are relatively opaque, particularly with respect to the selective nature of the harvesting task. These inadequacies necessitated the development of a novel evaluation method for the evaluation of a selective asparagus harvester, which was utilised to measure the performance of both AHR-1 and AHR-2. Several field trials of AHR-1 and AHR-2 were conducted on farms throughout New Zealand and California, USA. The resulting analysis revealed that AHR-2 achieved state-of-the-art performance, harvesting 45.9% of all harvest eligible spears with a precision of 87.2% at its nominal ground speed of 0.3m/s. The vision system detected 97% of all harvest-eligible spears with a precision of 74.5% at this speed. When operating at a ground speed of 0.7m/s AHR 2’s harvesting rate fell to 22% with 95% precision. The corresponding drop in the vision systems detection rate was relatively small, dropping to 92.5% with 87.2% precision at a ground speed of 0.7m/s. From this it was concluded that the vision system outperformed the available hardware. The perception pipeline proposed by this thesis achieved state-of-the-art performance at a variety of ground speeds. The pipeline demonstrated a robustness to the unstructured nature of commercial asparagus fields, providing spear locations which successfully facilitated the robotic harvest of green asparagus spears.
Type of thesis
Peebles, M. C. S. (2021). Time-of-flight perception pipeline for selective green asparagus harvesting: Theory and application (Thesis, Doctor of Philosophy (PhD)). The University of Waikato, Hamilton, New Zealand. Retrieved from https://hdl.handle.net/10289/14605
The University of Waikato
All items in Research Commons are provided for private study and research purposes and are protected by copyright with all rights reserved unless otherwise indicated.