The Robots Are Coming to Fieldays

A robotic asparagus harvester has been developed by the University of Waikato with the support of Callaghan Innovation and collaboration with RoboticsPlus Limited.

The fully operational prototype uses advanced robotics to harvest asparagus on commercial farms, and has been updated from its original iteration.

The original model was successfully demonstrated in California, USA in 2019, and the second iteration (known as version 1.5) developed in 2020 with successful field trials at asparagus farms in Matamata and Newstead last October and November.

“Asparagus growers like it,” says project lead Dr Shen Hin Lim, senior lecturer in Mechatronics and Mechanical Engineering at the University of Waikato. “They can see the potential and the need.”

Ag robots support the horticulture sector in a number of ways - automating easy and time-consuming tasks, enabling people to get on with other work.

“Agritech is very important to solving problems,” says Dr Lim, who is also the chair of NZ Robotics Automation and Sensing (NZRAS). “New Zealand is considered a world leader in agricultural innovation, and I believe that we can demonstrate that and have an edge using ag robots.”

The asparagus harvester has a vision system that detects the asparagus spears, computes their base location, and if it detects that the spear is tall enough to harvest, uses a robotic arm to cut it as the robot passes over.

With Covid-19 restrictions and closed borders impacting immigration and the labour force in New Zealand, the need for automated technology has increased, says Dr Lim.

“There is a need for ag robots and horticultural technology to solve problems, including the labour shortage on our orchards and farms,” says Dr Lim. “It’s not just that it’s back-breaking work, it’s difficult to find enough workers and that was even before Covid-19. Automating the process offers a sustainable solution.”

The original asparagus harvester was displayed at Fieldays 2019. This model is the second iteration - “version 1.5”, says Dr Lim.

The downside of the first version was its speed, says Dr Lim, and the new iteration improves on the original concept.

“The main reason we developed version 1.5 was to improve the cutting mechanism at the back, so it’s a quicker process.”

Dr Lim joined the University of Waikato in 2015 as a lecturer in mechanical engineering, and launched the mechatronic engineering programme at Waikato, which combines mechanical engineering with electronics.

Originally from Mentakab, a town near Kuala Lumpur in Malaysia, Dr Lim grew up with a love of robotics and how things worked. His office at the University of Waikato is full of elaborate Lego creations, a sign of his creativity and passion for building.

Dr Lim studied mechatronics engineering at the University of New South Wales (UNSW) in Sydney, getting his PhD and doing postdoctoral work before joining the University of Waikato.

He began working on the asparagus harvester robot in 2015, supported by PhD student Matt Peebles and a mechatronics engineer, Josh Barnett, and has collaborated with growers to develop the technology. The project involved partnering with asparagus grower Tendertips and robotics company, Robotics Plus.

“Robotics is a growth area world-wide,” says Dr Lim.

“Mechatronics is also a strong focus at Waikato University and we are creating some cutting edge horticultural robots here. We have particular expertise in hardware integration.”

The MaaraTech Vine Pruner has been developed by members of an innovative transdisciplinary co-design project funded by the Ministry of Business, Innovation & Employment (MBIE).

The collaboration includes researchers from the University of Waikato, Robotics Plus, the University of Auckland, the University of Canterbury, Lincoln Agritech, University of Otago and Plant & Food Research. Professor Mike Duke is the University of Waikato group leader.

The lack of skilled labour is driving the demand for ag robotics says Dr Benjamin McGuinness, a research and teaching fellow in mechanical engineering at the University of Waikato who is working on the project.

“Some of these orchardists and growers can’t get the numbers they require to do the job, especially with the covid-19 situation. Apples are one that have been in the media recently, with fruit left to rot in the orchards because they are unable to get the labour they need,” says Dr McGuinness.

Ag robotics typically includes two different streams, says Dr McGuiness: “full automation, which typically replaces a task that would be fulfilled by a human, and human-assist, where we are not taking the people out of the picture completely but helping them with some kind of mechatronic device or technology so they can do their job more effectively or efficiently”.

Auckland University holds the contract for the five-year MaaraTech project, and Waikato University is subcontracted to focus on the hardware development and physical aspects of creating the technology.

Teams from other universities have worked on different parts of the process which includes scanning and building a 3D model of the vine using cameras and sensors; detecting and measuring so the software has the necessary metrics to make a decision; then the mechatronics which take action.

“Taking action, that’s our strength,” says Dr McGuiness.

It's a work in progress, and the prototype has been developed over many iterations and modifications of commercial secuteurs.

The Waikato team have dubbed the cutting blade “the barracuda”, a name suggested by a grower.

Its innovative design helps ensure wires are not cut by mistake.

“People are really dexterous and can get around the wire, manipulate it and make the cut without cutting the wire. The design we’ve come up with has a series of slots in the anvil of the bottom blade. The idea is, the wire will fall into the slots but the cane is too big to fit in there, so the blade will cut the cane,” explains Dr McGuinness.

There are differing opinions on the best way to prune grapes. Typically, canes are spaced 1.5m apart and there are four canes on a plant - two on each side.

“The goal with cane pruning is to remove a cane, tie it down, and replace it with a younger one-year-old shoot. You want to pick canes coming close from the head that are down quite low, you want to keep growth close to the head.”

Cutting the right cane is critical.

“If you lose one of them, you use a quarter of your yield for the plant. The vine could take a couple of years to grow and recover.”

Work is still being done to refine the robot.

“At the moment it’s too slow, it moves around the plant, scanning it,” says Dr McGuinness. “Occlusion is a problem - you might not be able to see the whole plant as there are other canes in the way. We are still defining how many images you need to take to get a full model.”

The team have travelled to the Hawkes Bay to do much of their research, and have enjoyed collaborating with others on the project.

Colleagues at the University of Auckland and University of Canterbury are working on a virtual reality system that will also be on the Fieldays stand, giving punters the experience of working in a vineyard.

“You put on goggles and a vine will appear in front of you. It’s almost like a game. You make a cut and this cane might fall away; the software will tell you if you have made good decisions or not.”

Dr McGuiness started at the University of Waikato in 2003 in the School of Engineering and has done both his Master’s and PhD at Waikato.

He says that it is rewarding to be working on technology that solves real world problems.

The autonomous grape vine pruner reduces the reliance on manual labour and frees workers up to do other jobs.

“It’s not just about getting the numbers of people to do with work, but the need to retrain people each season in technical skills.”

“It’s too early” to predict the number of jobs the prototype could save, as is in the early stage.

“There is a very significant workforce doing this, and there is a long way to go before we’d actually replace them,” says Dr McGuinness.

The Zespri Orchard Survey Robot has been designed to help the kiwifruit industry make complex decisions and innovate in the face of challenges including labour shortages, disease management and increasing environmental compliance.

The survey robot uses a variety of sensors to autonomously navigate around the orchard to capture information that can provide actionable insights to growers, says Nick Pickering, a systems engineering lecturer at the University of Waikato, who is leading the project.

It can be programmed to capture a wide range of data through the life cycle of kiwifruit growing.

“We’ll be initiating the research later in the year starting with flower counting and canopy cover, with plans to expand the collaboration into the areas of pest and disease detection, fruit estimation and plant structure,” says Mr Pickering.

“The information from the robot will be used to support growers to make complex decisions to optimise fruit quality and quantity.”

In addition to assisting growers, there’s an opportunity to improve decision-making across the harvest, post-harvest, logistics and marketing functions.

The orchard survey robot has been funded by Zespri to advance innovation and research in the kiwifruit industry, to benefit all growers.

“It’s all about being better together,” says Mr Pickering.

“We didn’t want to build just another siloed system, so we’ve designed the robot and digital twin as a reusable platform to enable many different stakeholders, including industry and academic, to integrate their specialist sensors, models or visualisation tools. The aim is to enable partners to focus on what they do best rather than the funding being watered down by the need for everyone to build everything, every time.”

As a “common-user robot environment”, it will promote more collaboration and knowledge-sharing across the industry.

“Data capture is usually very expensive. Imagine, in the near future, we can have one operator dropping off a trailer-load of robots to navigate through the orchard with a variety of sensors on there being processed by artificial intelligence models from different collaborators to support productivity, environmental compliance and pest detection.  Think a Xero style partner ecosystem, but for orchard data capture and processing. That’s the long-term vision,” says Mr Pickering.

Mr Pickering has spent more than 20 years working in avionics and digital, including more than a decade in engineering integrating complex avionics systems and sensors, followed by 10 years in information technology across a range of digital platforms.

“Many of the sensors, hardware and artificial intelligence techniques we are utilising on orchards used to only be viable within the high budget domain of aviation. We’re now applying this Top Gun technology into agriculture robotics and have a real opportunity to leverage common equipment and digital technology to collectively benefit everyone, without impacting each partner’s own independence.”

Mr Picking is also a member of WaiRAS, a group of Waikato researchers specialising in artificial Intelligence, machine learning, computer vision, time-of-flight sensing, non-destructive testing, control and bespoke hardware development, robotics and data analytics.

“The problems we are trying to solve are complex,” says Mr Pickering. “Often companies come to us with a challenge or problem and we work with them to identify a range of potential solutions depending on their needs. WaiRAS has specialists in deep research focussing on the medium to long term but we also have the capability to apply research into novel applications to tackle a business need. We are collaborative, and work with other universities, organisations in industry and government agencies to bridge the traditional gap between research and commercialisation.”

He says that the airline industry provides an example of common-user equipment that the horticulture industry could follow.

“From my experience in the airline industry, I worked with competing airlines and government agencies that share software, equipment and information at the airport on a daily basis,” says Mr Pickering. “In addition to saving everyone money, the sharing of the information improves the end-to-end passenger journey through improved security, limited flight delays and a reduction in lost baggage.”

The survey robot and common-user robot environment is aimed to accelerate innovation and scale adoption for research, industry and government partners, for the betterment of New Zealand incorporated.

The prototype is still evolving.

“It’s early days,” says Mr Pickering.

He applauds Zespri for taking the initiative with technology, to boost innovation across the kiwifruit industry.

“It not only helps the growers grow more fruit which is good for them in the long run, it gives them a vision of what’s going on everywhere, across regions, and looking at weather and the seasons.”

He says growing good fruit is complex, with many factors to consider, but technology can support growers and the industry to get good data to guide good decision-making.

“The weather, the nutrients in soil, spraying, pruning, moisture and leaf wetness, the amount of sun, variety, altitude, wind – there are a lot of complex factors figuring out how kiwifruit can grow best. In other industries, there are vast amounts of siloed systems, but we are trying to build something that brings the data together for everyone’s benefit, while retaining their own independence.”

If the robot can deliver actionable insights to growers, the hope is it will aid with decision making and lead to optimum growth, delivering kiwifruit that tastes better, with increased fruit size and yield to improve overall profitability.

Orchard trials will be done in October and November, with analysis and reporting due by March 2021.

“This is a future-focused pilot in the robotics and automation space, but just one part of a much bigger strategy and a stream of work coming on line in this space," says Mark Graham, Orchard Innovation Lead at Zespri.