Importance of Photo-Realistic and Dedicated Simulator in Agricultural Robotics

Abstract

Agricultural robotics is a comparatively new field. Recently, it has gotten special attention because of agricultural labor shortages and rising labor costs. With the increasing operational use of agricultural robots, the need for a dedicated robotic simulator for such activity has been noted here. The need of a dedicated and specialty simulator for agricultural robotics research is the main focus area of this paper. Moreover, limitations have been pointed out when conventional robot simulators are used directly in agricultural settings. In a nutshell, it has been tried to emphasize that a dedicated simulator for agricultural robotics would be a timely advance and would accelerate the growth of agricultural robots.

Share and Cite:

Mansur, H. , Welch, S. , Dempsey, L. and Flippo, D. (2023) Importance of Photo-Realistic and Dedicated Simulator in Agricultural Robotics. Engineering, 15, 318-327. doi: 10.4236/eng.2023.155025.

1. Introduction

World population is increasing rapidly and expected to exceed 9 billion by 2050 [1] . To feed this many people, increases in world food production are mandatory. There are only two ways to surmount current limitations: one is to increase productivity per unit area and the second is to bring more land under cultivation. Agricultural robots have a vital role to play within both of these solutions. Agricultural robots are a combination of mostly ground robots and partially aerial robots (when unmanned aerial vehicle, UAVs are involved). Simulation is a common tool for robotic system development and not new to robotics developers, but recently it has drawn much attention because of the growing computational capabilities of computer hardware at a lower cost. The availability of robotic sensors like 3D LIDAR and depth cameras at lower prices has added more thrust to autonomous robot development. In recent years, autonomous robots have gained much attention, especially in agriculture, due to farm labor shortages [2] . Indeed, the agricultural robot market expected to grow from USD 4.9 billion in 2021 to USD 11.9 billion by 2026 [3] .

There are many stages in the development of a complete agricultural robotic system of which simulation is crucial as it provides a convenient way of testing robotic hardware and software. Such testing is especially important for outdoor robotic systems because of human safety issues and/or to avoid property damage. Also, outdoor robots face greater challenges than indoor ones due to the extensive variation in natural environments. Agricultural fields are no exception but are even more complex than external urban environments. Creation of the environment in which a robot operates is a crucial step in doing simulation of that robotic system. Various control algorithm types need to be evaluated against different field environments; for instance, picking robots operate within different environments than do weeding robots. But mimicking a real field condition is very challenging because so many variations exist in fields such as lighting conditions, dust levels, position/orientation of leaves/stems, etc. Therefore, it is impossible to create a simulation environment that will exactly match the conditions in any real field. That said, recent developments in the game industry have made environment creation easy and nowadays environments created by game engines look very natural. However, a natural appearance, while helpful for the simulation of vision-based sensors, is not enough. Also needed, are accurate physics engines that can deliver precise biophysical interactions, but this is still a distance goal because of the complex dynamics of both robotics systems and the physical world. There are several good review papers on robot simulators [4] [5] [6] and this paper does not attempt to do the same thing. It mostly focuses on the needs and importance of a dedicated simulator for doing agricultural robotics research.

2. Simulator for Agricultural Robotics

Shannon [7] gave an excellent definition of simulation as the process of designing a model of a real system and conducting experiments with this model to understand the system’s behavior and evaluate various strategies for operation of the system. A robot simulation program or robot simulator is a computer program used to mimic the real-world behavior of robots in different scenarios. Robot simulators are usually used to test robots where accurate hardware testing is difficult, costly, or dangerous to employ. There are different phases of doing robot simulation as shown in Figure 1 which is self-explanatory.

Simulation of robots for agricultural operations follows the same steps like traditional robot simulators with the special emphasis on environment creation. Here, the environment means the arrangement of crops or trees, lighting conditions, types of soil, conditions of dust or mud in the particular field in which robot operates. This environment is very dynamic and difficult to create (the reasons are mentioned in other sections). This is especially true if the environment

Figure 1. Common steps in robot simulation.

needs to be photo-realistic and similar to real world scenarios as close as possible. Here, comes the need for dedicated robot simulator in agricultural robotics research because it would help to create agricultural environment conveniently.

There are many different types of robot simulators. Some of them are open source or free and some of them are commercial. But most of them are for general purpose robots and not tailored to agricultural needs. Agricultural environments are different from those in which robots are most commonly used and, therefore, general purpose simulators cannot be used without custom modifications. In most cases, robots operating indoors, have a controlled environment or fixed arrangement of surroundings which is not the case for outdoor robotics. Outdoor environments, especially in agriculture, are very dynamic.

Gazebo is one of the most widely used robot simulators for all-purpose robotic applications. People have used it for agricultural simulation purpose [8] . It has easy integration with the Robot Operating System (ROS) which makes it more attractive to robotic community. It has a good collection of sensors ranging from simple ultrasonic sensor to complex 3D lidar sensor. CoppeliaSim (formerly V-rep) is another excellent robot simulator with many built-in models (such as houses, walls, roads etc.) that greatly enhance the creation of robotic operational environments. It allows easy interaction of those models for robot simulation. There are some other popular robot simulators such as Webots, MuJoCo, or CARLA which are general purpose robot simulator and not specifically designed for simulation of agricultural robots. There is a dedicated simulation platform for doing agricultural robotics simulation named as “Simulation Environment for Precision Agriculture Tasks using Robot Fleets” (SEARFS) [9] but it is not capable of doing photo-realistic visualization and hence not suitable for synthetic data generation that is useful for modern machine learning or image processing tasks. Some authors developed AgROS, a ROS based emulator for doing agricultural robotics research which purpose is similar to SEARFS but it also lacks the same visualization deficits [10] . Recently, there has been a trend to use game engines like Unreal Engine or Unity with a simulator to use the game engine’s capability of photo realistic rendering and its built-in physics engine. Example of this integration is CARLA (Unreal Engine), AirSim (Unreal Engine), RaiSim (Unity), Issac Sim (Unity + Unreal), and SOFA (Unity + Unreal). Although SOFA is a multi-purpose simulator and specifically intended for medical simulation. This integration makes the simulation more realistic and appealing but those simulators do not have any specific advantages of doing robotics simulation for agricultural operations.

3. Specific Needs for Agricultural Robot Simulation

There are mainly two types of environment for simulation of agricultural robots: indoor and outdoor environment. Indoor environment generally means greenhouse or similar types of controlled atmosphere where plants are grown in controlled manners. Outdoor environment usually means plants grown in open field conditions where there is no control of the weather and are fully subjected to natural conditions. These environments are very diverse and unpredictable. Different types of robots need different types of simulation environments as seen in Figure 2. Therefore, agricultural environment is not generalized like urban or

Figure 2. Environment for different types of agricultural robots [11] [12] . (a) Orchard environment; (b) Under crop canopy environment.

city traffic environment. Moreover, based on crop types and growth stage, the environments vary to a large extent.

All kinds of fruit-picking robots need some form of tactile or pressure sensors to sense the gripping of fruits unless it is vacuum picking of fruits. These grippers are not very accurate and their operating speed is also not very fast. Moreover, different fruits have different ripening stages, and simulating fruits needs soft body simulation. Hence, precise simulation of different fruits with varying degrees of softness requires soft body simulation as well as realistic operation of tactile sensors to simulate the fruit picking operations correctly.

Robots using vision systems need an accurate depiction of their surroundings for accurate work of vision algorithms. The crops are usually modeled using 3D modeling software like Blender, Maya, or 3D Studio Max. In most cases, people buy or use a free version of crop models and directly import it into the simulation environment. But these models are fixed and can’t be changed to include distortion of leaves or stems and only resizing of shape or rotation is allowed in the simulation environment which means the models are not very efficient to test vision algorithms because natural conditions are highly variable that is not comparable to standard models built by professional modelers. Therefore, provision should be incorporated in agricultural robotics simulators to include randomness and variability in plant morphology as per the need of a person who does the simulation.

Robots navigating autonomously in crop rows or under canopy situations, face several challenges such as muddy conditions, broken plants, plant debris, long weeds, and pivot ruts that are not common in urban or industrial environment. This kind of environment creation through an external program like Unreal Engine or Unity is challenging and time-consuming. Furthermore, it is very difficult to create under canopy environment using conventional robotics simulation software. It would have been convenient if there were some built-in environments for different plants exist in the database of simulator that could be easily placed during the simulation. Moreover, it would have been nice to have models of different crops/plants as well as different field conditions like muddy rows, a diverse range of plant debris and pivot ruts that could be easily dragged and dropped in the environment.

There has been a shifting trend of agricultural workforce to other professions and due to that fruit growers are facing severe labor shortage [13] . Travel restrictions due to COVID pandemic have accelerated this problem as few people are available for doing seasonal agricultural work. Many types of fruits and vegetables could be picked up by robots operating either in the greenhouse or in an open field. Fruits and vegetables have different maturity levels that need to be selected properly by the vision system of picking robots. Based on maturity levels, fruits and vegetables have different colors which are very important for vision algorithms as robots could pick under-mature or over-mature fruits/vegetables if improperly identified. Therefore, fruits/vegetables should look natural in terms of color. Photo-realistic visualization of plants as well as fruits/vegetables can serve this purpose but it is a challenging task and can take a very long time to produce even for an experienced 3D modeler. So, a specialty simulator for this kind of work would enhance the development process of harvesting robots.

Pruning and trimming are very important for plants and shrubs. These are very much essential for some plants such as grapevines. Not only grapevines, but most ornamental and commercially grown plants require pruning from time to time which is very labor-intensive work. Robots could play a big role here. But designing a robot that can do this work properly is very challenging and requires a series of trials to make the control algorithm perfect. Furthermore, this kind of robot needs photo-realistically rendered images for the vision system (to distinguish sections to cut) and a real-looking arrangement of leaves/stems/buds for a robust manipulator (to guide the cutter over plants) design. Hence, there is a need for a specialty simulation environment that can handle this kind of task.

Agricultural robots used to carry implements for different agricultural operations such as tilling, mowing and weeding. While doing these operations, implements interact with different types of soils. These interactions greatly influence overall power requirements of the robots. For small autonomous robots, power issue is very important and special consideration should be given while designing the robot. To address this, a physics engine that could replicate the interaction of different implements with soils should be incorporated in agricultural robot simulators.

Synthetic data (image) generation for machine learning algorithms is a very good candidate for photo-realistic simulation of agricultural environment. Small autonomous robot navigating through crop rows or under canopy situations, need simulated data of different environments to optimize their autonomous behavior such as navigation, harvesting, pruning or crop phenotyping. Without using any kind of simulation, generation of this kind of large volume data is very difficult and time consuming.

4. Drawbacks of Existing Simulators

Agricultural environments such as cultivated cropland, tree plantation or orchard are not static but, rather, change on short time scales. For example, the operational environment early in a growing season, when crops/trees are small, is quite different than later when the plants are mature. Hence, robots need different types of simulation environment to simulate agricultural operations properly. Functional-structural plant models (FSPM) are different from CAD (Computer Aided Design) models and special kind of models used to simulate growth of plant or trees at different stages considering plant’s physiology and environmental factors with 3D visualization of plant’s structure. But the creation of those models requires botanic or agronomic knowledge [14] which is generally not possessed by robotic system designers. Designers want to quickly create robot environments and conveniently test their algorithms across multiple scenarios. Moreover, FSPM generally model plants at different stages but not the complete environment (variations of soil surfaces, dust levels etc.) that is required by robotics designers. It focuses on crop or plants growth whereas CAD model are used to create robot operating environment.

Generally, the surfaces on which robots navigate in robotic simulators are not close to the real world of agriculture. This creates a problem for vision-based robots which use a camera to do certain tasks based on gathered images and hence the vision algorithms’ performance varies. For example, robots operating on flat ground or ground filled with stones, pebbles or boulders would not be the same. Sometimes, robotic vehicle control system relies on odometer inputs. But when there were stones present on the ground, wheels could be slipped and ultimately impacted the vehicle’s autonomous navigation performance. Moreover, grass and plant debris like leaves/stems exist in agricultural settings which are not common in an urban/industrial environment.

Plant/fruit models commonly used in agricultural simulation do not often output photo-realistic as shown in Figure 3. Such non-realistic models could not generate synthetic image sensor data that would be good enough for the purpose of autonomous navigation or the design of a robotic manipulator. The more realistic the models would be, the more usable data will be generated. Furthermore, it is well known that more realistic models would help in more realistic synthetic data generation using technique like GAN (Generative Adversarial Networks).

Distortion of acquired images by dust, mud or rain falling on the lens is a great problem for machine learning algorithms or vision-guided navigation and is very common in agricultural scenarios. Most of the widely used robot simulators can simulate cameras but not dirty lens effects. Water droplets from rain or moisture condensation can occur on camera lenses and render image analysis problematic. Not only water, but dust also plays a critical role in clear and crisp image acquisition in agricultural fields. Majority of agricultural field operations are usually done in summer when soils are dry. Dry soils are prone to generate dust even when little wind blows. Camera lenses contaminated with both water and dust can easily be unusable for vision-based tasks. Example of images from cameras obstructed with rain and dust are shown in Figure 4. Therefore, the more realistically we can simulate these situations, the more we can make a vision-based algorithm robust. The creation of this situation is not possible in a

Figure 3. CAD model of plants and fruits in simulation environment [15] .

Figure 4. Effects of water and dust on camera lens [16] . (a) Image distortion due to water; (b) Image distortion due to dust.

conventional robot simulator. At present, available simulation environments are not comparable to the real environments prevailing in agricultural fields.

In a real environment, variation of lighting conditions is quite common and that could make a sensor unusable. For example, infrared range sensors are widely used in field robotics but are very susceptible to lighting conditions. In very sunny conditions, they might not work at all or could give very poor results [17] . Most of the currently used simulation programs are not capable of mimicking real-life agricultural environments although recent improvements of game engines like Unreal Engine 5, Unity or robot simulator NVIDIA’s Isaac Sim have made a good stride toward photo-realistic rendering. Still, there is plenty of room for further development.

For most robot simulators, only wheels or tracks of robots are being considered which means the kinematics of robots are usually taken into consideration. But kinematics is not only important, the dynamics of robots play a vital role. Dynamic effects are mostly absent from conventional robotic simulators. For example, bumps from roads, and friction with grasses or roads/pavements are not taken into consideration although crucial for realistic simulation. Game engines like Unreal Engine provides some features to manipulate some parameters like steering angle, drag coefficient, and torque but there are many factors needed to include to make simulation dynamic in nature. Recent development in Nvidia’s Issac Sim simulator addressed several dynamics of a vehicle through the usage of their physics engine PhysX 5 but there are many factors needed to be addressed for complete dynamics analysis. In addition to that, most of these dynamic effects are tailored toward an industrial or urban environment and agricultural environments are not emphasized. Agricultural environments are different than urban or industrial environments and the dynamic behaviors of robots are also different.

Popular robot simulation software like Gazebo, CoppeliaSim, or Webots have several built-in objects or different types of robots or robotic manipulators but lack components for agricultural operations such as weeding, planting, or harvesting. AWS RoboMaker is a great tool to create an environment and simulate robots easily but it also lacks components for agricultural environment creation. Nvidia’s Isaac Sim simulator can do photo-realistic simulation but it does not have any kind of agricultural environments/components that can be easily used. In summary, there are some common tools such as different types of sensors in all of the simulators that could be used in agricultural simulation but the existing simulation environment is not enough to fulfill the simulation need of agricultural robotics research.

5. Conclusions

Agricultural robots could play a pivotal role in increasing food production as well as reducing production cost for labor intensive agricultural works. To develop robotic systems for agricultural operations, simulation is an essential and trusted technique for testing different algorithms to control robots or robotic manipulators. Recently, simulation in robotics has reached to a new level with the advent of high-power computing at low cost and the improvement of game engines. But simulation software is mostly targeted at simulating city, traffic, or industrial environment. Simulation of off-road conditions is not very common and a specialty simulator for modern agricultural robotics research is absent. In this paper, the need and justification for a dedicated simulator for agricultural robotics research have been presented. It has been shown that how the agricultural environment is different from a city or industrial environment and why there is a need for dedicated simulator to accelerate the growth of agricultural robotics. This kind of robot simulator will also help in simulation of off-road autonomous vehicles or robots.

Conflicts of Interest

The authors declare no conflicts of interest regarding the publication of this paper.

References

[1] Alexandratos, N. and Bruinsma, J. (2012) World Agriculture towards 2030/2050: The 2012 Revision. ESA Working Paper No. 12-03, FAO, Rome.
[2] Taylor, J.E., Charlton, D. and Yúnez-Naude, A. (2012) The End of Farm Labor Abundance. Applied Economic Perspectives and Policy, 34, 587-598.
https://doi.org/10.1093/aepp/pps036
[3] Report, R. (2023, January) Agricultural Robots Market by Type.
https://www.marketsandmarkets.com/Market-Reports/agricultural-robot-market-173601759.html
[4] Collins, J., Chand, S., Vanderkop, A. and Howard, D. (2021) A Review of Physics Simulators for Robotic Applications. IEEE Access, 9, 51416-51431.
https://doi.org/10.1109/ACCESS.2021.3068769
[5] Shamshiri, R.R., Hameed, I.A., Pitonakova, L., Weltzien, C., Balasundram, S.K. and Yule, I.J. (2018) Simulation Software and Virtual Environments for Acceleration of Agricultural Robotics: Features Highlights and Performance Comparison. International Journal of Agricultural and Biological Engineering, 11, 15-31.
https://doi.org/10.25165/j.ijabe.20181104.4032
[6] Bechar, A. and Vigneault, C. (2016) Agricultural Robots for Field Operations: Concepts and Components. Biosystems Engineering, 149, 94-111.
https://doi.org/10.1016/j.biosystemseng.2016.06.014
[7] Shannon, R.E. (1998) Introduction to the Art and Science of Simulation. Proceedings of the 1998 Winter Simulation Conference, Washington DC, 13-16 December 1998, 7-14.
https://doi.org/10.1109/WSC.1998.744892
[8] Biber, P., Weiss, U., Dorna, M. and Albert, A. (2012) Navigation System of the Autonomous Agricultural Robot Bonirob. In: Workshop on Agricultural Robotics: Enabling Safe, Efficient, and Affordable Robots for Food Production (Collocated with IROS 2012), Vilamoura.
[9] Emmi, L., Paredes-Madrid, L., Ribeiro, A., Pajares, G. and Gonzalez-de-Santos, P. (2013) Fleets of Robots for Precision Agriculture: A Simulation Environment. Industrial Robot, 40, 41-58.
https://doi.org/10.1108/01439911311294246
[10] Tsolakis, N., Bechtsis, D. and Bochtis, D. (2019) AgROS: A Robot Operating System based Emulation Tool for Agricultural Robotics. Agronomy, 9, Article No. 403.
https://doi.org/10.3390/agronomy9070403
[11] Hu, G., Chao, C., Jun, C., Sun, L., Sugirbay, A., Chen, Y., Jin, H., Zhang, J. and Bu, L. (2022) Simplified 4-DOF Manipulator for Rapid Robotic Apple Harvesting. Computers and Electronics in Agriculture, 199, Article ID: 107177.
https://doi.org/10.1016/j.compag.2022.107177
[12] Zhang, Z., Kayacan, E., Thompson, B. and Chowdhary, G. (2020) High Precision Control and Deep Learning-Based Corn stand Counting Algorithms for Agricultural Robot. Autonomous Robots, 44, 1289-1302.
https://doi.org/10.1007/s10514-020-09915-y
[13] Luo, T. and Escalante, C.L. (2017) US Farm Workers: What Drives Their Job Retention and Work Time Allocation Decisions? The Economic and Labour Relations Review, 28, 270-293.
https://doi.org/10.1177/1035304617703933
[14] Soualiou, S., Wang, Z., Sun, W., de Reffye, P., Collins, B., Louarn, G. and Song, Y. (2021) Functional-Structural Plant Models Mission in Advancing Crop Science: Opportunities and Prospects. Frontiers in Plant Science, 12, Article ID: 747142.
https://doi.org/10.3389/fpls.2021.747142
[15] Shamshiri, R.R., Hameed, I.A. and Weltzien, M.K. (2018) Ch. 5. Robotic Harvesting of Fruiting Vegetables: A Simulation Approach in V-REP, ROS and MATLAB. In: Hussmann, S., Ed., Automation in Agriculture—Securing Food Supplies for Future Generations, IntechOpen, Rijeka.
https://doi.org/10.5772/intechopen.73861
[16] Dellmann, T. and Berns, K. (2022) Toward a Realistic Simulation for Agricultural Robots. In: Ronzhin, A.B., Ed., Agriculture Digitalization and Organic Production. Smart Innovation, Systems and Technologies, Springer, Singapore, 3-13.
https://doi.org/10.1007/978-981-16-3349-2_1
[17] Ismail, R., Omar, Z. and Suaibun, S. (2016) Obstacle-Avoiding Robot with IR and PIR Motion Sensors. IOP Conference Series: Materials Science and Engineering, 152, Article ID: 012064.
https://doi.org/10.1088/1757-899X/152/1/012064

Copyright © 2024 by authors and Scientific Research Publishing Inc.

Creative Commons License

This work and the related PDF file are licensed under a Creative Commons Attribution 4.0 International License.