RGB and Thermal Camera Integration for Advanced Perception of an Agricultural Robot

Over the past six decades, significant advancements in computer vision, a key aspect of Artificial Intelligence (AI), have revolutionized fields such as robotics and environmental monitoring. Initially inspired by neurophysiological discoveries in 1959, computer vision now enables machines to interpret and extract information from digital images and videos.

In robotics, it facilitates autonomous navigation, object recognition, and intelligent interaction with surroundings. In environmental monitoring, it supports ecological studies and climate research through UAVs and agricultural autonomous robots (AMRs). However, comprehensive and diverse datasets are essential for accurate analysis and training in computer vision. This thesis explores the synergy between computer vision, robotics, and environmental monitoring, leveraging the integration of visual perception technologies with robotic systems to address challenges and drive innovation in these interconnected fields.

The primary focus of this research is the development of a dual camera system composed of an Intel RealSense RGBD camera and an Optris Xi 400 thermal camera, aimed at detecting the Crop Water Stress Index (CWSI) in lettuce plants. The dual camera system integrates the color imaging capabilities of the Intel RealSense camera with the thermal imaging capabilities of the Optris Xi 400, providing a valuable instrument for analyzing plant health and water stress level. The RGBD camera captures high-resolution color images, enabling analysis of plant morphology and structure. The thermal camera detects temperature variations across the plant surface, indicative of water stress levels.

By combining these two data streams, the system can assess the CWSI, a critical parameter for determining plant water status. The development of this system involved several stages. Initially, the cameras were calibrated and synchronized to ensure accurate data fusion. YOLOv8, an advanced image processing algorithm, was employed to merge the RGBD and thermal data, allowing for precise identification of stressed areas on the plants. The system was tested and validated in various environmental conditions to ensure its robustness and accuracy. Field experiments demonstrated the system’s effectiveness in real-time water stress detection.

This dual camera system has broader applications in precision agriculture and environmental monitoring, offering a versatile tool for sustainable farming. Future work will address challenges such as dataset requirements and performance improvements in extreme conditions to enhance the system’s capabilities further.

Project Date: