04 — Publications / Explainer

Near harvest, the AI looked
at the grain heads to predict yield

Matching the light and the image size to the growth stage made predictions more accurate

European Journal of Agronomy, 2025 Read the paper ↗ 日本語で読むLeer en español
Drone image looking straight down on the study fields in Tomioka, Fukushima. Golden and green rice fields lie next to brown fields that have already been harvested.
The study fields in Tomioka, Fukushima Prefecture.Adapted from Figure 2 of Yamaguchi et al. (2025), CC BY-NC 4.0

We use drone images of paddy fields and artificial intelligence (AI) to predict how much rice a field will yield. Here we changed when the images were taken, which kinds of light they contained, and how much area each one covered, to find the combination that predicts most accurately. Near harvest, the best results came from light that included red edge and from cutting out each image a little wider than the harvested patch. Maps of where the AI looked showed that it focused on the grain heads (panicles) near harvest, which explains these results.

Key points

  1. The best combination of light depended on when the images were taken. Near harvest, the combination of green, near-infrared, and red edge light gave the most accurate predictions.
  2. Cutting out each image a little wider than the patch we harvested made predictions more accurate. The gain was largest for images taken near harvest.
  3. Maps of where the AI looked showed that it attended to the whole plant at mid-ripening (about three weeks after the panicles appear) and to the panicles near harvest. This difference explains the first two findings.
Dr. Hoo, where in a photo does the AI look when it predicts yield?
Near harvest, it focused on the rice panicles. The study checked this by mapping where the AI was looking.
01

Why predict rice yield with drones and AI?

Rice is the world's most consumed staple food crop. Knowing how much rice a field will produce, its yield, matters for farmers' decisions and for national planning.

Drones are now widely used to estimate yield. They can photograph fields in fine detail whenever needed, which is especially useful in Japan, where fields are small and the weather is often cloudy or rainy.

For predicting yield from images, a widely used AI is the convolutional neural network (CNN), a type of deep learning that finds useful image features on its own.

Rice changes quickly as it ripens

Does rice really change that much before harvest?
It does. The panicles change color and the leaves start to wither. The heavy panicles droop and spread out past the edge of the plants.

After heading, when the panicles (the grain-bearing heads) emerge, the grains gradually fill. This period is called ripening. Earlier drone and CNN studies found that images from mid-ripening onward give more accurate and stable predictions.

Yet the plants keep changing as harvest approaches. The grains fill and the panicles turn golden, while the leaves begin to wither. The heavier panicles droop into the gaps between plants and beyond (Figure 1).

Side view of rice plants at mid-ripening and at maturity. At maturity the panicles are golden and droop past the dashed lines that mark the edges of the harvested patch.
Figure 1 How rice changes from mid-ripening to harvest. The panicles turn golden, grow heavy, and droop past the edge of the plant. The dashed lines mark the edges of the harvested patch.

So the most useful light, and how much area each image should include, may depend on when the images are taken. These input image conditions had not been studied in detail.

Another challenge is that it is hard to see why an AI gives a particular answer. Methods that reveal the reasons behind an answer in a form people can understand are called explainable AI (XAI). In agricultural science, XAI had almost never been used with CNNs that predict numbers such as yield.

We therefore compared input image conditions and used XAI to look for the reasons behind the differences. We expected the AI to rely on the whole plant at mid-ripening and on the panicles near harvest.

02

Three years of images and yields, compared under many conditions

What conditions did the study compare?
It changed four things. The timing of the photos, the mix of light, how wide each image was cut out, and whether shifted copies were added. Then it looked for the most accurate combination.

Fields and yield data

We ran the trials from 2021 to 2023 in farmers' paddy fields in Tomioka, Futaba District, Fukushima Prefecture, Japan. In 37 fields covering about 0.9 hectares, we grew five varieties used in the area. They were Fukuhibiki, Fukunoka, Gohyakugawa, Sakurafukuhime, and Tennotsubu.

We measured yield in small patches of six hills (two rows of three), each 60 cm × 54 cm. A hill is a clump of rice plants growing together. We harvested each patch, dried the plants in a plastic greenhouse, threshed them, and divided the grain weight by the patch area.

Over three years we collected yields from 2,184 patches. They ranged from 1.3 to 13.4 metric tons per hectare, a very wide spread.

Drone images

The drone carried a multispectral camera, which records light separately for each color, or wavelength. It can also capture near-infrared (NIR) light, which our eyes cannot see. We used green, red, red edge, and NIR light. Red edge is the narrow band between red and near-infrared.

We flew at two stages, at mid-ripening (about three weeks after heading and two weeks before harvest) and at maturity (around harvest). On sunny days the drone flew 30 or 50 meters above the ground, and we stitched its photos into one map-like image with accurate positions, called an orthoimage. Each pixel covers about 2 cm of ground.

What the AI saw

We located each harvested patch in a drone photo taken right after harvest. We then cut a square out of the drone image, centered on each patch, and showed it to the AI (Figure 2).

Each square stacked three of the four kinds of light, just as an ordinary color photo stacks red, green, and blue.

Four images. Top left is a photo taken right after harvest, with a red grid over 12 patches. The other three are black-and-white images taken before harvest, with red squares 40, 60, and 80 cm wide drawn around each patch.
Figure 2 Examples of the images the AI saw. Top left is a photo taken right after harvest, and the red grid marks 12 harvested patches. The other three are multispectral images taken before harvest, shown in black and white, and the red squares show the cut-out areas.Adapted from Figures 3 and 4 of Yamaguchi et al. (2025), CC BY-NC 4.0

The AI that predicts yield

We used a CNN for the predictions. A CNN slides small filters across an image to pick up patterns, then combines them layer by layer until it outputs a single number, the yield (Figure 3).

To train it, we showed it many images paired with the yields measured in the field. Step by step, the CNN adjusts its internal values to shrink the gap between its predictions and those yields. Of the three CNN designs we tried, Xception was the most accurate, so we used it.

Diagram of a CNN. An image made of three kinds of light goes in, layers of small filters pick up patterns, and a predicted yield comes out. It is compared with the measured yield, and the network adjusts itself to shrink the gap.
Figure 3 How a CNN predicts yield from an image. Small filters scan the image for patterns, and the layers combine them. The network learns by adjusting itself again and again to shrink the gap between its predictions and the measured yields.

We checked accuracy only on patches not used for training. We split the data into groups, trained on all but one, tested on the one left out, and repeated this for each group in turn before averaging the results. This is called cross-validation.

Four conditions we compared

We varied four conditions (Figure 4). The first was timing, mid-ripening or maturity. The second was the light combination. Choosing three of the four kinds gives four options.

The third was the cut-out size. We tried squares 40, 60, and 80 cm on a side, which are smaller than, about equal to, and larger than the 60 cm × 54 cm patch. A wider square might capture more, such as drooping panicles, but it might also take in neighboring plants and confuse the prediction.

The fourth was whether to add shifted copies. We moved the cut-out square by 10 cm and gave the new image the same yield as the original (the paper calls this buffer extraction). We shifted it in 4 directions (up, down, left, and right) or 8 (adding the diagonals), making the training data 5 or 9 times larger.

Overview of the four conditions. Timing has 2 options, the light combination has 4, the cut-out size has 3 (40, 60, and 80 cm squares), and the shifted copies have 3 (none, 4, or 8).
Figure 4 The four conditions we compared. The three cut-out sizes are centered on the harvested patch of six hills (60 cm × 54 cm). Shifted copies were added only to the training data.

We compared them in two steps. First, we compared the light combinations at each stage. Then, using the best combination for each stage, we compared the cut-out sizes and the numbers of shifted copies.

Gohyakugawa lodged (fell over) across almost the whole field in 2021, so we left out its data from that year. With no mid-ripening images of it from other years, the mid-ripening comparison used the other four varieties.

Seeing where the AI looked

Can we really tell where the AI was looking?
One way is to blank out a single spot in the image and predict again. The more the prediction changes, the more the AI relied on that spot.

For the most accurate model at each stage, we used two methods to map which parts of the image the AI relied on. Such maps, which show values as colors, are called heat maps.

The first method is occlusion sensitivity. We blank out one spot (set its value to zero), predict again, and see how much the prediction changes. The bigger the change, the more the AI relied on that spot. We repeat this for every spot (Figure 5).

Four steps of occlusion sensitivity. Predict from the image, blank one spot in black and predict again, paint the spot brighter the more the prediction changes, and repeat for every spot to build a map.
Figure 5 How occlusion sensitivity works. Blank one spot, predict again, and paint that spot brighter the more the prediction changes. Repeating this for every spot gives a map of what the AI relied on.

The second is the gradient method. It calculates in one pass how much the prediction would shift if the value at each point changed slightly. We applied both methods to one image per variety at each stage.

03

The best light and image size depended on the stage

Which combination worked best?
Near harvest, it was green, near-infrared, and red edge light. Cutting the images out a little wider made the predictions even more accurate.

Combinations of light

We measured accuracy mainly with the coefficient of determination (R²). It shows how well the predictions match the measured values, and the closer it is to 1, the better.

At mid-ripening, green, red, and NIR gave the most accurate predictions (R² = 0.401). At maturity, green, NIR, and red edge did best (R² = 0.441).

At both stages, the only combination without NIR (green, red, and red edge) gave the lowest values (Figure 6). NIR light is known to track the amount of leaves and the size of the plants well, and it appears to matter for yield prediction too. Red edge was more useful in images taken at maturity.

Bar chart of R² for each combination of light. At mid-ripening, green, red, and NIR is highest at 0.401. At maturity, green, NIR, and red edge is highest at 0.441. At both stages, green, red, and red edge is lowest.
Figure 6 Prediction accuracy (R²) for each combination of light. Dark green bars show the best value at each stage. At both stages, the combination without NIR scored lowest.Drawn from values reported in Yamaguchi et al. (2025)

Cut-out size and shifted copies

At both stages, accuracy tended to rise as the square got wider and as more shifted copies were added (Figure 7).

At maturity, the widest square (80 cm) with 8 shifted copies was the most accurate. Its R² of 0.492 was the highest in the study, and the typical prediction error was about 15% of the average yield (relative root mean squared error, RRMSE = 0.148).

At mid-ripening, the best results came from 60 or 80 cm squares with 4 or 8 shifted copies, all with an R² of 0.455. The benefit of a wider square was larger at maturity.

Two tables of R² by cut-out size and number of shifted copies. The best value is 0.455 at mid-ripening. At maturity it is 0.492, for 80 cm squares with 8 copies. Values tend to rise with wider squares and more copies.
Figure 7 R² for each cut-out size and number of shifted copies. Darker cells are more accurate, and gold frames mark the best value at each stage.Drawn from values reported in Yamaguchi et al. (2025)

In the field, rice plants spread their leaves and panicles beyond the harvested patch. A slightly wider square seems to capture the plants of each patch more fully. Until now, studies have usually cut out images the same size as the harvested area. Showing that a slightly wider cut-out improves accuracy is a new contribution of this study.

Where the AI looked

So the AI looked at different places at different stages?
At mid-ripening it looked at the whole plant, and near harvest at the panicles. Panicles droop outward and respond strongly to red edge light. That fits the earlier results nicely.

The heat maps show that at mid-ripening the AI spread its attention evenly over the whole area covered by plants. At maturity, it focused strongly on the panicles, while the leaves showed a mix of spots it attended to and spots it largely ignored (Figure 8).

Both methods showed the same pattern, and it held for every variety, even though flight dates differed by a few days between varieties and years. The AI also paid attention to panicles outside the harvested patch.

Drone images of a Fukunoka patch and heat maps from two methods. At mid-ripening the attention is scattered over the whole image. At maturity it is strong inside orange ovals around panicles and weak inside white ovals around leaves.
Figure 8 Maps of where the AI looked, for the variety Fukunoka. The left column shows mid-ripening and the right column maturity. In the middle and bottom rows, brighter or redder areas drew more attention. Orange ovals mark panicles that drew strong attention, white ovals mark leaves that drew little, and the red frame shows the approximate harvested patch. The maps are smoothed for easier reading.Adapted from Figure 8 of Yamaguchi et al. (2025), CC BY-NC 4.0

The AI's focus on panicles at maturity fits the two findings above. Previous studies report that red edge light responds to panicles once they emerge and their color and shape change. A slightly wider square also takes in panicles that droop outward, which likely helped accuracy.

04

Toward more accurate and trustworthy yield predictions

Why does it help to know where the AI looks?
If the AI looks at places that make sense for how rice grows, we can trust its predictions more. It also helps us find the model's weak spots.
Summary figure of the paper in English. Three panels show the background and hypothesis, the image conditions compared, and the best conditions at each stage with the XAI results.
Figure 9 The whole study in one picture.Graphical abstract from Yamaguchi et al. (2025), CC BY-NC 4.0

This study shows that when drones and CNNs are used to predict rice yield, the input images should be matched to the growth stage. Near harvest, light that includes red edge and a square slightly wider than the harvested patch work well.

More accurate predictions could help produce better yield maps, which show where a field yields more and where it yields less. Knowing the yield in advance supports farmers' decisions.

The XAI results matter too. When an AI attends to places that make sense in terms of how rice grows, we can trust its predictions more.

How accurate is it?

Our R² values (0.347 to 0.492) are lower than those of drone and CNN studies on soybean and wheat, but similar to those of earlier rice studies. They are far higher than those obtained with vegetation indices, which are simple greenness measures calculated from reflected light. Our data span three years, five varieties, and a wide range of yields, and we consider this accuracy sufficient for comparing image conditions.

What we do not know yet

These results come from rice. We have not yet tested whether the same conditions work for crops such as maize and wheat. Their grain heads droop less and change color differently, so the best conditions may also differ.

We also imaged only two stages. Other studies report that combining images from more stages improves accuracy, but each extra flight costs time and money. Finding the right balance is a task for future work.

Next steps

Checking where an AI looks can also reveal weaknesses in the current model and point to improvements. Next, we want to build more trustworthy models that, from an agronomic point of view, look at the right parts of the plant.

So near harvest, it helps to cut the images a little wider so the panicles fit.
Exactly. Remember that this was shown in rice. Whether the same holds for maize or wheat is still unknown.

Glossary

Panicle
The grain head of a rice plant. It is the branched cluster at the top of a stem where the grains form, and it bends down as the grains fill.
Multispectral camera
A camera that records light separately for each color, or wavelength. It can also capture near-infrared light, which our eyes cannot see.
Near-infrared (NIR) and red edge
Near-infrared light lies just beyond red, is invisible to our eyes, and tracks the amount of leaves and the size of plants well. Red edge is the band between red and near-infrared.
Convolutional neural network (CNN)
A type of AI, and a kind of deep learning, that scans an image with small filters and combines the patterns it finds over many layers.
Explainable AI (XAI)
Methods that show, in a form people can understand, why an AI gave its answer. Here we used XAI to map where the AI looked in each image.
Heat map
A map that shows values as colors or brightness. In this study, brighter or redder areas are where the AI focused more.
Occlusion sensitivity and gradient method
Two ways to find where an AI looks. Occlusion sensitivity blanks out spots one at a time and watches the prediction change. The gradient method calculates how sensitive the prediction is to a small change at each point.
Mid-ripening and maturity
Ripening is the period after heading when the grains fill. In this study, mid-ripening means about three weeks after heading, and maturity means around harvest.
Coefficient of determination (R²)
A number that shows how well predictions match measurements. The closer it is to 1, the better. A value of 0 means the predictions are no better than always guessing the average.
Cross-validation
A way to test a model on data it has not learned from. Each group of data takes a turn as the test set, and the results are averaged.

About the paper

Tomoaki Yamaguchi, Taiga Takamura, Takashi S.T. Tanaka, Taiichiro Ookawa & Keisuke Katsura (2025). A study on optimal input images for rice yield prediction models using CNN with UAV imagery and its reasoning using explainable AI. European Journal of Agronomy, 164, 127512. https://doi.org/10.1016/j.eja.2025.127512

The paper is open access, so anyone can read it free of charge. It is published under the CC BY-NC 4.0 license, which allows non-commercial reuse with credit. Some figures on this page were made from the paper's figures under this license.