02 — Methods / Tutorial

Reconstructing and Preserving a Plant's 3D Form
from a Smartphone Video

Anyone can do it, with no special equipment

Tomoaki Yamaguchi and Kota Nakajima

Filming a rice plant with a smartphone inside a tent

This article explains how to reconstruct a plant's three-dimensional form from a video shot on a smartphone, at true real-world scale, and preserve it as data. It also explains how to obtain plant height, surface area, and volume from the reconstructed 3D data.

We believe that keeping a plant's form as three-dimensional data has a value that recording numbers alone cannot offer. To measure is to summarize a plant's form into a few numbers. Put another way, it is to reduce information. If you save the three-dimensional data, none of the original information is lost. You can measure different traits later, and you can observe the plant again just as it was that day.

All it takes is a smartphone, LumaAI (a cloud service that rebuilds 3D shapes from video), and Meshmixer (a free program for working with 3D data). In case Meshmixer does not work on your computer, a separate page shows how to take the same measurements in MeshLab, another free program.

Dr. Hoo, can a phone video really turn a plant into 3D?
It can. Circle the plant twice while filming, and you get a 3D model that keeps its real size. From it, you can measure plant height, surface area, and volume.

What you need (all software is free)

  • A smartphone for filming two slow laps around the plant.
  • A LumaAI account for building a 3D model from your video automatically. Sign up at lumalabs.ai/dashboard/captures.
  • Meshmixer for editing and measuring the model. It is a free 3D program from Autodesk, and there are versions for both Windows and Mac (see below for where to get them).

Where to get Meshmixer. Autodesk has stopped both distributing and developing it, and the old site (meshmixer.com) now redirects elsewhere. The most reliable source today is the Internet Archive's copy of the final release, 3.5.474. Both files below are the installers Autodesk itself used to distribute, preserved as they were.

Windows (.exe, about 101 MB)
Mac (.pkg, about 94 MB)

A community-run site, meshmixer.org, also walks through the installation. Note that it is not an Autodesk site.

If you cannot get Meshmixer or install it, you can follow the MeshLab steps instead.

Installing on a Mac

Double-click the .pkg you downloaded and follow the prompts. Two things commonly trip people up on first launch.

"macOS cannot verify the developer." Open "System Settings → Privacy & Security," scroll down to the Meshmixer entry, and click "Open Anyway." Holding control while clicking the Meshmixer icon and choosing "Open" works too.

"Do you want to install Rosetta?" Meshmixer was built for Intel Macs, so Apple Silicon (M-series) machines need Rosetta 2 to translate it. Click "Install" and macOS handles the rest.

01

Collect the plant and transplant it into a pot

The whole workflow is done with the plant growing in a pot, so the first step is simply to get your plant ready.

When you collect the plant, dig it up gently so you do not damage it. Lifting it out together with the roots and the surrounding soil keeps the impact on later growth to a minimum.

Collecting the plant without damaging it
Lift the plant out gently, soil and all, to protect the roots.

Transplant the plant into a pot. Since the pot becomes your reference for filming and measurement, plant it so it stands upright.

Transplanting the collected plant into a pot
Transplant it into a pot so it stands upright.

* Rice illustrations by Prof. Keisuke Katsura (Kyoto University)

The pot's diameter, or the length of its side, can serve later as your real-world scale reference. Choosing a pot whose size you know makes the following steps easier.

02

Film a 360° video with your smartphone

Is there anything to watch out for while filming?
Walk around slowly at a steady pace. The trick is to keep both laps in one video, without a break.

Circle the plant twice with your smartphone while filming, and build the 3D model from that single video. How cleanly the model turns out depends a great deal on how you film. As in the clip below, circle the plant slowly and at a steady pace.

Keep the whole plant and pot in frame, and walk two slow laps.
The same filming seen from outside. Keep a steady distance and walk slowly around the plant.

Keep these points in mind when you film.

  • Film in a bright, wind-free environment. Indoors or under a tarp tent works well. Choose a spot that is bright enough, where the brightness stays steady, and where there is no wind. If the leaves move or the lighting shifts, the shape will not settle and the model breaks down.
  • Keep the whole plant and pot in frame. If you are too close, the rotation is hard to track. If you are too far away, the resolution drops. Stay at a distance where everything fits on screen.
  • Include something of a known size. Putting something whose dimensions you know in the shot, such as the pot's diameter, a cube-shaped box, or a ruler, lets you set the real scale correctly later.
  • Make the background deliberately busy. A background scattered with objects of different shapes and colors helps the software recognize how far around the plant you have moved. Avoid a plain, empty background.
  • Film at the highest resolution and frame rate you can. The higher the resolution and FPS (frames per second), the smoother and more detailed the model.
  • Hold steady and circle slowly. Hold the phone with both hands and go around at an even pace. Sudden movements cause blur.
  • Go around twice, at different heights. Make the first lap looking down at an angle, then continue straight into the second lap from the side, about level with the plant. Filming from different angles captures the whole plant, and the smooth, continuous motion makes the rotation easier to track.

Film the first and second laps in a single video, with no break between them. Once filmed, upload it to LumaAI in the next step.

03

Build a 3D model from video with LumaAI

Is it true the model can end up without its colors?
It is. If the folder name or its path contains Japanese or other non-ASCII characters, the model opens without colors. Keep the files inside the folder exactly as they are, too.

Upload your video to LumaAI Capture, and it reconstructs a full 3D model from the footage automatically.

Using LumaAI Capture

Open lumalabs.ai/dashboard/captures and upload your video from "Create." There is no limit on how many times you can use it, and you can process up to three videos at once. The first time, you will need to sign in with your email address.

You can also use the iPhone app

There is an iPhone app called Luma 3D Capture. You can upload a video you have already shot directly from the app, or build the 3D model while filming with the app's camera. This saves you from moving files to a computer, so it helps when you want to shoot and check on one device. Models are saved to the same account as the web version, so you can continue the work on your computer. The app is currently available only for iPhone, not for iPad.

The clip below shows the result. A complete 360° model rises straight out of the video.

The 3D model reconstructed from the video, solid all the way around.
What happens inside LumaAI

On its official plugin page and in its official web rendering library on GitHub, LumaAI shows that it uses two AI representations, NeRF (Neural Radiance Fields) and 3D Gaussian Splatting. NeRF trains a neural network to learn the color and density at each point in space. 3D Gaussian Splatting represents the scene as countless small, blurred ellipsoids. Because it is light to render, it has become the main approach today. The original papers for both methods, NeRF (Mildenhall et al., 2020) and 3D Gaussian Splatting (Kerbl et al., 2023), are openly available.

The details of the pipeline are not public. These methods need to know where each frame was taken from, so a step that estimates the camera positions presumably runs first. It would be similar to Structure from Motion (SfM), a technique from photogrammetry. The OBJ file you download comes from a step that the former official API called "meshing." In this step, the surface is extracted from these representations and converted into a polygon mesh. The specific algorithm has not been disclosed. It is because the data becomes a mesh that Meshmixer can measure volume and surface area.

Once the model is ready, check in the viewer that the plant is cleanly separated from the background. Rotate the view and make sure nothing unwanted has crept in.

Checking the reconstructed 3D model in the LumaAI viewer(view larger image)
Check the reconstructed model in the viewer.
Rotating the view to check separation from the background(view larger image)
Rotate the view to see how well the plant is separated.

Reconstruction does not always succeed. The background may not separate cleanly, or the plant may come out broken. In that case, trimming the video to just one of the two laps and uploading it again can help. If that still fails, re-film the video.

If some background is still included, use the bounding box (the frame around the model) to keep only the area you need.

Using the bounding box to keep only the area around the plant(view larger image)
Adjust the box so only the plant remains.

When it looks right, download the model. For the export format, choose OBJ, and pick high poly to keep as much of the shape as possible. High-poly files are large, though, and the later steps can feel slow on some computers. If that happens, try medium poly or low poly as well, and find a balance between detail and ease of use.

Downloading the model from Your Captures(view larger image)
Download the model from "Your Captures."
Choosing OBJ high poly as the export format(view larger image)
Export as OBJ (high poly).

The download comes as a ZIP archive. On Windows, right-click the file and choose "Extract All." On a Mac, double-click the file and it unzips itself. Make sure the extracted folder name and its path contain no Japanese or other non-ASCII characters. If they do, the model opens in Meshmixer without its colors. A safe choice is a folder with an alphanumeric name (English letters and numbers only) and a short path, such as C:\lumaai on Windows or ~/Documents/lumaai on a Mac.

What is inside the unzipped folder

(unzipped folder)/
├── mesh.obj   … the 3D model itself (its shape)
├── mesh.mtl   … instructions for applying the images
└── textures/ … images of the surface colors

mesh.obj holds the shape of the plant. The textures folder holds images of its surface colors and patterns. mesh.mtl tells the software how to wrap those images around mesh.obj. When you open mesh.obj in Meshmixer, it follows mesh.mtl to load the images and shows the model in color.

Keep this folder structure exactly as it is. mesh.mtl points to each image by where it sits inside the textures folder. If you rename the files or the folder, or move the textures folder somewhere else, the images can no longer be found and the model loses its color. When you move things, move the whole unzipped folder together.

Viewing the model without installing anything

If you just want to look at the model first, try Online 3D Viewer, a free viewer that runs in your web browser. Drag the ZIP file onto the page without unzipping it, and the model appears in color. You can also drop the whole unzipped folder.

The files you open are not uploaded to any server. Everything happens inside your own browser, as the official manual states. You do not need an account, and the source code is public, so anyone can check how it works. The site is also handy when you share 3D data with others, because they can view it without installing any software.

If Meshmixer gives you trouble

You now have a 3D model of your plant. From here on, you will clean it up and measure it in Meshmixer.

Meshmixer is no longer distributed or developed, so on some computers it may fail to install, refuse to start, or stop partway through. If that happens, follow the same steps in MeshLab instead. MeshLab is a free program that is still being developed, and it runs on Windows, Mac, and Linux.

04

Import the OBJ into Meshmixer and set its orientation

Launch Meshmixer, click "Import," and select mesh.obj inside the unzipped folder.

Importing the file on the Meshmixer start screen(view larger image)
Click "Import" on the start screen and choose mesh.obj.

Right after import, the model may be tilted or sitting on a slanted floor. Open "Edit → Transform" and handles appear on the model. Drag the arrows to move it and the arcs to rotate it, until the plant stands upright.

You can also adjust the position and rotation with "Edit → Align." Pick a surface or point as the reference, and the whole model snaps straight to match it. When you want to set the base flat on the floor, this can be quicker than rotating the model by hand with Transform.

Moving the view

With a mouse, Windows and Mac behave the same. Drag with the right button to rotate the view, and drag while pressing the scroll wheel to pan.

On a laptop without a mouse, hold a key while you drag. That key is Alt on Windows and option (⌥) on a Mac. Hold it and drag with the left button down to rotate the view. To pan instead, hold Shift as well (Alt + Shift, or option + shift). A two-finger swipe up or down zooms in and out on both.

Dragging with the right button held down also rotates the view. On a Mac trackpad, a right-click means either a two-finger click or a click with control held down.

These move your viewpoint rather than the model, so use them alongside Transform.

Adjusting the orientation of the imported model(view larger image)
Adjust the model's orientation.
Positioning the model so the plant stands upright(view larger image)
Position it so the plant stands upright.
05

Set the scale (real-world size)

Why does setting the scale matter so much?
A 3D model has no sense of size on its own. This step decides how accurate the volume and surface area will be. Filming something of known length, like the pot, makes it easy.

By itself, a 3D model has no unit of length. Open "Analysis → Units/Dimensions" and choose your unit (cm, for example). Then left-drag on a part whose length you know to draw a reference line, and enter its real size in the "Set Target Length" field. The whole model is scaled to match.

Changing the unit from mm to cm(view larger image)
First, change the unit from mm to cm.
Drawing a reference line on a part of known length(view larger image)
Draw a reference line on a part of known length.
Entering the real length of the reference line(view larger image)
Enter the line's real length to set the scale.

This step decides the accuracy. The reliability of your volume and surface area comes down to this calibration. It helps to place something of a clearly known length in the shot when you film, such as the pot's diameter, a cube-shaped box, or a ruler.

06

Isolate the plant

Remove parts you do not want to measure, such as the pot and the ground. Open "Edit → Plane Cut" and rotate and move the cutting plane into position. Set "Cut Type" to "Cut (Discard Half)" and "Fill Type" to "Remeshed Fill," then click "Accept" to slice everything below the plane away in one go.

With "Remeshed Fill" selected, the cut cross-section is automatically closed with a new surface. An open cross-section would make the volume come out wrong, so after cutting, look at the bottom of the model and confirm it is sealed without holes.

Removing the pot with a plane cut(view larger image)
Use a plane cut to keep only the plant.
07

Repair the 3D model

How do you fix holes and stray bits?
Open the Inspector, and the problem spots get marked. Press Auto Repair All to fix them together. Once every mark is gone, you are done.

A model built from video can be left with small holes or stray floating fragments. Open "Analysis → Inspector" and markers appear on the problem spots. Blue means a hole, pink a tiny fragment, and red an invalid shape. Set "Hole Fill Mode" to "Flat Fill," use "Small Thresh" to set the size of the fragments to remove, and click "Auto Repair All" to fix everything at once. When all the markers are gone, the model is ready to measure.

Detecting holes and stray fragments in the model(view larger image)
Detect holes and stray fragments.
Cleaning up the model with automatic repair(view larger image)
Run the automatic repair so the model is ready to measure.
08

Re-save the cleaned, scaled model

Now that the model is scaled and contains only the plant, save this version as its own file. When you export to OBJ from Meshmixer, a single save creates several files at once.

Files created by the export

(name).obj … the model itself (its shape), including your scaling, cuts, and repairs
(name).mtl … instructions for applying the texture images to the .obj
Texture images … pictures of the surface colors, split across several files

Because the export writes several files at once, create a folder for them yourself first, and save into it. That keeps them from mixing with other files, and you will always know which files belong together. Give the folder an alphanumeric name and put it somewhere like C:\lumaai\rice01 on Windows or ~/Documents/lumaai/rice01 on a Mac. If the folder name or its path contains Japanese or other non-ASCII characters, the model will open without its colors later.

Once the folder is ready, open "File → Export," choose OBJ as the format, and save into that folder under an alphanumeric name. Alongside the original download, you now keep the data exactly as it was measured.

As noted at the top of this page, keeping the 3D data means you can measure new traits later, or observe the plant again as it was. Adding the filming date and a plant ID to the filename makes it easier to find afterward.

Keep all of the saved files together in that one folder. As with the LumaAI files, the .mtl points to where the images are, so moving the .obj alone gives you a colorless model. Areas filled during repair carry no color, but this does not affect the measurements.

09

Check the plant height

Before measuring, check the model's height. Open "Analysis → Units/Dimensions" to see the model's dimensions (X, Y, Z), and look at the vertical (Y) value. If it is close to the real plant's height, the scale was set correctly. A value far from reality is a sign to go back and check.

Checking the model's height(view larger image)
If the height matches reality, the scale is correct.

Sometimes, even after the pot has been deleted, the dimension box still encloses the region where the pot used to be, and the Y value comes out larger than the real plant height. In that case, left-drag to draw a straight line on the plant itself, just as you did when setting the scale. The "Real length" field shows the actual length of that line, so you can read the height directly.

10

Read the volume and surface area

From the finished model, read off the measurements. Open "Analysis → Stability" and the "Volume" and "Surface Area" values appear. Take a moment to check whether the numbers fall in a realistic range.

Reading the volume and surface area(view larger image)
Check the measured volume and surface area.

If a value comes out far too large or too small, you may have missed something when setting the scale or repairing the model. Go back to those steps and check.

11

Split the mesh to measure parts separately

Sometimes you want to measure only a certain part, such as a stem or a leaf, rather than the whole plant. Open "Select," set "Brush Mode" to "Unwrap Brush," and adjust the brush "Size." Paint over the part you want by dragging with the left button, and the selected area turns black.

Painting the part to select with the Unwrap Brush(view larger image)
Paint over the part you want with the brush (it turns black).
Splitting off the selection with Edit → Separate(view larger image)
Split off the selection with "Edit → Separate."

After separating, open "View → Show Objects Browser" to pick each part from the list. Then repair with Inspector and measure with Stability, part by part, just as before.

Measuring volume and surface area part by part(view larger image)
Measure the volume and surface area part by part.

With just a phone video and free software, you can capture a plant's form as 3D data. Because the method is non-destructive, you can measure the same plant again and again to follow how it changes over time. As long as you film carefully and set the scale with care, it is an easy method to get started with.

Since nothing is destroyed, you can follow the same plant as it changes.
Exactly. Film carefully and set the scale with care, and anyone can start.