A new computer algorithm is helping scientists record audio using a potted plant

Researchers at MIT have come up with a way to reproduce audio signals by examining video footage of the tiny vibrations made by ordinary objects.

“When sound hits an object, it causes the object to vibrate,” explained Abe Davis to MIT News. “The motion of this vibration creates a very subtle visual signal that’s usually invisible to the naked eye. People didn’t realize that this information was there.”

Davis is a doctoral student in electrical engineering and computer science and the first author on the study.

Labs at MIT have looked at amplifying and visualizing small motions in video before, he says. Previous work by his co-authors revealed that, in certain cases, the motions being amplified were caused by sound.

In the new study, Davis says, they started to ask, “How much audio information can we actually extract from these motions?”

So far they’ve had success recovering sound from potted plants, bags of various sorts and a glass of water, among other things. The objects that work best are “light sort of shell-like objects, like a bag or a piece of foil,” Davis says.

In one particularly convincing experiment, Davis and his team were able to reconstruct intelligible speech from the vibrations of a potato-chip bag photographed from 15 feet away through soundproof glass.

This is not going to work with just any arbitrary video, Davis admits. In most cases, a high-speed camera is needed. But they did have surprising success, Davis says, using cameras that record at just 2,000 to 6,000 frames per second, far less than the best available cameras, which can exceed speeds of 100,000 frames per second.

Recovering sounds from objects, Davis says, provides information about the sound that’s going on around the object, but it also provides information about the object itself, because different objects respond to sound in different ways. The researchers have begun trying to determine material and structural properties of objects from their visible response to short bursts of sound, according to MIT News.

Should we be concerned that this technology is being used — or might be used — for surveillance purposes?

“I wouldn't be,” Davis says. ‘It's easy to hear about this kind of technology and think that it's going to work in an arbitrary situation. … It's amazing that the signal is there and that this can actually work, but there are limitations. … So, you don't have to worry about the government suddenly using this on anyone.”

This story first aired on PRI's Science Friday.