Scientists Reconstruct Brainsâ€™ Visions Into Digital Video In Historic Experiment

Makaze · Sep 23, 2011

UC Berkeley scientists have developed a system to capture visual activity in human brains and reconstruct it as digital video clips. Eventually, this process will allow you to record and reconstruct your own dreams on a computer screen.

I just can't believe this is happening for real, but according to Professor Jack Gallant—UC Berkeley neuroscientist and coauthor of the research published today in the journal Current Biology—"this is a major leap toward reconstructing internal imagery. We are opening a window into the movies in our minds."

Indeed, it's mindblowing. I'm simultaneously excited and terrified. This is how it works:

They used three different subjects for the experiments—incidentally, they were part of the research team because it requires being inside a functional Magnetic Resonance Imaging system for hours at a time. The subjects were exposed to two different groups of Hollywood movie trailers as the fMRI system recorded the brain's blood flow through their brains' visual cortex.

The readings were fed into a computer program in which they were divided into three-dimensional pixels units called voxels (volumetric pixels). This process effectively decodes the brain signals generated by moving pictures, connecting the shape and motion information from the movies to specific brain actions. As the sessions progressed, the computer learned more and more about how the visual activity presented on the screen corresponded to the brain activity.

An 18-million-second picture palette

After recording this information, another group of clips was used to reconstruct the videos shown to the subjects. The computer analyzed 18 million seconds of random YouTube video, building a database of potential brain activity for each clip. From all these videos, the software picked the one hundred clips that caused a brain activity more similar to the ones the subject watched, combining them into one final movie. Although the resulting video is low resolution and blurry, it clearly matched the actual clips watched by the subjects.

Think about those 18 million seconds of random videos as a painter's color palette. A painter sees a red rose in real life and tries to reproduce the color using the different kinds of reds available in his palette, combining them to match what he's seeing. The software is the painter and the 18 million seconds of random video is its color palette. It analyzes how the brain reacts to certain stimuli, compares it to the brain reactions to the 18-million-second palette, and picks what more closely matches those brain reactions. Then it combines the clips into a new one that duplicates what the subject was seeing. Notice that the 18 million seconds of motion video are not what the subject is seeing. They are random bits used just to compose the brain image.

Given a big enough database of video material and enough computing power, the system would be able to re-create any images in your brain.

Scientists Reconstruct Brains' Visions Into Digital Video In Historic Experiment In this other video you can see how this process worked in the three experimental targets. On the top left square you can see the movie the subjects were watching while they were in the fMRI machine. Right below you can see the movie "extracted" from their brain activity. It shows that this technique gives consistent results independent of what's being watched—or who's watching. The three lines of clips next to the left column show the random movies that the computer program used to reconstruct the visual information.

Right now, the resulting quality is not good, but the potential is enormous. Lead research author—and one of the lab test bunnies—Shinji Nishimoto thinks this is the first step to tap directly into what our brain sees and imagines:

Our natural visual experience is like watching a movie. In order for this technology to have wide applicability, we must understand how the brain processes these dynamic visual experiences.
Click to expand...

Click to expand...

Watch/read more.

The implications of something like this are the things that dreams are made of, literally. The potential that this kind of development for research alone is enough to get me excited. I look forward to more developments upon this groundbreaking premise.

Amaury · Sep 23, 2011

That sounds interesting, and I hope it succeeds.

I have to admit, I've always wanted to watch my dreams, even the weird ones and nightmares.

Technic☆Kitty · Sep 24, 2011

This could turn into something way bigger than just "Dream TV" imagine being able to make an image in your mind (while conscious) and capture it. With enough time skill and practice you could even use this to create videos. The brain is the ultimate processor, graphics card, etc. so the detail you could make . . .

Yes it is bad to say but the first thought that came into my mind was this is going to be a media tool. Of course you would have your Dreality (Dream Reality) TV shows that come out that people go head over heels for. There isn't much else to say except I think this could be a big event in media/human history.

Llave · Sep 24, 2011

Hmm, intriguing would be appropriate for a lack of a better word. This is simply stunning, such a breakthrough is definitely a barrier breaker in the way we perceive the world... literally.

I took the liberty of watching the video clip from the link, and i must say it was quite an interesting thing to see. I truly wonder if that is how we truly recollect things in our mind... it is like a "fog of void" that emits itself here and there... The faces are quite vague like the article says, and it seemed to be multiple "images" at once on the result of the mind. I suppose that is true, as we can think of many things in a few mere nanoseconds.

The thing i am most curious is; as this progresses with more accurate depictions of the mind's recollection, I would be curious to see if there are any significant differences between the processing and speed of many different images between a male and female.

NemesisPrime · Sep 24, 2011

Mindblowing! If this becomes mainstream I could make all my videos with ease. Because It's never been a problem making them it's been how to recreate them using available technology. Also, this should make trying to figure out how people tick alot easier.

But I can't help but feel we're getting closer to places where man should not tread.

Llave · Sep 24, 2011

NemesisPrime said: ↑

Mindblowing! If this becomes mainstream I could make all my videos with ease. Because It's never been a problem making them it's been how to recreate them using available technology. Also, this should make trying to figure out how people tick alot easier.

But I can't help but feel we're getting closer to places where man should not tread.
Click to expand...

Indeed, it is quite an interesting thing, but if it gets to a point of perfection, it could be both effective and destructive. It could be a devise that would replace the lie detector, as a suggestion of something usually spawns uncontrollable thoughts about the said subject.

Noroz · Sep 24, 2011

This is definitely a step forward when it comes to understanding and interpreting the brain, but this won't work on something abstract as dreams, I believe, because of:

Given a big enough database of video material and enough computing power, the system would be able to re-create any images in your brain
Click to expand...

When I first read the title, I was thinking that it might be able to recreate memories, which would be an extremely powerful tool against alzheimers and forgetfulness in general. However, if there is a necessity for an already existing database of images and information, the images which we would be able to construct are limited purely to the extent of information we already have. In other words, we can't reconstruct dreams accurately, because dreams are often abstract.

So though I was a bit disappointed about the extent of the possibilites thus far, it's still, as I mentioned, quite the step forward.

Patman · Sep 24, 2011

The program could use the collected data to create pictures instead of searching its closest match in a you-tube database. I suppose they didn' t want to bother writing such a complex algorithm without first checking out if their data processing method is reliable.

Makaze · Sep 24, 2011

Patman said: ↑

The program could use the collected data to create pictures instead of searching its closest match in a you-tube database. I suppose they didn' t want to bother writing such a complex algorithm without first checking out if their data processing method is reliable.
Click to expand...

It would essentially become a kernel for the brain, which would be complex indeed. Waiting is all we can do about it, I am afraid.

Log in or Sign up

Scientists Reconstruct Brainsâ€™ Visions Into Digital Video In Historic Experiment

Makaze Some kind of mercenary

Amaury Chaser

Premium

Technic☆Kitty Hmm

Premium

Llave Superless Moderator

Moderator

NemesisPrime Hollow Bastion Committee

Llave Superless Moderator

Moderator

Noroz I Wish Happiness Always Be With You

Premium

Patman Bof

Premium

Makaze Some kind of mercenary