My earliest memory is seeing snow for the first time when I was four. I remember rolling a snowball down a little hill until it got big enough to be the snowman’s head. I remember the twigs my sister stuck in for arms and my mom sticking the carrot in for the nose.

Looking back, it’s unclear how much of that memory is real and how much of it is me telling a story about the photo we took that day. When I think about the memories my daughter will create in the future, I wonder what version of truth she will remember. Would she choose to remember an idealized version of the past. Or would the memories she create come from vividly capturing the experience of being there?

Idealized memories

Idealized memories remember the past as how we would have liked it to have been. If we wanted every photo to show us at our best, what would we change? To understand what we can change in a photo, lets break down what a photo describes,

  • Subject, who is in it and what are they doing?
  • Scene, where are they?

Changing the subject

Todays apps edit photos to remove blemishes and touch up less flattering parts of our appearance. But these changes amount to little more than pixel hacking. What if we could really change the contents of the photos themselves? If we didn’t like the angle we were facing, what if we could reposition ourself to face sideways? Instead of hiding a double chin, what if we could tilt our head up in the photo after it was taken? Eyes closed? Tap to open them.

Think of it as a scene in a CGI movie where every characters position and expression can be controlled. For our cameras to be able to capture that, they will need to capture reality not just as pixels but rather as 3D models of the world.

Today, our cameras capture enough details that we can use these photos to render 3D models of places. There is a rising community of photogrammetry enthusiasts, who have been using these capabilities to create VR experiences for real world places like the one for the Cologne Cathedral Clock.

If we could swap our photo with a detailed enough 3D model of ourselves, we could create a model of ourselves to have the perfect angle and expression every time. This is something that we can do today with the cameras in our phones.

Photos of others if a photo become a snapshot of an editable 3D model, then defining who is in it is no longer tied to the moment the photo was taken. This means that you could both add people after the photo was taken, but equally interesting is that you or they could remove themselves. We will need to create a rich new set of permissions for photos that enable me to allow whether my 3D avatar can be inserted or modified.

Changing the scene

Editing photos has always been about preventing reality from getting in the way. The perfect memory often blurs away the crowds, the bad weather, the obstructions.

In the idealized photo, crowds of people getting in the way of a perfect shot would just be edited out. Allowing you to create a special moment with just the people you care about and removing the rest.

Weather would be a knob that you set to recolor the photo from cloudy to sunny. Using cameras smart enough to do it automatically.

Instead of queuing up to take the same shot as everyone else, you could select the current location and it automatically serves as the background for your photo.

You could do all of this today with enough time and patience. What’s different is that instead of a editing the photo afterwards, cameras will be smart enough to do it automatically.

Immersive memories

Immersive memories capture moments in their entirety so that they can be experienced again. But what makes a memory immersive? While not comprehensive, I think there are two parts to what we remember about a moment,

  • Sensations, what we saw and heard
  • Emotions, the feelings we experienced

Visualizing the moment In Harry Potter, Rowling imagines a Pensieve that enables someone to view a memory from a third person perspective. Imagine if in the future we could summon a drone that followed us around to record every key moment in our lives.

Alternatively with the number of cameras embedded around us, maybe we could stitch together a moment from the photos taken by the people around us. Earlier this year, Facebook shared an immersive photo, recreating a room from photos that had been taken of the event. Imagine if the photos that powered this experience could come from every camera that was present at that time and place.

Feeling the moment How would we share what a moment felt like? Today we use stickers on photos to communicate a sentiment, but imagine capturing a photo that included our heart rate from our Smart Watches.

Capturing that information is truly effective only if you can recreate that sensation. Imagine viewing photos and having your watch give haptic feedback to match the heart rate of the person taking the shot.

Authenticity of our future memories

When creating your own reality becomes trivial, it raises the question of how do you know what to trust. #nofilter emerged organically to call out unedited images. But what if every photo was watermarked with some sort of a checksum to show that this was truly what the sensor captured.

What do you think?