The other day, some of our researchers based in Paris wrote an article for the Rakuten Technology Blog about how they approach the user experience of storytelling in 360 video. By working with a photographer and movie director they have been working on how to translate traditional storytelling mechanisms into a 3D world, with some very fascinating insights.

We are republishing the post here so that our readers don’t miss it, but for more posts about technology coming out of Rakuten (not only Rakuten Institute of Technology), make sure to keep an eye on the Rakuten Technology Blog as well.

The original post can be found here:

Rakuten Technology Blog – User Experience and Storytelling Innovations with 360 Video

User Experience and Storytelling Innovations with 360 Video

JANUARY 10, 2019, KEN PREPIN, LAURENT ACH

360 video cameras allow to create useful content for virtual visits where users feel immersed in remote surroundings and can look in any direction. It will be useful for Rakuten businesses where the quality of service depends on the characteristics of real-world places like beauty salons, travel destinations, and the technology will be helpful for explaining about Rakuten ecosystem or work environment based on real-life situations. Some experiments are ongoing in this domain with Rakuten Beauty and with Rakuten France HR department. The equipment for recording images and sounds and reproduce them in virtual reality headsets is now available at a reasonable cost (we can for instance insert a mobile phone in a cardboard headset that costs 15€). But significant differences between the user experience in 360-degree and traditional video brings several artistic and technical challenges:

1. The user becomes a character. Being at the center of the scene and able to look anywhere around, the user becomes part of the story. When this fact is ignored, it can cause an awkward feeling like if the user was a kind of ghost.
2. The field of view is not limited. When an action happens somewhere around the user, it may remain unseen. We need to attract user’s attention to some directions at particular times for him not to miss important parts of the story.
3. Motion sickness is never far away. Moving the camera creates a discrepancy between visual information and other perceptual contents, and abruptly changing camera location introduces visual discontinuity. Well known techniques in film editing have to be reinvented or adapted to 360-degree movies, taking into account this constraint.

Rakuten Institute of Technology in Paris has worked with Jérôme Plon, a photographer and movie director to transpose some traditional cinema techniques to 360 video and our approach was focused on three mechanisms: user’s attention, identification, and sequence transitions.

Attention

With traditional movies, people’s attention may be distracted from the screen, when they look at something else, like their watch! With 360 video, the audience is captive inside the virtual environment and the goal is to grab viewers’ attention at particular moments in particular directions, while allowing his gaze to wander anywhere the rest of the time.

Identification

When watching a fiction movie, we may identify with some characters and share emotions with them. When immersed in 360 video, we can directly feel emotions and sentiments and the identification mechanism exists in the form of the connection between the actual viewer and a virtual character who stands at the camera location. The question is whether this virtual character is an invisible observer or someone interacting with other characters of a story. From the storytelling viewpoint, the viewer may be considered as an observer or as an actor and it is possible to alternate between the two options to make the scenario more interesting.

Transitions

Our choice is to use static shots only, to avoid motion sickness, but we need to change the location of the camera from one scene to another and it risks to create a break in the visual continuity and a break in “presence”, this feeling of being actually connected to the visualized world. We need to build a structure for the user to fill the gaps and understand spatial and temporal ellipses. In traditional movies, the illusion of visual and narrative continuity is excellent, thanks to decades of film editing history, with several technical and theoretical breakthroughs. From Griffith to Eisenstein, motion picture pioneers have strived to create credible and continuous imaginary worlds with a variety of viewpoints while removing everything unnecessary or unrelated to the plotline. We now discover how to create similar imaginary continuity for stories in 360 video. We tried to transpose existing techniques for guiding attention, dealing with identification mechanisms, and managing smooth transitions.

We extend the classical principles of film editing, like continuity editing and match cut, to the case of 360 video, by considering half-spheres from two different shots that we stitch together. This allows to move from one area to another by attracting user’s gaze to the next scene while diverting his attention from the discontinuity in time or space at the moment the change occurs. We will call the initial sphere S1 and the new sphere S2, and S1 will be transformed into S2 in 3 steps: full S1, half S1-S2, full S2. We will list techniques to prevent the user from noticing the discontinuity between S1 and S2.

Figure 1: Principle of half-spheres

1. Causality

It is accepted in traditional movies that when an event happens in a shot, the next shot shows the consequence of that event. Applied to 360 video, the user can for example be in a sphere S1 where a character C1 is giving a phone call. The user then hears behind his back a character C2 answer the call, he turns around and sees character C2, who stands in another area displayed on half another sphere S2. As the user usually quickly moves his gaze from one half-sphere to the other, he does not notice the discontinuity, and accepts the new scene as a consequence of the previous one. Once his gaze points to the new half-sphere, we complete the display of the whole sphere S2 and as a result the user has moved to a new area without feeling any discontinuity.

        2. Eyeline

Eyeline matching is a very common film editing technique and Hitchcock’s movie Rear Window for instance, makes an extensive use of it. The principle is that a character in one shot is precisely looking at something and we consider the next shot as showing the thing he is looking at. Applied to 360 video, we can use this technique in two ways:

• Switching the halfsphere S2 to the new shot before the user turns his gaze in the direction the character is looking at. After the user is looking at half-sphere S2, we switch the other half-sphere from S1 to S2 and have the complete sphere S2 with the character. The user is following the action in the same space-time surrounding the character, like embedded in his imaginary world.

Figure 2: Eyeline transition #1

• Using the gaze of the character to make the user turn his head to the opposite halfsphere S1, and then changing something in the half-sphere S1 that is now in his back. We can for instance change the lighting or the costume of the character or make a new character appear. We create a narrative ellipsis in the sphere S1, which appears visually continuous only to the user. The user is therefore more like a ghost in his own space-time, which is not the character’s space-time anymore.

Figure 3: Eyeline transition #2

        3. Movement

A character C1 starts to move in one shot and continues in another shot in another area. For 360 video, we need to shoot the same movement in two areas and stich the halves of the corresponding spheres. At the moment we change half-sphere S2, we keep user’s attention in the sphere S1, for instance by having another character C2 talking to him, and after the character C1 has moved, we attract his attention to half-sphere S2, by the character C1 calling him. After his focus is on half-sphere S2, we complete the display of the whole sphere S2. By following the movement of character C1, the user has smoothly moved from one area to another.

Figure 4: Movement transition

        4. Sounds

Sound spatialization plays an important role in the techniques we described. We use ambisonic microphones and sound spatialization software to give the impression that events and speeches come from precise directions. For smooth transition, when sound continuity is ensured and users focus on audio signal, it allows visual changes we don’t want them to notice, to appear as background information.

Conclusion

The transitions we propose use half-spheres with a strategy of distracting the user’s attention away from the discontinuity of images introduced by stitching shots together. By encapsulating a succession of half-spheres, we progressively build an immersive imaginary space, which seems continuous to the user, evolving according to the narration. When we observe users watching the movie through a headset, we see they are attracted by the sounds and follow the eyelines, turning around their head to follow the story. According to questionnaires they answered, they find the movie dynamic and very immersive. Thanks to techniques transposed from traditional movies, film editing can also become a language in 360 video.

Jérôme Plon, Photographer 
Ken Prepin, Laurent Ach, Rakuten Institute of Technology Paris