A newly formed collaboration between the Immersive Arts Space and Pro Helvetia South America was launched in 2023. Within this framework, artists form countries in South America get the chance to do a residency in Switzerland. During that time they have the opportunity to come to ZHdK and work and develop their projects within the Immersive Arts Space.
Joaquina Salgado is a media artist and VJ from Argentina and is currently in Switzerland doing her residency. Her multimedia studies at the Universidad de Artes de La Plata led her to present her work in diverse exhibition contexts such as digital art festivals, night clubs, physical galleries and the metaverse. She has exhibited at festivals such as Mutek and Mirage, and collaborated on projects with Marshmallow Laser Feast (MLF). She is also part of Amplify DAI, an international network of women artists and curators working in the digital arts.
Her current project is called DERIVA. It is an audiovisual performance centered on the playing of cellist Violeta Garcia. With the help of motion capture, the music as well as the movements are transferred into a virtual world, which gives the audience new insights. The performance took place on April 28th 2023 in the Immersive Arts Space.
Shifting Realities was an overarching interdisciplinary research project, that started in Spring 2021. The goal is to explore the interplay of real and virtual experiences as well as the interaction of VR-users and by-standing spectators in shared experiences. Combining these different perspectives creates scope for new practices and forms of expression in Extended Reality (XR).
The research is structured in four binary focus areas (or pair of opposites): • Virtual vs. augmented environments • Virtual vs. real spaces, objects or humans • Binaural vs. spatial audio • Storytelling vs. game mechanics
The research focuses at the interfaces defined by the pairs of opposites above. Through them, differences in perspective, shifts in perception, and the extent of interactivity can be shaped and controlled. The primary goal is the development of prototypical unique solutions in limited subareas.
In a first phase, basic concepts of reality shifts were explored and developed. They all depict situations, in which the users find themselves in transitional states between virtuality and reality. The aesthetics to create such transitional settings are defined by rough 3D scans, sketchy point clouds created in real time by depth cameras or the Oculus passthrough feature. The resulting imperfection of such low-end processes induces an awareness of our fluctuating and at times uncertain perception while experiencing mixed reality content.
In a second phase, three prototypical experiences were developed. Two are device-based with mixed reality components displayed in VR goggles: • PASSING THROUGH THE REAL and • FRAGMENTS OF REALITY.
One is installation-based with spatial augmented reality contingent on projection mapping: • FRAGMENTS OF A CONVERSATION (watch video).
For the first time, the three experiences were presented in preliminary versions at the LabInsights in June 2022.
Team: Researchers: Florian Bruggisser, Martin Fröhlich, Valentin Huber, Norbert Kottmann, Eric Larrieux, Chris Elvis Leisi, Oliver Sahli, Stella Speziali. Production manager: Kristina Jungic; Chief Technician: Sébastien Schiesser. Principal Investigator: Prof. Christian Iseli
Power and Presence explores meaningful and empowering interaction in virtual reality and how it can be implemented as game mechanics without breaking the feeling of being in another world. A critical analysis of game design theories and how they need to be applied to VR is demonstrated through a game that uses phonetic interaction.
Oliver Sahli, research associate at the Immersive Arts Space and graduation student in Master in Game Design, showcased his project Power and Presend within the diploma exhibition of the ZHdK in June 2021.
Diploma project, MA Game Design, by Chris Elivis Leisi
In today’s VR games, the body often serves as the controller. However, when the player enters the virtual world, the connection to the physical environment is often lost. This master’s thesis deals with immersion mechanics in VR and reveals the potentials that arise when one’s own home can be integrated into the virtual world as a play area.
Chris Elvis Leisi, research associate in the Immersive Arts Space and graduate student in Master in Game Design, exhibited his graduation project Virtual Real World within the diploma exhibition of the ZHdK in June 2021.
The Umbrella Project is a research project that explores the use of 3D audio and projection mapping to achieve a sense of immersion without isolating participants from the real world, essentially, enabling an imaginary fantasy world to come to life in our own. We employ multiple levels of 3D audio and projection mapping (both directly within and on the umbrella, as well as throughout the room itself) in order to transport the participant into this virtual world.
The end goal of the project was to create a series of navigable compositions in the form of exploratory sonic worlds, as well as intereactive experiences where the participants’ behaviours (relative to each other and the world) shape the sonic and visual environment. Furthermore, we were investigating sonic and visual paradigms where the umbrellas can function both as objects existing in and can interact with the virtual world, as well as being windows onto these other worlds.
Naturally, these environments are best experienced from directly underneath the umbrella, where one can best appreciate the various levels of MR.
KUSUNDA is a virtual reality documentary experience about what it means to lose a language and what it takes to keep one alive.
Narrated by two of its co-creators — 86-year-old Kusunda shaman Lil Bahadur and his 15-year-old granddaughter Hema — the experience contrasts two generations set apart by their lifestyles and brought together by the struggle for their indigenous identity. In the VR experience, you join Hema as she reminds her grandfather of his forgotten mother tongue. You navigate by speaking words in the endangered Kusunda language and join an audible fight against its extinction.
With the help of the motion capture facilities at the Immersive Arts Space, actors were recorded recreating these sequences from Lil Bahadur’s past. This not only simplifies and speeds up the process of character animations but also offers unique possibilities for the documentary storytelling form.
KUSUNDA Speak to Awaken has earned international recognition and was awarded at the Tribeca Film Festival 2021.
Crew at ZHdK: Cast: Yan Balistoy, Offir Limacher, Johannes Voges, Ferhat Türkoğlu, Liliana Heimberg, Corinne Soland, Oliver Sahli, Kristina Jungic Mocap coaching and production: Corinne Soland Mocap recording: Tobias Baumann IASpace producer: Kristina Jungic Further ZHdK support: Chantal Haunreiter, Martin Fröhlich, Stella Spezialli
ZHdK Residency The residency at the Zurich University of the Arts has been made possible by the Ernst Göhner Foundation, Switzerland, as well as by additional support of the ZHdK film program and by the Immersive Arts Space.
Further crew members: Co-creators: Gyani Maiya Kusunda, Hema Kusunda, Lil Bahadur Kusunda; Storytelling/Production coordination: Felix Gaedtke, Gayatri Parameswaran; Executive Producer: Rene Pinnell; Associate Producer: Mia von Kolpakow; Co-Producers: Emma Creed, Aliki Tsakoumi, Sönke Kirchhof, Philipp Wenning, Kuan-Yuan Lai; Lead Developer: Tobias Wehrum; Art Director, 3D designer & animator: Moritz Mayerhofer (and team); Volumetric Video post-processing: INVR.SPACE; Photogrammetry post processing: realities.io; AI speech recognition: Valerio Velardo; Sound designer: Mads Michelsen; Project website: Tom Lutherburrow | Nepal Production team: Direction/Production: Felix Gaedtke, Gayatri Parameswaran; Line Producer Nepal: Deepak Tolange, Sandeep Bhaju; Volumetric Video & Photogrammetry: Felix Gaedtke, Gayatri Parameswaran; DoP and Drone pilot: Aditya Thakuri; Sound recordist: Mia von Kolpakow; Kusunda linguistics researcher: Uday Raj Aale; Driver: Dharmendra Shakya
This artistic research project focuses on the interplay of presence and absence of real dancers and virtual characters. It is based on augmented projection mapping, motion capture and movable stage elements. Dancers disappear behind stage elements while their avatars are projected on these elements. When the dancers step from behind the elements, their virtual characters vanish immediately. This principle is varied, when the body of a dancer is only partially hidden by an element. In this case, the spectators witness a figure who is half avatar and half human being.
Further variations of presence and absence are made possible with stage elements that allow dancers to walk or jump through the walls made out of elastic ribbons. Thus, the avatars appear immediately when the dancers are covered by the ribbons and vice versa.
The technical setup includes a motion capture system with a large tracking space that also covers non-visible areas of the stage. Furthermore, a projection mapping system with multiple projectors and a performative 3D mapping software is needed as well as game engine that guarantees real time performance of up to eight virtual characters.
#Keywords: Motion capture, projection mapping, virtual characters, real time rendering, game engine, modular stage elements, dance performance.
The project ‘Presence and Absence’ was connected to the workshop and performance Dancing Digital.
At the shoot of the short film LUX (Director: Wendy Pillonel, DoP: Ramón Königshausen) at the ZHdK studio. (Photo by Christian Iseli, ZHdK 2019)
The research project “Virtually Real – Aesthetics and the Perception of Virtual Spaces in Film” is the first IASpace project to be financed by third-party funds. The project concerns itself with the increasing virtualisation of film production, focussing on the transition to 3D recording of real environments and objects using laser scanning and photogrammetry. For a comparative study, short feature films will be recorded both virtually (in previously scanned 3D spaces) and conventionally (in the corresponding real spaces). The film variants are used to investigate the effects on perception and changes in work processes.
Studio shoot in front of green screen with real-time display of the scanned location (compare top center with the monitor on the bottom right). Project: LUX by Wendy Pillonel. (Photo by Christian Iseli, ZHdK 2019)
Project lead and principal investigator: Prof. Christian Iseli. Co-applicant: Dr. David Weibel (Institute of Psychology, University of Bern). Researchers: Tom Gerber, Wendy Pillonel, Miriam Loertscher, Martin Fröhlich, Valentin Huber. Project partners: Michael Schaerer, Max Rheiner. Industry partners: InstaLOD GmbH, Stuttgart; Leica Geosystems, Heerbrugg. Funded by the Swiss National Science Foundation (SNSF) / Call: Digital Lives
The aim of the project is to develop a versatile co-location multiuser system, with which several users can dive into virtual worlds together in the same room. The system can be used for many different areas. As a multiplayer game in which the users feel even closer to their fellow players through increased physical presence. As a virtual film set to try out camera movements or interactions of actors without much effort. Or as an interactive experience that transports the user into a new dimension of immersion.
An iPad also serves as a window into the virtual world. The user can use this virtual camera to move through the room and in some cases even interact with the virtual world. The virtual world can also be placed on an interactive screen. This gives outsiders the opportunity to change the entire scene.
The co-location multiuser system can be used wherever the headsets can connect to a network thanks to the Mobile VR-Headsets. The system is developed by IA-Space staff members Chris Elvis Leisi and Oliver Sahli.
Home in the Distance is based on a short story, written by director and VFX artist Andreas Dahn when he was about 17 years old. In his residency at the Immersive Arts Space he made an animated short film and an interactive VR experience out of it, and applied unique methods of Virtual Production. In contrast to the traditional animation and VFX workflow, which is heavily based on storyboarding and previsualization, the conscious decision was made to leave out these steps, in order to find out if new technologies make it possible to create a more emotional and spontaneous film experience.
Andreas Dahn first constructed the locations, his character and all other digital assets in 3D. With the help of Motion Capture technology he animated the complete story. And finally he shot the 2D film with a virtual camera that he operated in the 3D space with VR goggles. So, he literally shot the film in the virtual space. After the 2D film was finished, Andreas made an interactive VR experience based on the same 3D assets and the same story.
Andreas Dahn (writer and director) on the left, with Pascal Holzer (actor) on the right, during MoCap recording.
The character was sculpted by hand with dough and photogrammetry was used to create the virtual 3D model. Rigging was done with Human IK in Autodesk Maya. Students from the 3D Akademie Stuttgart helped to model the set and props. In addition, an actor`s performance was motion-captured with OptiTrack in the IASpace at the Zurich University of the Arts (ZHdK).
The camera work was done with a VR tool, developed by Mirko Lempert (Stockholm University ot the Arts) and Simon Alexandersson (Royal Institute of Technology, Stockholm) and rendered in realtime in Unity.
Shooting the 2D movie with a virtual camera in the virtual 3D space.
Crew at ZHdK Cast: Pascal Holzer Mocap coaching: Corinne Soland Mocap recording: Norbert Kottmann Further ZHdK support: Valentin Huber, Robin Disch, Marco Quandt, Lucien Sadkowski, Andreas Birkle, Chantal Haunreiter, Claudia Hürlimann, Thomas Gerber, Stefan Jäger, Martin Fröhlich
ZHdK Residency The residency at the Zurich University of the Arts has been made possible by the Ernst Göhner Foundation, Switzerland, as well as by additional support of the ZHdK film program (led by Sabine Boss) and by the Immersive Arts Space (led by Prof. Christian Iseli).
Crew at 3D Akademie Stuttgart Props & set modeling & texturing: Gerrit Gaietto, Katharina Rodak, Kimberly Niesner Pierre Urbanek (Head of 3D Akademie)
Further crew members Unity VR support: Mirko Lempert, Simon Alexanderson Junior producer: Jana Günther Assistent director: Aimée Torre Brons Sound design: Luis Schöffend, Marc Fragstein Title design: Timo Kreitz Screenplay translation: Karen Ma Special thanks: David Maas, Renate Schirrow, Ella Steiner, Felix Bucella, Alireza Sibaei, Astrid Weitzel
