Category: Archive [projects]

Filmmaker in Residence Gayatri Parameswaran

KUSUNDA is a virtual reality documentary experience about what it means to lose a language and what it takes to keep one alive.

Narrated by two of its co-creators — 86-year-old Kusunda shaman Lil Bahadur and his 15-year-old granddaughter Hema — the experience contrasts two generations set apart by their lifestyles and brought together by the struggle for their indigenous identity. In the VR experience, you join Hema as she reminds her grandfather of his forgotten mother tongue. You navigate by speaking words in the endangered Kusunda language and join an audible fight against its extinction.

With the help of the motion capture facilities at the Immersive Arts Space, actors were recorded recreating these sequences from Lil Bahadur’s past. This not only simplifies and speeds up the process of character animations but also offers unique possibilities for the documentary storytelling form.

KUSUNDA Speak to Awaken has earned international recognition and was awarded at the Tribeca Film Festival 2021.

Crew at ZHdK:
Cast: Yan Balistoy, Offir Limacher, Johannes Voges, Ferhat Türkoğlu, Liliana Heimberg, Corinne Soland, Oliver Sahli, Kristina Jungic
Mocap coaching and production: Corinne Soland
Mocap recording: Tobias Baumann
IASpace producer: Kristina Jungic
Further ZHdK support: Chantal Haunreiter, Martin Fröhlich, Stella Spezialli

ZHdK Residency
The residency at the Zurich University of the Arts has been made possible by the Ernst Göhner Foundation, Switzerland, as well as by additional support of the ZHdK film program and by the Immersive Arts Space.

Further crew members:
Co-creators: Gyani Maiya Kusunda, Hema Kusunda, Lil Bahadur Kusunda; Storytelling/Production coordination: Felix Gaedtke, Gayatri Parameswaran; Executive Producer: Rene Pinnell; Associate Producer: Mia von Kolpakow; Co-Producers: Emma Creed, Aliki Tsakoumi, Sönke Kirchhof, Philipp Wenning, Kuan-Yuan Lai; Lead Developer: Tobias Wehrum; Art Director, 3D designer & animator: Moritz Mayerhofer (and team); Volumetric Video post-processing: INVR.SPACE; Photogrammetry post processing: realities.io; AI speech recognition: Valerio Velardo; Sound designer: Mads Michelsen; Project website: Tom Lutherburrow | Nepal Production team: Direction/Production: Felix Gaedtke, Gayatri Parameswaran; Line Producer Nepal: Deepak Tolange, Sandeep Bhaju; Volumetric Video & Photogrammetry: Felix Gaedtke, Gayatri Parameswaran; DoP and Drone pilot: Aditya Thakuri; Sound recordist: Mia von Kolpakow; Kusunda linguistics researcher: Uday Raj Aale; Driver: Dharmendra Shakya

---

Augmented projection mapping with virtual characters

This artistic research project focuses on the interplay of presence and absence of real dancers and virtual characters. It is based on augmented projection mapping, motion capture and movable stage elements. Dancers disappear behind stage elements while their avatars are projected on these elements. When the dancers step from behind the elements, their virtual characters vanish immediately. This principle is varied, when the body of a dancer is only partially hidden by an element. In this case, the spectators witness a figure who is half avatar and half human being.

Further variations of presence and absence are made possible with stage elements that allow dancers to walk or jump through the walls made out of elastic ribbons. Thus, the avatars appear immediately when the dancers are covered by the ribbons and vice versa.

The technical setup includes a motion capture system with a large tracking space that also covers non-visible areas of the stage. Furthermore, a projection mapping system with multiple projectors and a performative 3D mapping software is needed as well as game engine that guarantees real time performance of up to eight virtual characters.

#Keywords: Motion capture, projection mapping, virtual characters, real time rendering, game engine, modular stage elements, dance performance.

The project ‘Presence and Absence’ was connected to the workshop and performance Dancing Digital.

Crew:
Visual artist: Tobias Gremmler
Set designer: Mariana Vieira Gruenig
Augmented projection artist: Martin Fröhlich
Motion capture & Unity: Tobias Baumann, Norbert Kottmann, Chris Elvis Leisi, Oliver Sahli
MoCap coaching: Corinne Soland
Project manager: Kristina Jungic
Performers: Chantal Dubs, Aonghus Hode, Svenja Koch,Lucas del Rio Estevez, Johannes Voges
Project lead: Christian Iseli

---

Home in the Distance is based on a short story, written by director and VFX artist Andreas Dahn when he was about 17 years old. In his residency at the Immersive Arts Space he made an animated short film and an interactive VR experience out of it, and applied unique methods of Virtual Production. In contrast to the traditional animation and VFX workflow, which is heavily based on storyboarding and previsualization, the conscious decision was made to leave out these steps, in order to find out if new technologies make it possible to create a more emotional and spontaneous film experience.  

Andreas Dahn first constructed the locations, his character and all other digital assets in 3D. With the help of Motion Capture technology he animated the complete story. And finally he shot the 2D film with a virtual camera that he operated in the 3D space with VR goggles. So, he literally shot the film in the virtual space. After the 2D film was finished, Andreas made an interactive VR experience based on the same 3D assets and the same story.

The character was sculpted by hand with dough and photogrammetry was used to create the virtual 3D model. Rigging was done with Human IK in Autodesk Maya. Students from the 3D Akademie Stuttgart helped to model the set and props. In addition, an actor`s performance was motion-captured with OptiTrack in the IASpace at the Zurich University of the Arts (ZHdK).

The camera work was done with a VR tool, developed by Mirko Lempert (Stockholm University ot the Arts) and Simon Alexandersson (Royal Institute of Technology, Stockholm) and rendered in realtime in Unity.

Crew at ZHdK
Cast: Pascal Holzer
Mocap coaching: Corinne Soland
Mocap recording: Norbert Kottmann
Further ZHdK support: Valentin Huber, Robin Disch, Marco Quandt, Lucien Sadkowski, Andreas Birkle, Chantal Haunreiter, Claudia Hürlimann, Thomas Gerber, Stefan Jäger,  Martin Fröhlich

ZHdK Residency
The residency at the Zurich University of the Arts has been made possible by the Ernst Göhner Foundation, Switzerland, as well as by additional support of the ZHdK film program (led by Sabine Boss) and by the Immersive Arts Space (led by Prof. Christian Iseli).

Crew at 3D Akademie Stuttgart
Props & set modeling & texturing: Gerrit Gaietto, Katharina Rodak, Kimberly Niesner Pierre Urbanek (Head of 3D Akademie)

Further crew members
Unity VR support: Mirko Lempert, Simon Alexanderson
Junior producer: Jana Günther
Assistent director: Aimée Torre Brons
Sound design: Luis Schöffend, Marc Fragstein
Title design: Timo Kreitz
Screenplay translation: Karen Ma
Special thanks: David Maas, Renate Schirrow, Ella Steiner, Felix Bucella, Alireza Sibaei, Astrid Weitzel

---

Together with visual artist Tobias Gremmler, choreographer Nadav Zelner and set designer Mariana Vieira Gruenig, students of the Contemporary Dance program at the Zurich University of the Arts explored the connection between real dancers and virtual characters in a modular stage design.

The prototype presentation took place during the REFRESH #2 conference on Thursday, September 26th.

# Keywords: Motion Capture, virtual characters, real-time performance, projection mapping, augmented reality.

Dancers: Freeda Electra Handelsman, Rabii Hadane, Aonghus Hode, Pornpim Karchai, Francesca Lapadula, Liza Lareida, I-Fen Lin, Rozemarijn Louwerse, Lara Müller, Stefanie Olbort, Elena Paltracca, Lucas del Rio Estevez, Kristin Tims, Suzanne Vis
Visual artist: Tobias Gremmler 
Choreographer: Nadav Zelner
Assistant choreographer: Denise Lampart
Set designer: Mariana Vieira Gruenig
Augmented projection artist: Martin Fröhlich 
Motion capture: Tobias Baumann, Norbert Kottmann, Corinne Soland, Benjamin Thoma
Sound: Eric Larrieux, Hans-Jürg Hofmann
Production manager: Kristina Jungic 
Project lead: Christian Iseli

---

For several years now, the ZHdK has maintained a lively exchange with cultural and educational institutions in Hong Kong. On the occasion of an official visit to the ZHdK from the head of government of Hong Kong, Chief Executive Carrie Lam in January 2018, temporary partner labs were set up in both cities. The goal was to promote interdisciplinarity within a framework of research and development into technology-supported arts.
The resulting TwinLab project is premised upon the availability of the same technical infrastructure at both locations. The cooperation can thus be based primarily on data exchange and be sustainable. Real-time transmissions allow for synchronous experiments and productions.

The first TwinLab project “The Hidden Formula – The Heavenly Palace” was developed in cooperation with the experimental theatre group Zuni Icosahedron which is based in Hong Kong and the two performances could be experienced simultaneously in both locations. The movements of the dancers were recorded at both locations using motion capture technology to create real-time visualizations in a video projection. Despite the geographical distance of over 9000 kilometres and a six-hour time difference, the motion data was transmitted in real time, thus enabling virtual interaction.

Based on Plato’s cave parable, the ZHdK performance “The Hidden Formula” explored mechanisms of suppression and restriction as well as the human will to be free. The piece was made in close dialogue with “The Heavenly Palace” by Zuni Icosahedron, which refers to the 16th century Chinese novel A Journey to the West.
The live performance of “The Hidden Formula” took place on 25 October at the ZHdK in Zurich and simultaneously at the Community Center in Hong Kong and was broadcast live in Zurich’s Bahnhofshalle as part of the Swiss Digital Day.

For data transmission, the TwinLab project relied on the findings and technological developments of the SNF research project “Telematic Performance”, which is housed at the Institute for Computer Music and Sound Technology (ICST), ZHdK.

Cast & Crew of the ZHdK
Dance: Sophie Bertschy, Alex Ferro, Denise Lampart, Cary Shiu
Choreography: Denise Lampart
Visual Art: Tobias Gremmler
Music: Thierry de Mey
Digital set artist: Martin Fröhlich
Technical crew: Simon Broggi, Martin Fröhlich, Norbert Kottmann, Eric Larrieux, Olav Levrik, Hansruedi Näf, Marc Nathmann, Matthias Röhm, Marco Quandt, Viktoras Zemeckas and many others.
Production management: Andreas Birkle, Kristina Jungic
ZHdK producers: Chantal Haunreiter, Christian Iseli
Project steering: Hartmut Wickert
Project lead: Christian Iseli

Cast & Crew Zuni Icosahedron, Hong Kong
Dance: Chang Yu-chau, Rady Nget
Artistic direction: Danny Yung, Mathias Woo
Digital art: Dan Fong, Benny Woo
Lighting design: Mak Kwok Fai
Music: Steve Hui
Technical assistance: Dan Fong, Benny Woo, Lee Kin Tak, Tso Chi Yan, Ha Yan Pui, Ko Man Kit and many others
Production: Carmen Cheng, Chun Yin Chow, Satina Shum, Leung Kin Kai
Project management: WyWong Yuewai, Cedric Chan

---