Immersion and Embodiment

October 12th | Christian Iseli | 17:15- 18:30h
DOUBLE TROUBLE: DIGITAL AVATARS ON STAGE

October 19th | Anna Lisa Martin-Niedecken | 17:15- 18:30h
EXPLORING BODY-CENTERED DESIGN APPROACHES IN MIXED REALITY SPORTS

October 26th | Olav Lervik | 17:15- 18:30h
GAMES IN CONCERT: LESSONS FROM MUSIC IN VIRTUAL REALITY

November 23rd | Marie France Rafael | 17:15-18:30h
POST-DIGITAL RELATIONALITY – EMBODIED EXPERIENCES

December 7th | Melody Chua | 17:15-18:30h
BLACK BOX FADING: (DIS)EMBODIMENT WITH AN IMMERSIVE IMPROVISATION MACHINE

To attend the lectures on site, just follow the links for each lecture underneath and reserve your ticket via Xing. In addition, you will need a Covid-19 certificate to enter the facilities of Zurich University of the Arts. More information on the current regulations > here.



OCTOBER 12th, 17:15 -18:30 | Kino Toni, ZHdK | Live stream

DOUBLE TROUBLE: DIGITAL AVATARS ON STAGE

Christian Iseli, Immersive Arts Space

Digital characters have been around for decades. In mainstream cinema, photorealism proved to be a cost-intensive challenge and a long road through the Uncanny Valley. In the game universe, the need for real-time rendering required simplification and abstraction. Meanwhile however, with the help of ever-faster computers, game characters have come closer to their movie siblings and with the real-time capacity of game engines, digital avatars have now entered the live environment of the stage.
In this lecture, the development of digital characters in the media context will be outlined, in order to then address the specific problems that arise when doubles and twins enter the universe of theater and performances. Here, the audience needs to be able to see both the actor (in a motion capture suit) and his or her avatar so as to confirm that the avatar’s performance is indeed being created live, and not the result of a pre-recording. However, the double appearance also has its downside. The spectators need to constantly decide whom to look at: The actor or his or her digital avatar. The research project Presence and Absence of the Immersive Arts Space addressed this specific problem and focused on alternatives to the twin appearance. The outcome of the artistic explorations are discussed in detail.

Christian Iseli has been teaching and researching at the Zurich University of the Arts ZHdK since 1995. He holds a professorship for Immersive Arts, heads the Immersive Arts Space and teaches in the MA Film program. After studying history, German and English literature at the University of Bern, Iseli was a director of documentary films and worked in editing and cinematography on feature films and documentaries.
  
-> Reserve a ticket for this lecture here.



OCTOBER 19th, 17:15 -18:30 | Kino Toni, ZHdK | Live stream


EXPLORING BODY-CENTERED DESIGN APPROACHES IN MIXED REALITY SPORTS

Anna Lisa Martin-Niedecken

Powerful technologies such as Augmented Reality (AR) or Virtual Reality (VR) take sports, fitness and rehabilitation to a next level. Traditional movement concepts are expanded through interactive technologies and blended with immersive and playful experiences: Gamified fitness training, physical esports, and immersive rehabilitation are pointing us towards the future of physical activity.
In this context, the multisensory and -modal involvement of the user’s body becomes a crucial component and quality feature of these body-centered experiences. However, there is still a lot of untapped potential in the area of attractive and effective mixed reality training applications.
In this lecture, approaches, results and learnings from various interdisciplinary R&D projects in the fields of fitness, rehabilitation and (e)sports will be presented and possible directions for future sports design will be outlined. In particular, topics such as technology-based balance of fun and exertion, empowerment, and social immersion will be highlighted and discussed as potential enhancers of immersion and embodiment in mixed reality sports.

Anna Lisa Martin-Niedecken is head of the Institute for Design Research as well as Senior Researcher at the Subject Area Game Design, Department of Design at Zurich University of the Arts. With her background in sports science (PhD, Institute of Sports Science, Technical University of Darmstadt) and her expertise in game research, Anna’s research and teaching activities mainly focus on the design and evaluation of Serious & Applied Games for health, (e)sports, fitness and rehabilitation, as well as health design.
Besides her job in academia, Anna is Co-Founder and CEO of the ZHdK spinoff Sphery Ltd, a fitness gaming startup, which received multiple prestigious awards for their research-based core product, the ExerCube, as well as for the establishment of the world’s first physical inclusive eSports league, the ExerCube League.

-> Reserve a ticket for this lecture here.



OCTOBER 26th, 17:15 -18:30 | Kino Toni, ZHdK | Live stream

GAMES IN CONCERT: LESSONS FROM MUSIC IN VIRTUAL REALITY

Olav Lervik, Institute for Computer Music and Sound Technology

Games in Concert was a research project funded by the Swiss National Science Foundation at the ZHdK from 2016-2018. The goal was to explore the possibilities of musical environments for musicians in VR and to look for the artistic potentials that lie in the implementation of mechanics more known to the world of game design.
An visual VR environment was programmed with virtual instruments specifically designed to use the advantages of VR. Unto three musicians were able to interact musically with each other in VR while a forth character was used to lead the public through the performance while it was happening. Finally the project was presented at SigGraph Asia 2018 shortly after its conclusion.
During this lecture, while diving into the in and outs of “Games in Concert” the question arises which mechanics and which concepts hold potential for future musical VR environments and which challenges and lessons unfolded during this project.

Olav Lervik is a classically trained freelance composer and pianist who studied in Weimar, Stuttgart and Zurich. His works cover a wide variety of disciplines including scoring music for games and films and sound design for VR/AR applications but also classical orchestration. Olav is currently a lecturer in composition for film, theatre and media and also in sound design at the Zurich University of Arts.

-> Reserve a ticket for this lecture here.



NOVEMBER 23rd, 17:15 -18:30 | Kino Toni, ZHdK | Live stream

POST-DIGITAL RELATIONALITY – EMBODIED EXPERIENCES

Marie-France Rafael, Department of Fine Arts

What does it mean to think, to see, to act, to feel and to live through the post-digital? Since the 2010s the boundaries between public and private, online and offline have become increasingly blurred due to digitalization and social media. In contemporary art digitality has assumed a new kind of presence—no longer a mere virtual sphere of sociality, but increasingly as a technological interface that structures our most embodied experiences. Today, contemporary art cannot escape digital image production, circulation, and consumption, so what may it look like to engage it–thematize it even–in the work of art itself? A younger generation of artists is specifically focused on the mass circulation, connectivity, and constructedness of images. The production of artistic content, its circulation, and its reception are thus increasingly intertwined processes that are no longer distributed among different instances: Artistic practice has undergone a drastic transformation: it is shaped by the notion of the presence of the digital even beyond digital media and explores new forms of (post-)digital relationality as well as embodied experiences between on- and offline.

Marie-France Rafael is a Tenure Track professor in the Master program of the Department of Fine Arts at the Zurich University of the Arts (ZHdK) since 2019. She holds a PhD in Art History. She studied Art History and Film Studies in Berlin and Paris. From 2011 to 2015 she was a research associate at the Free University of Berlin and until 2019 at the Muthesius University Kiel, Department of Spatial Strategies/Curatorial Spaces. Her monograph «Reisen ins Imaginativ. Künstlerische Displays und Situationen» (Cologne: Walther König, 2017), examines artistic strategies of presentation in contemporary art. Other publications include «Brice Dellsperger. On Gender Performance» (Berlin: Floating Opera Press, 2020), «Ari Benjamin Meyers. Music on Display» (Cologne: Walther König, 2016), and «Pierre Huyghe. On Site» (Cologne: Walther König, 2013).

-> Reserve a ticket for this lecture here.



DECEMBER 7th, 17:15 -18:30 | Kino Toni, ZHdK | Live stream

BLACK BOX FADING: (DIS)EMBODIMENT WITH AN IMMERSIVE IMPROVISATION MACHINE

Melody Chua, Master of Arts in Transdisciplinary Studies

The improvisation machine is an elusive figure. It has no universally agreed-upon form but is instead in a state of continuous fluidity of audiovisual representations. Throughout the performance, it shifts between being treated as a distinct musical agent to being regarded as an embodied posthuman extension to its human counterpart. It is always in tension between the attempt to recognize it as a distinctly encapsulated entity and at the same time as a synergy of independent entities, the fluctuations between which are shaped through immersive representations of itself and the performance environment.
The development of and performance with an improvisation machine can, with careful reflection, serve as a practical method upon which to be relinquished from our traditional understandings of embodiment. The form of the improvisation machine—fragmented through spatially separated electronic devices and mediums—is inherently discombobulated. Indeed, the fragmentation of the improvisation machine reflects a mindbody conceptualization offered by media artist and researcher Lisa Blackman: “We are never a singular body, but are multiple bodies that are brought into being and held together through complex practices of self-production.”
In this lecture, I reflect upon the facets of immersion and embodiment as they manifest in the field of improvised music performance between human and nonhuman agents, using my work black box fading as a case study. How do immersive representations of and experiences between such agents on both audio and visual levels instigate different considerations of what embodiment can be? How can these reflect our biases concerning control, (dis)orientation, and vulnerability in immersive environments?

Melody Chua is an interdisciplinary artist working with interactive technologies in improvisation settings as both vehicles of narrative expression and as opportunities to destabilize the normatives present in one’s relationship with such technologies. The aesthetics of her work gravitate towards immersive futuristic fictions with undertones of nostalgia and introspection, pulling audiences into intimate worlds. Awarded a Fulbright-Swiss Government Excellence Scholarship for the development of a sensor-augmented flute (Chaosflöte), Melody has performed and guest-lectured internationally at festivals and institutions such as the Swiss Digital Day, Network Music Festival, Atlantic Music Festival Future Music Lab, La Côte Flute Festival, Performing Media Festival, Montreal Contemporary Music Lab, New Interfaces for Music Expression (NIME) Conference, University of Music and Performing Arts Stuttgart, University of South Florida, and Embry-Riddle Aeronautical University, among others.

-> Reserve a ticket for this lecture here.


Lab Insights

During REFRESH #4, team members will offer insights into the projects Shifting Realities (explorations within the intersection of reality and virtuality), Neural Volumetric Capture (experimental methods for capturing photorealistic 3D models), Digital Twins (development of animated digital avatars based on photogrammetry and Meta Humans) and cineDesk (a real-time collaborative simulation tool for the development of film scenes, VR experiences and games in 3D spaces). 

Lab Insights at the Immersive Arts Space will feature impressive demos, the opportunity to immerse oneself in virtual worlds as well as individual hands-on experiences. Members of the ISpace team will be on hand with support and available to answer questions.

Wednesday, Nov 10th 19:00-19:45/19:45-20:30h
Thursday, Nov 11th 18:30-19:15h/19:15-20h


Power and Presence

Diploma project, MA Game Design, by Oliver Sahli

Power and Presence explores meaningful and empowering interaction in virtual reality and how it can be implemented as game mechanics without breaking the feeling of being in another world.  A critical analysis of game design theories and how they need to be applied to VR is demonstrated through a game that uses phonetic interaction.

Oliver Sahli, research associate at the Immersive Arts Space and graduation student in Master in Game Design, showcased his project Power and Presend within the diploma exhibition of the ZHdK in June.

Virtual Real World

Diploma project, MA Game Design, by Chris Elivis Leisi

In today’s VR games, the body often serves as the controller. However, when the player enters the virtual world, the connection to the physical environment is often lost. This master’s thesis deals with immersion mechanics in VR and reveals the potentials that arise when one’s own home can be integrated into the virtual world as a play area.

Chris Elvis Leisi, research associate in the Immersive Arts Space and graduate student in Master in Game Design, exhibited his graduation project Virtual Real World within the diploma exhibition of the ZHdK in June.

Chimaera

Diploma project, Master Transdisciplinarity, by Bojan Milosevic

The use of human data in combination with computer algorithms creates a kind of post-human entity. In form of an improvisational dance performance, Chimaera explores the interaction between a performer and his avatar – a human-machine hybrid. The performance took place in March in the Immersive Arts Space and was streamed live.

Crew

Bojan Milosevic (project leader, audio and video coding)
Petra Rotar (dance, choreography)
Carmen Stüssi (dramaturgy)
Patrick Müller (mentoring)
Tobias Baumann (support motion capture)
Eric Larrieux (support 3D sound)
Schiesser Sébastien (IAS technician)
Martin Fröhlich (support projection mapping)


The Umbrella Project

Photo by Regula Bearth, © ZHdK 2020

The Umbrella Project is an ongoing research project that explores the use of 3D audio and projection mapping to achieve a sense of immersion without isolating participants from the real world, essentially, enabling an imaginary fantasy world to come to life in our own. We employ multiple levels of 3D audio and projection mapping (both directly within and on the umbrella, as well as throughout the room itself) in order to transport the participant into this virtual world.

The end goal of the project is to create a series of navigable compositions in the form of exploratory sonic worlds, as well as intereactive experiences where the participants’ behaviours (relative to each other and the world) shape the sonic and visual environment. Furthermore, we are investigating sonic and visual paradigms where the umbrellas can function both as objects existing in and can interact with the virtual world, as well as being windows onto these other worlds.

Naturally, these environments are best experienced from directly underneath the umbrella, where one can best appreciate the various levels of MR.

A spin-off of The Umbrella Project was presented at the REFRESH conference: An installative performance with the title A Day at the Beach.

Crew:
Eric Larrieux (lead), Stella Speziali, Martin Fröhlich, Corinne Soland, Mariana Vieira Grünig


Kusunda

Filmmaker in Residence Gayatri Parameswaran

KUSUNDA is a virtual reality documentary experience about what it means to lose a language and what it takes to keep one alive.

Narrated by two of its co-creators — 86-year-old Kusunda shaman Lil Bahadur and his 15-year-old granddaughter Hema — the experience contrasts two generations set apart by their lifestyles and brought together by the struggle for their indigenous identity.

In the VR experience, you join Hema as she reminds her grandfather of his forgotten mother tongue. You navigate by speaking words in the endangered Kusunda language and join an audible fight against its extinction.

Most of Lil Bahadur’s story happens in the past — especially his life in the forest as part of a hunter-gatherer group. The recreation of this non-existent past lends itself naturally to the use of animations within virtual reality.

With the help of the motion capture facilities at the Immersive Arts Space at the Zürcher Hochschule der Kunst (ZHdK), actors were recorded recreating these sequences from Lil Bahadur’s past. This not only simplifies and speeds up the process of character animations but also offers unique possibilities for the documentary storytelling form.

Crew at ZHdK:
Cast: Yan Balistoy, Offir Limacher, Johannes Voges, Ferhat Türkoğlu, Liliana Heimberg, Corinne Soland, Oliver Sahli, Kristina Jungic
Mocap coaching and production: Corinne Soland
Mocap recording: Tobias Baumann
IASpace producer: Kristina Jungic
Further ZHdK support: Chantal Haunreiter, Martin Fröhlich, Stella Spezialli

ZHdK Residency
The residency at the Zurich University of the Arts has been made possible by the Ernst Göhner Foundation, Switzerland, as well as by additional support of the ZHdK film program (led by Sabine Boss) and by the Immersive Arts Space (led by Prof. Christian Iseli).

Further crew members:
Co-creators: Gyani Maiya Kusunda, Hema Kusunda, Lil Bahadur Kusunda; Storytelling/Production coordination: Felix Gaedtke, Gayatri Parameswaran; Executive Producer: Rene Pinnell; Associate Producer: Mia von Kolpakow; Co-Producers: Emma Creed, Aliki Tsakoumi, Sönke Kirchhof, Philipp Wenning, Kuan-Yuan Lai; Lead Developer: Tobias Wehrum; Art Director, 3D designer & animator: Moritz Mayerhofer (and team); Volumetric Video post-processing: INVR.SPACE; Photogrammetry post processing: realities.io; AI speech recognition: Valerio Velardo; Sound designer: Mads Michelsen; Project website: Tom Lutherburrow | Nepal Production team: Direction/Production: Felix Gaedtke, Gayatri Parameswaran; Line Producer Nepal: Deepak Tolange, Sandeep Bhaju; Volumetric Video & Photogrammetry: Felix Gaedtke, Gayatri Parameswaran; DoP and Drone pilot: Aditya Thakuri; Sound recordist: Mia von Kolpakow; Kusunda linguistics researcher: Uday Raj Aale; Driver: Dharmendra Shakya


BATVISION

© ZHdK 2020

«Have you ever wondered how a bat perceives the world?» BATVISION offers the chance to playfully explore how bats use echo-location to detect their surroundings. The VR-experience visualizes the bat’s auditory sensation and makes it more tangible. Surrounded by complete darkness, the virtual world only becomes visible when the users start shouting. BATVISION simulates the ultrasonic navigation of bats, enables new forms of perception and raises the awareness for an endangered species.

BATVISION is a collaboration between the ZHdK Industrial Design program and the Immersive Arts Space. In their bachelor thesis Eliane Zihlmann and Raffaele Grosjean developed the concept of the VR experience and the associated hardware design. They were supported by IASpace staff members Oliver Sahli (programming, visual implementation, interaction control), Chris Elvis Leisi (implementation of multi-user functionality) and Florian Bruggisser (3D scanning and point-cloud processing).