Daniel Strutt
Search for other papers by Daniel Strutt in
Current site
Google Scholar
PubMed
Close
Dancing into the metaverse
Creating a framework for ethical and ecological telematic dance practice and performance

When COVID-19 hit we all quickly adapted to using two-dimensional video conferencing technologies to connect with others. However, these social interfaces were, for many, an alienating experience. It was especially challenging within the fully- embodied discipline of dance, with practices stripped of the co-presence, touch, and physical communication that performers depend on. At this time, a Goldsmiths-based network of academics, dancers and creative technologists turned our attention to think about the kinds of motion capture and real-time, generative graphics technologies that could converge to achieve a more effective framework for remote dance interaction. We offered Goldsmiths Mocap Streamer, a tool to bring multiple performers from anywhere in the world into connected virtual spaces. While initially framed as simple solution to physical distancing during the pandemic, our research led to the consideration of transformational experiences for dance and beyond. Through a series of workshops and showcases we evidenced that we are not only at a point of sufficient technical and aesthetic advancement to offer high-quality virtual dance work, but  that this mode of performance also has potential to mitigate the serious climate impact of touring productions. Furthermore, collaborator interviews and Q&As revealed that a distinctive mode of choreographic contact and intimacy emerges through these remote practices that have the potential to cultivate unique structures of empathy and awareness. Thus, we ask, can ‘dancing in the metaverse’ now develop as not only legitimate and ecological, but also as an ethical form of dance practice? 

In the last 12 years, several developments and convergences in motion capture (mocap) and immersive media technology have seen the synthesis of new and hybrid systems for embodied, extended and augmented digital performance.1 In the same way in which ‘camcorder’ home video recording and playback became affordable in the 1970s and 1980s and subsequently revolutionised many arts and performance practices to become ‘multimedia’, the recent waves of hardware and software tools have quickly come down in price, such that they have become accessible to small companies and even enthusiastic individuals.2 More recently, the conditions surrounding the various coronavirus (COVID-19) pandemic lockdowns – with artists frustrated by both the impossibility of sharing analogue/physical spaces, and the two-dimensional limitations of videoconferencing for sharing work and practising – generated a clear and decisive turning-point in immersive tech production practices, both in their take-up within the performing arts and in wider cultural engagement with a discourse about the kinds of artistic expression that ‘Metaverse’-type virtual performance systems can afford.3

Around the same time as the onset of the pandemic in 2020, a network of academics, dancers and creative technologists based at Goldsmiths, University of London, began to think about the kinds of existing technologies that could be drawn together to build a system for meaningful remote dance interaction. We offered a concept of a system for the streaming of motion capture data, asking could we bring three-dimensional dance movement from remote performers around the world into a connected virtual space, live and in real time, as a legitimate form of dance practice? We developed a simple-to-use, open-source software tool named ‘Goldsmiths Mocap Streamer’, which could grab motion capture data locally and send it to any other computer in the world to be rendered within a real-time graphics engine (Figure 2.1), or indeed within a cloud-based Metaverse provider. Our team then used this framework to address experimentally a set of critical aesthetic and technical questions. What kinds of meaningful movement and dance interaction and communication can be generated remotely through the use of immersive tech? What kinds of alternative or augmented aesthetics of dance interaction could be explored in shared virtual spaces, where the dancer’s avatar body does not even need to be fully ‘corporeal’? What are the limitations of, and obstacles to, this kind of remote collaborative dance practice in terms of connection, interaction and sense of presence?

These original research questions are now complemented by an awareness of the need, in the current state of climate emergency, to clearly connect innovation ideals with actual societal needs. At the end of 2020, the meeting of the International Network for Contemporary Performing Arts (IETM) set an agenda for ‘greening the performing arts’ that was positioned both within the ongoing climate emergency, and within the pandemic itself. Their report, Climate Action and the Performing Arts, evoked the landscape in which it was written:

Skies cleared over the most polluted cities on Earth, while people sought cover from a deadly virus whose origin we now know to be directly related to damages humans inflicted on ecosystems. Climate change became more visible and tangible than ever. And so did the urgency to tackle it within the performing arts scene. (Skolczylas, 2020: 3)

This vivid picture of a cessation of normal polluting activity invites a reconsideration of the current Zeitgeist moment in performance tech (and Metaverse hype) through a more socially pragmatic lens, framing it through the possibility of mitigating the need for environmentally damaging touring practices. In addition, could this moment also instigate the lowering of barriers to access into collaborative digital spaces, so that people, regardless of geography, disability or social status, can come together over great distances to move and create with a convincing sense of embodied presence, interaction and communication? We now move to ask: could ‘dancing in the metaverse’ come to be seen, not only as legitimate, but even as the ethical form of future dance performance?

As we have discovered in our research, technical, aesthetic and environmental innovation goals are not necessarily in opposition to each other. In contemplating the historical lineage of remote, networked, distributed or telematic performance practice, developing as a form since the 1960s through the work of artists such as Roy Ascott, Paul Sermon (see Chapter 3 in this volume), Laurie Anderson, Joan Jonas and ORLAN, we can see that practical concerns around embodied presence and liveness, collaboration and interaction have often coexisted with more philosophical questions of conscious evolution, empathy and ethics. This has usually involved speculative and utopian ideas around altered states of extended embodiment, what it means to be ‘together’ or virtually present, and of harmony and care for the other. However, this philosophical approach to immersive networked technology may also frame solutions to ecological and other societal concerns. Such ethical practice might generate an understanding about the role of the performing arts in future visions of digital society, and of a social and cultural metaverse.

In this chapter I work towards a genealogically supported vision of telematic dance and performance work as driving the sort of innovation that might address some of these ecological and other societal concerns. To do this I draw together some historical and then institutional discourses before turning to a model of analysis of ‘user’ experience, to frame the research of Goldsmiths Mocap Streamer through the words of the dancers themselves. In doing this, I approach the speculative ‘promise of the digital’ through not only short-term technical and aesthetic innovation concerns, but also through affective, ethical and ‘technoetic’ philosophical frameworks.4

From telematic to metaverse futurity

In 1966, cybernetic visionary and artist Roy Ascott

articulated a cybernetic vision for the arts that began with the premise that interactive art must free itself from the modernist ideal of the perfect object. He proposed that artwork be responsive to the viewer rather than fixed and static, suggesting that l’esprit cybernétique offered the most effective means for overcoming the separation between artwork and spectator. Ascott’s vision of networking as a shared activity that is ‘both dance and an embrace’ resonated with performance artists who … brought together remote participants to produce multi-site, real-time, interactive dance, music and theater experiments. (Giges and Warburton, 2010: 25)

In what is effectively an alternative origin for the type of shared virtual space now dubbed the ‘metaverse’, Ascott imagined a non-linear, processual interaction between participant and artwork that ‘is not primarily a thing, an object, but a set of behaviours, system, actually a system of systems’ (1990: 243). He emphasised the emotional and embodied nature of this vision, in a relationship of mutual becoming – where the artwork ‘embraces the viewer in the creation of meaning’ (1990: 242) in a ‘spiritual interchange’ imbued with a sense of intimacy and affection (1990: 247).

First framed by cybernetic theory, Ascott’s artworks and writings later became primarily positioned through the idea of the ‘telematic’, a term introduced in 1978 by Simon Nora and Alain Minc (Shanken, 2000). As a concept, ‘telematic’ simply refers to remote subjects brought together through computer-mediated communication. But even early on, it gained, particularly for Ascott, a decidedly utopian nuance, regarding the evolution of human consciousness:

With the computer, and brought together in the telematic embrace, we can hope to glimpse the unseeable, to grasp the ineffable chaos of becoming, the secret order of disorder. … More than a technological expedient for the interchange of information, networking provides the very infrastructure for spiritual interchange that could lead to the harmonization and creative development of the whole planet. (Ascott, 1990: 247)

While aspects of this utopianism seem somewhat anachronistic today, much of Ascott’s telematic vision does still remain relevant at a time when tangible interfaces are disappearing and we are becoming increasingly immersed in virtual worlds. Ascott offered a cyborg vision of conscious fusion with the computer that clearly suggests itself to new immersive technologies: ‘As we come to see more, we shall see the computer less and less. It will become invisible in its immanence, but its presence will be palpable to the artist engaged telematically in the world process of autopoiesis, planetary self-creation’ (Ascott, 1990: 247).

Current metaverse visions can be seen as simply the next iteration of this same idealistic imperative, now framed by the convergence of new VR and XR interfaces, real-time generative graphics and motion capture systems. However, it should be noted that when we arrive at current simulation technologies, a decisively new dynamic relationship between felt presence and embodiment emerges. Performance theorist and practitioner Johannes Birringer describes this as a ‘historical shift … between video dance (choreography for the camera) and interactions systems’, described as ‘sensing systems worn on the body which generate data in real time or produce data that can be digitally processed and transcoded’ (2008: 25). We now see simulation computation rendering complex models in real time and on immersive interfaces, with open-ended, generative images forming in response to participants’ actions. This, for Birringer, yields a significant disruption of previous practices, even a new ontological landscape:

For example, [post-human performance artist] Stelarc’s strategic exploration of the non-conscious agent, of telematic scaling and the engineering of external, extended, and virtual nervous systems for the body undermines the humanist and rationalist conceptions of the body and subjectivity, of mind, instrumental reason, and consciousness. (Birringer, 2008: 25)

However, while this technology is no doubt disruptive to the structures of representation that have developed through media technology over the last 150 years, and while it may indeed shift our understanding of humanity, in a more pragmatic sense can we truly say that it addresses our contemporary and urgent real-world needs?

Contemporary concerns and realities

What twenty-first-century developments in virtual and immersive media mean in relation to ‘physical’ reality, consciousness and experience is hotly debated.5 The quasi-spiritual futurist vision of a ‘technoetic’ evolution of consciousness that Roy Ascott still pursues to this day can seem naïve when tech corporations manipulate global democracy through the harvesting of sensor data, and when fresh hardware with built-in obsolescence is released daily while a climate disaster looms (Chua, 2006: notes 12–14). What seems called for is a more context-specific and pragmatic approach to assessing the value of metaverse technologies, including our own Mocap Streamer tool, within our present-day eco-political context. This might mean exploring whether the new immersive technologies for remote collaboration and performance offer a viable alternative to touring that might help address the climate impact of the performing arts. While lockdowns rather forced the issue of maintaining arts and culture in online forms, can we now consider the more long-term ecological benefits and potentialities of this shift to virtualised media arts?

The environmental sustainability of arts, culture and the performing arts was an issue that was being institutionally foregrounded prior to the pandemic. The 2019 Arts Council England Sustaining Great Art and Culture Report acknowledged that ‘culture has a carbon footprint, and data matters … There should be no opposition between what art is and how art does: reducing emissions prompts creativity and activism, and vice versa’ (ACE, 2019). This report offered figures on the most common impacts that can be meaningfully tracked – of energy, water, waste, business travel and touring – across 747 arts and culture organisations in England. Travel and touring were found to equate to 114,547 tonnes of CO2, with dance and theatre together generating about 20% of this. While this damaging activity pretty much ceased during the lockdowns, at the time of writing, and in our haste and desperation to get back to ‘normal’ regardless of how damaging that ‘normal’ might have been, we seem to have backslid in terms of progressive environmental policy without learning any lessons from the pandemic.

The 2020 IETM report gathered many ideas, suggestions, tools and experiences that considered the role of performing arts in addressing climate change.6 The report includes a section on ‘the promise of the digital’ which asks a set of technical and aesthetic questions to frame the potential of digitally streamed performance, motion capture and gaming tech, for sustainability and reduced environmental impact:

Producing digital works opens up the question of the live quality of such recordings, of whether it can be replicated, and if not, is it necessary? What is the value of these new experiences? How do we direct the gaze of the public in such a setting? What audiences does this include (people with disabilities or unable to afford the ticket in the opera), and which does it exclude (those with poor internet access or insufficient digital skills)? How to create performances that can work as hybrids, and can be rehearsed online? Can video games provide interesting solutions designing engaging, first-person experiences for the audiences? Finally, can performing arts participate in shaping new technologies, rather than passively adopting them? (Skolczylas, 2020: 11)

It is worth noting that even these digital modes of access to performance through streaming and cloud computing are not carbon neutral of course, and in fact got progressively more polluting during the pandemic. Media philosopher and activist Laura Marks, creator of the Small File Media Festival (which challenges the need for high-definition media images), speaks to the real impact of our media streaming culture:

We can now corroborate the [French think tank] The Shift Project’s analysis that streaming video is responsible for over 1% of greenhouse gas emissions worldwide. Exacerbated by new habits established during the Covid-19 pandemic, that figure is currently estimated at 1.2% to 1.4% and rising fast. (Marks, 2021: paragraph 2)

The main impact of this activity is from the power consumption of media streaming giants such as YouTube and Netflix, where the pursuit of ever higher quality images directly compounds carbon output. Videoconferencing, due to the standard lower resolutions, pollutes considerably less than streaming video. Furthermore, the streaming of motion capture data rather than of whole images, with tiny packages of lightweight data, is less energy consuming than this again.7 This does not mean, however, that our own Goldsmiths Mocap Streamer tool, and this mode of virtual production more broadly, is without impact. We still extensively use video conferencing platforms adjunct to the streamer itself, to meet, plan, share the image back to remote dancers and provide a simulcast video feed of the physical dancer(s) for audiences (to allow them to understand the performance’s liveness). We must acknowledge that technological innovation in the performing arts is not simply a quick solution to climate realities. The promise of the digital, as much as it is about technical and aesthetic innovation, must also be guided by an ethical approach of moderation and sensitivity towards ecology, embedded in day-to-day practices, that weighs audience experience and artistic aspiration against environmental impact. We can, however, think about how this sensitivity can catalyse innovation, creativity and activism in ways that might allow the arts to shape new technologies for social good, rather than merely adopting them.

Goldsmiths Mocap Streamer: User experience

It is perhaps easy to accept in principle that the new metaverse-type iterations of telematic performance work are a good thing, in terms of their potential not only to connect performers and audiences remotely but also to diminish the climate impact of the performing arts. However, in the same vein as the stated goal of the Small File Media Festival – ‘Movies don’t have to be big to be bingeworthy!’ (Marks 2021, section ‘Call for work, Second Annual Small File Media Festival’) – can we yet say that good live performance doesn’t require the physical presence of either audience or performers? Or confidently state that we can deliver experiences which are aesthetically satisfying, and no longer merely a substitute for live theatrical performing arts?

Much recent research has turned to audience research to address these questions. A recent UK government policy POSTnote (May 2022) acknowledged that, while there is a relative abundance of quantitative research ‘mostly focusing on audiences or organisations’, … ‘there is limited evidence on how performers and artists are affected by digital technology’ (Cîrstea and Mutebi, 2022: 4). Acknowledging this dearth of research on how performers actually feel about digital technologies within their disciplinary practice, it is perhaps also key to note, as performance theorist Johannes Birringer does (2008: 35), that in interactive performance systems, the distinction between audience and performer, or between ‘viewer’, ‘participant’, and ‘user’ can become complexified, or even break down. Understanding the new kinds of interactions, actors and interfaces within non-linear, interactive performative events can mean that it becomes difficult to ascertain what ‘audience’ even means, as all participants effectively become co-creators.

In what follows, I seek to explore this convergence of experience by bringing a structured model of audience research to my own qualitative and often informal research into performer experience. I focus on the experience of our project’s collaborating practitioners, who have led or participated in the workshops and showcases in which we have connected dancers from Singapore, New York and Hong Kong to those in the UK. In these sessions, the dancers wore Noitom ‘Perception Neuron’ inertial motion sensor (IMU) capture kits, allowing them to move and interact in a virtual landscape in real time. The virtual performance spaces were generated by real-time games engines Unity and Unreal Engine, with live interaction design through sets of digital tools such as shaders, colliders, and GPU particle effect. These tools permit reactions between avatars and objects, and give dancers the feeling of virtual touch, proximity and gravity, that allows a sense of kinesthetic embodiment within the virtual scene. More than this, it allows a kind of augmentation of both kinetic force and proprioception which can both extend and distort normative modes of dance embodiment (Figure 2.2). Throughout the research we asked dancers about how this mode of practice feels, the kinds of obstacles that they face both cognitively and physically, and if they can see themselves productively and creatively engaging with these systems in the future.

The IMPACT audience research model

In 2018, Nesta and i2 Media Research group released a report on Evaluating Immersive User Experience and Audience Impact, which drew attention to the ‘challenges of understanding the value of creative content while … traditional quantitative measures of measuring impact are not always available or reliable’ (Nesta and i2, 2018: 1). They proposed to ‘develop a research methodology for testing and evaluating the experimental immersive content that is being made now, in a way that will help us predict the potential audience appetite, cultural impact, and commercial opportunity in the future’. This report identified some key predictors of experience in immersive experiences through concepts of positive affect, engagement, and sense of presence, drawn from psychological models which render these states quantifiable. i2 Research then honed their own model as a toolkit for both the creation and evaluation of immersive experiences, offering what they call quantifiable ‘high level categories’ as ‘helpful thinking aids to stimulate conversation and deliberation during design sprints’ (Kurta and Freeman, 2022: 209).8 These categories of experience, or ‘states’, are offered with the acronym IMPACT (Kurta and Freeman, 2022: 200):

  • Interesting: Capturing initial interest through automatic and personalised attentional cues.
  • Meaningful: Connecting to user values and cultural or universal goals. Meaning goes beyond initial interest.
  • Personalised: Personalising design features. Personalisation focuses on individual relevance, e.g. achieves personal ‘to be goals’. [sic]
  • Affective: Emotionally arousing. The affective dimension acknowledges how changes in arousal level are important for maintaining interest and engagement.
  • Collective: Enabling social experience or ability to connect and share with others. Features may connect individually or demonstrate collective action or community.
  • Transportive: Feeling presence, sustained attentional engagement. The transportive dimension guides design which resolves user friction and minimises disengagement via distraction.

While these categories are deduced as insights into the key domains of audience experience, they suggest themselves as useful modes of thinking about the relative quality or success also from the performer’s point of view. Below, I adopt the IMPACT model as a thinking aid to reflect in a structured way on the experience of our own project’s dancers and choreographers in their collaborative practice with our 3D interaction designers and creative technologist. This interview material was gathered through question-and-answer sessions and personal interviews with some of the dancers, choreographers and project partners that we worked with in the course of the project. The dancers quoted here are Mavin Khoo, Creative Associate of Akram Khan Company, and Alexander Whitley and Tia May Hockey of Alexander Whitley Dance Company.

The performer’s experience

 Interesting/meaningful

The first two IMPACT states of ‘interesting’ and ‘meaningful’ are intimately connected. The first ‘captures initial interest’ through a measure of novelty. The second is concerned with sustaining interest and attention through embedding technological innovations within a meaningful structure (Kurta and Freeman, 2022). The i2 researchers note that the ‘psychological impact of content can be “broken” or diminished by poor technology implementation’ (Nesta and i2, 2018: 10), referring to the possibility of tech failure or a glitch that would draw the participant out of the experience. They also note how perceived gimmickry may cause the same ‘break’, as the initial point of interest proves to be depthless. Mavin Khoo spoke eloquently on this point in relation to his company’s storytelling ethos:

One of the things that I have been questioning is: how are we able to embrace this element of motion capture creatively into our work without it becoming an impressive tool that is there to impress? How does it sit within the poetic nature of the kind of work that our company does? Is there a possibility for it to become a poetic collaborator in the kind of dramaturgical frameworks of the kind of work that the company makes? And so I think that has been a consistent framework that is always there to push me, to ensure that it doesn’t just become an impressive tool. (Khoo, 2022)

In conversation with the author, Khoo expanded on this point of striving to avoid gimmickry. Having just extensively used a form of motion capture animation (rotoscoped) in Akram Khan Company’s production of The Jungle Book (2022), he mused:

One of the things I’ve discussed many times in the process is, for me, it’s such a dangerous point when an audience gets taken into an emotional experience journey, and suddenly for one second they go: ‘wow that’s really clever!’, and that really clever part means we’ve lost them from the story, and we’ve lost the ability for the work to speak. (Khoo, 2022)

In many ways, Khoo’s issue of the technology being a ‘poetic collaborator’ became the guiding mantra throughout our research. We are acutely aware that the technology must serve the intention of the dancer if it is to become embedded meaningfully within practice. On this point, Khoo had some critical thoughts about some of our 3D avatar and interaction design (see Figure 2.3).

There’d be something so beautiful that Neal [Coghlan, 3D designer] would have constructed and designed so beautifully, and it’s amazing, and then my question is, okay, after two minutes we’ve got it, and then what? One rule I would always have is that, however much it’s integral in that process for those elements to come together, the rule that I have on myself is also independent of all those elements, i.e. how strong is the dancing? (Khoo, 2022)

Khoo’s comment highlights how important it is that the whole team, from software developer to animator, to dancer, and then, ultimately, to the audience, focuses on the poetics of the performance, which needs to lead (if not limit) the technological marvel. There is, then, a magical moment when this human/technology collaboration clicks. As dancer Tia May Hockey observed:

What was interesting for me was that when we all had the same ideas, and timings of when those ideas were happening, even though the movement wasn’t necessarily choreographed, we were connected by the same ideas … It was very interesting to experience [the difference between] days where we were just playing around with no idea set and we’re all just exploring in our own worlds, to then suddenly as soon as we all have the same intention it does feel like there’s meaning. (Hockey, 2021)

Khoo and Hockey’s comments give insight into the way that, from the perspective of both dancer and audience, the technical and aesthetic innovation of the work must be in the service of artistic expression and poetry. To be interesting and meaningful it must mesh with both the intention of the performer and the broader discipline in which they work – to generate meaning (embedded within a dramaturgical framework or ‘journey’), to exhibit a disciplined artistry and to ‘allow the work to speak’.

 Personalised

For dancers within the mocap streaming framework, the question of personalisation as ‘individual relevance’ seems, to a large extent, to come down to the feeling of embodiment within the avatar figure. Our dance partner Alexander Whitley spoke about the negotiation involved in achieving this sense of personal investment with the virtual body, particularly when the expression of the body is not exactly figurative:

I mean there’s a few things going on; there’s the avatar figure itself which obviously has kind of added appendages and things that aren’t naturally part of your body. So there’s some interesting kind of working out of how it relates to you, which naturally changes how you’re thinking through your body. The different qualities and rigidities or softness of the figures you embody quite naturally, … so when you recognize enough of yourself in those figures you automatically start to take on their characteristics or qualities. (Whitley, 2021)

It is this process of ‘taking on’ and internalising the traits of these avatars that seems to enhance the personalisation of the experience, where one starts to identify, and to feel and act differently. In a second interview, Whitley explained:

The thing that really just stood out to me immediately is that the process of visual feedback about action perception and negotiation of space is really different and just really interesting in terms of how it provokes responses in you as a mover. These real-time live setups are so exciting in terms of how you can embed ideas or concepts into a system, in a process that will have a really significant effect on the kind of movement that you then generate. (Whitley, 2022)

This real-time feedback, and the practice of feeling-out the kinaesthetic possibilities of the avatar, are a process through which we come to ‘recognise’ it as an extension of self. Moreover, this is a modified self, structured by a set of intentions and concepts made ‘flesh’ and with which the dancer can explore alternative or augmented subjective and personal experiences of embodiment.

 Affective

The affective dimension of experience seems to follow from and extend this sense of personalised embodied investment in the avatar and its potentialities into a deeper emotional sense of ‘presence’, such that the dancer can feel that they are expressing something of their interiority through it: to ‘reveal ideas of choreographic intention or performance intention through the kind of visuals that are generated’ (Whitley, 2022). For Whitley, this kind of ‘deep’ connection is really the purpose of continuing to engage and experiment in this realm. He states, ‘I guess facilitating those more deeply embodied experiences through this kind of technology is what really excites me. And it obviously presents some great solutions to the challenges we’ve been experiencing over the last year [of the pandemic]’ (Whitley, 2021).

For Khoo, however, there were always reservations. He confessed, early in our research process in 2019: ‘I was genuinely concerned that the presence of motion capture in my process would be problematic in generating a kind of emotive human ownership to the work’ (Khoo, 2022, in Strutt, 2022). Later, he again noted the aesthetic and design element as a major factor in being able to achieve this sense of investment: ‘whatever the creation of that avatar is, if you like the nature of that, plays a huge importance in terms of how far one is able to invest in it from an emotional perspective’ (Khoo, 2022). He describes:

The dancer has to go to a space where they understand the potential of what the avatar is and what the potential of their physical body is in relation to those other elements of the form. They start off just negotiating and navigating that, and then they become engaged and aware of these supporting avatars around them, and then eventually merge into one. (Khoo, 2022)

Our research attests that it is certainly possible to experience a kind of affective immersion or ‘merging’ with the system through an iterative process of exploration and experimentation. While Khoo and Whitley’s remarks show how this ‘merger’ is initiated through a dancer’s exploration of their own avatar form, one of the most significant findings of our research is how emotional ownership is dramatically heightened when the participant starts to move and ‘merge’ with other avatar performers within the remote streaming ‘metaverse’ framework. Ironically perhaps, it is in the displacement of the attention on self, through focusing on moving with the other avatar figures, that the potent, transcendent affect of ‘deep embodiment’ seems to occur. This collective sharing of experience, with real-time sensory feedback from a partner, enhances the sensation of deep kinaesthetic embodiment within the avatar form.

 Collective

Fitting neatly into the IMPACT model’s categorisation of collective experience, the primary purpose for the development of our own mocap streamer tool from the outset has been to ‘enabl[e]‌ social experience or the ability to connect and share with others’ (Kurta and Freeman, 2022: 200). The dancers’ reflections then show how the collective aspect of the work takes on an even more affective and almost ethical nuance of ‘taking responsibility’ for the remotely connected other. Khoo described: ‘There’s a sense for the dancers of having to first take responsibility in terms of what their own potential is, and then of course the minute there is the engagement with the second or third avatars, to take responsibility for the others as well’ (Khoo, 2021).

Of course, this is something that would happen in the physical studio quite normally. However, as Whitley observes, the remoteness of the streaming tool is not too much of an obstacle to this, though there is a need to develop a set of task-based, semi-improvisational techniques to lean into this sensitivity, honing attunement with the rhythm, intention and nuanced expressivity of the other, remote, dancers through their avatars.9

Choreographing something would defeat a lot of the purpose of this and what makes it really special as a tool is its liveness, and the fact that we can be interacting and responsive to each other in the moment. I think the fact that we could be as responsive to each other, and work around the kinds of tasks that we would normally work with in a studio setting, shows the success of this tool – that it … really enables us to interact in three-dimensional space with other people in a way that feels quite natural and intuitive to us as dance practitioners. (Whitley, 2021)

Tia May Hockey adds nuance to this idea of responsivity, and responsibility, in an interesting analogy between the avatar and an animal (Figure 2.4):

When I had a shared intention to play with in improvisation with another performer, I felt moments of connection with our virtual characters and through to my being. It’s kind of like the connection you make with an animal and you both know that you’re watching each other, there’s a level of sensitivity required by both parties to listen, anticipate, predict, and respond. It was hard to maintain this sensitivity, but I did experience flashes of it. It’s easy for this experience to be cold and alienating, and so far it has mostly felt this way. However, with shared intentions, heightened awareness, conscious effort and sensitivity I believe a connection can be formed and enhanced. (Hockey, 2021)

There is a complexity to this work that a dancer, perhaps intuitively, understands more than most, due to their already disciplined practice of sensing, anticipating, predicting and responding to the movement of another. However, what we have noted is that there is implicit ethical inflection to these practices when the ‘other’ is remote and not even within their own body. We could perhaps call this an ‘alien phenomenology’ which can yield transformative experiences of connection and interaction through exploration of alternative forms and ‘augmented’ embodiments.10

 Transportive

With her expression of the experience of ‘connection through to her being’, Tia May Hockey nicely leads into the IMPACT model’s notion of the transportive as a sense of flow or deep unbroken attentiveness. In the same interview, she explained:

There were definitely some out-of-body experiences and moments where I felt the potential of the virtual connection with the other performers. It required me to fully engage with my imagination and to let the reality around me disappear in order to let my attention be fully present in the virtual space. (Hockey, 2021)

Similarly, Khoo noted that there is a kind of altered state that needs to be reached, where ‘psychologically the dancer has to come to a space where they are able to let go of what their habitual pattern would be’. This led him to reflect that ‘[y]‌ou get quite easily in this state like “I’m in another realm and so much new is happening”. … It’s so interesting and there is something in that psychology, I think, that is actually bigger than we consider’ (Khoo, 2022). It seems that here, in the collective moment that is facilitated by the motion capture streamer as much as by the elements of avatar and interaction design, there can be an intense and transportive sense of almost ‘out-of-body’ engagement that fixes our attention on our attunement and sensitivity to the other figures. This, as Khoo notes in terms that seem close to Roy Ascott’s utopian dreams of global harmonisation, is perhaps ‘actually bigger than we consider’.

Conclusion

This chapter set out to question whether, within our current moment, we are truly at a point of sufficient technological advancement in which remote connection can become a legitimate, mainstream and carbon-conscious alternative medium for dance practice and performance. It is clear, however, that there are no quick or direct fixes in terms of technological or aesthetic innovation. We need to look a bit deeper, and more speculatively, at the ‘promise of the digital’ (here, specifically, of metaverse-adjacent tech) as a performance medium that can meaningfully connect people toward shared experience. Indeed, this is not a new critical approach, but one that is broadly in line with a longer genealogy of telematic connection in the arts.

Though in one way these questions about consciousness and rationality might still seem a little disconnected from the clear and present climate emergency, there is a burgeoning literature on how this more philosophical approach does in fact speak to these concerns. While there is a fair amount of good critical work on VR and immersive media as an ethical ‘empathy machine’,11 it is particularly interesting to think about Laura Marks’ recent work,12 where she speaks of a ‘talismanic image’ that cultivates an active form of sensory engagement with images, and which makes us palpably feel the ‘interconnectedness’ of things. She says:

Some films carry out operations on the universe: performative films, and fabulative documentaries, movies that make things happen. Many movies, including some environmentalist films, activate sympathy among entities at different levels of the cosmos. A very few works make an affective fold that reaches all the way from the cosmos to your body, through delicate and risky processes of contact, correspondence, sympathy, and passion. (Marks, 2020: 253)

She asserts that this kind of image can not only alert us to our responsibility to other entities, but also stimulate activism and action: ‘When images make contact with us, we recognise that we are strung in webs of causality. By cultivating affective response and indexical awareness, we can draw the universe close, even rearrange it’ (Marks, 2020: 255).

Ending on this point, I suggest, drawing Roy Ascott close to Laura Marks, that the promise of ‘dance in the metaverse’ is not only that it might just mitigate the carbon footprint of the performing arts, but moreover that telematic performance has the potential to cultivate structures of empathy and awareness in ways that are not just abstract messages. Rather they are embodied and connected practices and processes which might lead into action (the modification of existing damaging behaviours) and even activism (spreading the word and creating new kinds of content). Returning to Tia May Hockey’s words, ‘with shared intentions, heightened awareness, conscious effort and sensitivity’, we might get there yet.

Notes

1 Developments such as inertial motion sensors (IMU mocap systems emerging in around 2016 onto the consumer market), machine learning tools (ML), Virtual Reality 2.0 (VR, with this phase generally acknowledged to have commenced with the arrival of the Oculus Rift in 2014), and increasingly sophisticated real-time graphics engines such as Unity and Unreal Engine.
2 To illustrate, the most recent version of the Perception Neuron kit (v.3) is priced at $3,400, the Rokoko Smartsuit pro at €2,800. Virtual Reality headsets have become more accessible at £400, with Sony PlayStation VR at only around £200. Games engines for real-time graphics rendering such as Unreal Engine and Unity are now largely free to use.
3 See the foreword to the European Dance Network’s 2021 report Virtualised Dance? Digital Shifts in Artistic Practices for a neat summary of this: ‘Since March 2020, we have observed that artists and dance professionals have altered their practices and that the creation, production, and distribution of artistic work have increasingly explored digital spaces and tools. Pre-existing trends have become predominant’ (Floch, 2021).
4 Ascott describes technoetics as the evolution of consciousness through technology (2008: 204)
5 For instance, in David Chalmers, Reality+: Virtual Worlds and the Problems of Philosophy (2022).
6 Similarly, during lockdown, the ‘Theatre Green Book’ was produced through in-depth studies, workshops and consultations with the UK theatre community, offering guidelines around sustainable production, venues and operations. See https://theatregreenbook.com/.
7 A common streaming service requires 7GB per hour of streaming in high video quality (Ultra HD or 4K). A standard videoconferencing service uses about 2.5GB per hour (Obringer et al., 2021). Streaming motion capture in two directions (a duet) within our own project, based on our own cloud server usage, peaks at 1.8GB per hour.
8 This was achieved through extensive surveying of the audience for the RSC’s 2021 Dream online production eliciting some 6,500 audience responses.
9 Attunement is a psychological term that has moved through affect theory. A definition of attunement ‘is a kinesthetic and emotional sensing of others, knowing their rhythm, affect, and experience by metaphorically being in their skin, and going beyond empathy to create a two-person experience of unbroken feeling connectedness by providing a reciprocal affect and/or resonating response’ (Erskine, 1998: 236).
10 Alien Phenomenology is a term coined by Ian Bogost within an Object-Oriented ethical and speculative philosophy, a project that displaces humans as the sole-makers of meaning and explores alternate subject positions: ‘A bold new metaphysics that explores how all things – from atoms to green chillies, cotton to computers – interact with, perceive, and experience one another’ (Bogost, 2012).
11 See for instance Rueda and Lara (2020) ‘Virtual Reality and Empathy Enhancement: Ethical Aspects’ as an example of good work that speaks about ‘fostering empathy-related abilities through virtual embodiment in avatars’.
12 I have reflected on the connection of VR and environmental awareness in greater length before in the article ‘Mystical-Type Experience at The Virtual Reality Interface: Technics, Aesthetics, and Theology in the Search for Cosmic Connection’ (Strutt, 2021).

References

Arts Council England (2019), Sustaining Great Art and Culture, Environmental Report. Available at: www.artscouncil.org.uk/sustain ing-great-art-and-culture-environmental-report-201819 (accessed 9 January 2023).
Ascott, Roy (1990), Is there love in the telematic embrace? Art Journal 49.3, 241247. DOI: 10.2307/777114
Ascott, Roy (2008), Cybernetic, technoetic, syncretic: The prospect for artLeonardo 41.3, 12. DOI: 10.1162/leon.2008.41.3.204
Birringer, Johannes (2008), Performance, Technology, and Science. New York: PAJ.
Bogost, Ian (2012), Alien Phenomenology, or, What it’s Like to Be a Thing. Minneapolis: University of Minnesota Press.
Chalmers, David (2022), Reality+: Virtual Worlds and the Problems of Philosophy. London: Allen Lane.
Chua, Eu Jin (2006), Laurie Anderson’s Telepresence, Postmodern Culture 16.2. Available at: pmc.iath.virginia.edu/issue.106/16.2chua.html (accessed 7 December 2023).
Cîrstea, Ana-Maria and Natasha Mutebi (2022), UK Parliament POSTnote 669. Available at: https://researchbriefings.files.parliament.uk/docume nts/POST-PN-0669/POST-PN-0669.pdf (accessed 5 January 2023).
Erskine, R. G. (1998), Attunement and involvement: Therapeutic responses to relational needs. International Journal of Psychotherapy 3.3, 235244.
Floch, Johan (2021), Foreword. In M. Fol, Virtualised dance? Digital shifts in artistic practices. EDN. Available at: www.ednetwork.eu/news/edn-publication-virtualised-dance (accessed 9 January 2023).
Giges, Bob and Edward C. Warburton (2010) From router to front row: Lubricious transfer and the aesthetics of telematic performance. Leonardo 43.1, 2432. Available at: https://muse.jhu.edu/article/372 222 (accessed 7 April 2023).
Hockey, Tia May (2021), Personal interview.
Khoo, Mavin (2021, 2022), Personal interviews and Q&A conversation in showcase ‘Virtual Touch, Virtually Dancing’. Available at: www. mocapstreamer.live (accessed 9 January 2023).
Kurta, Leah and Jonathan Freeman (2022), Targeting IMPACT: a new psychological model of user experience. International Conference on Human–Computer Interaction. Cham: Springer, pp. 196212. DOI: 10.1007/978–3-031–05637–6_12
Marks, Laura U. (2020), Talisman-images: From the cosmos to your body. In Radek Przedpełski and Steven Elliot Wilmer (eds), Deleuze, Guattari, and the Art of Multiplicity. Edinburgh: Edinburgh University Press, pp. 231260.
Marks, Laura U. (2021), Tackling the carbon footprint of streaming media. Available at project site www.sfu.ca/sca/projects---activities/streaming-carbon-footprint.html; ‘Streaming Carbon Footprint’ and ‘Call for work, Second Annual Small File Media Festival’ blog entries at www.sfu.ca/~lmarks/blog/ (accessed 9 January 2023).
Nesta & i2 Media Research (2018), Evaluating Immersive User Experience and Audience Impact. A report produced for Digital Catapult. Available at: www.immerseuk.org/wp-content/uploads/2018/07/Evaluating_Immersive_User_Experience_and_Audience_Impact.pdf (accessed 9 January 2023).
Obringer, Renee, et al. (2021), The overlooked environmental footprint of increasing Internet use. Resources, Conservation & Recycling 167. DOI: 10.1016/j.resconrec.2020.105389
Rueda, Jon and Francisco Lara (2020), Virtual reality and empathy enhancement: Ethical aspects. Frontiers in Robotics and AI 7. DOI: 10.3389/frobt.2020.506984
Shanken, Edward A. (2000), Tele-agency: Telematics, telerobotics, and the art of meaning. Art Journal 59.2, 6477. DOI: 10.1080/00043249.2000.10791997
Skolczylas, Natalia (2020), Climate Action and the Performing Arts. Report from the IETM Galway Satellite Meeting. Available at: www.ietm.org/system/files/publications/ietm_report_galway_satellite.pdf (accessed 9 January 2023).
Strutt, Daniel (2022), Motion capture and the digital dance aesthetic: Using inertial sensor motion tracking for devising and producing contemporary dance performance. In Carla Fernandes, Vito Evola and Cláudia Ribeiro (eds), Dance Data, Cognition, and Multimodal Communication. London: Routledge.
Whitley, Alexander (2021, 2022), Personal interview and Q&A conversation in showcase ‘Virtual Touch, Virtually Dancing’. Available at: www.mocapstreamer.live (accessed 9 January 2023)
  • Collapse
  • Expand

All of MUP's digital content including Open Access books and journals is now available on manchesterhive.

 

Metrics

All Time Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 511 446 91
PDF Downloads 71 46 4