In the last 12 years, several developments and convergences in motion capture (mocap) and immersive media technology have seen the synthesis of new and hybrid systems for embodied, extended and augmented digital performance.¹ In the same way in which ‘camcorder’ home video recording and playback became affordable in the 1970s and 1980s and subsequently revolutionised many arts and performance practices to become ‘multimedia’, the recent waves of hardware and software tools have quickly come down in price, such that they have become accessible to small companies and even enthusiastic individuals.² More recently, the conditions surrounding the various coronavirus (COVID-19) pandemic lockdowns – with artists frustrated by both the impossibility of sharing analogue/physical spaces, and the two-dimensional limitations of videoconferencing for sharing work and practising – generated a clear and decisive turning-point in immersive tech production practices, both in their take-up within the performing arts and in wider cultural engagement with a discourse about the kinds of artistic expression that ‘Metaverse’-type virtual performance systems can afford.³

Around the same time as the onset of the pandemic in 2020, a network of academics, dancers and creative technologists based at Goldsmiths, University of London, began to think about the kinds of existing technologies that could be drawn together to build a system for meaningful remote dance interaction. We offered a concept of a system for the streaming of motion capture data, asking could we bring three-dimensional dance movement from remote performers around the world into a connected virtual space, live and in real time, as a legitimate form of dance practice? We developed a simple-to-use, open-source software tool named ‘Goldsmiths Mocap Streamer’, which could grab motion capture data locally and send it to any other computer in the world to be rendered within a real-time graphics engine (Figure 2.1), or indeed within a cloud-based Metaverse provider. Our team then used this framework to address experimentally a set of critical aesthetic and technical questions. What kinds of meaningful movement and dance interaction and communication can be generated remotely through the use of immersive tech? What kinds of alternative or augmented aesthetics of dance interaction could be explored in shared virtual spaces, where the dancer’s avatar body does not even need to be fully ‘corporeal’? What are the limitations of, and obstacles to, this kind of remote collaborative dance practice in terms of connection, interaction and sense of presence?

Figure 2.1Screenshot from the second Goldsmiths Mocap Streamer showcase performance ‘Dancing in the Metaverse’, November 2021. Two performers in the UK (Jack Thomson, Hannah Burfield) were able to dance in real-time with another dancer in Brooklyn, New York (Kristia Morabito) (image author’s own).

These original research questions are now complemented by an awareness of the need, in the current state of climate emergency, to clearly connect innovation ideals with actual societal needs. At the end of 2020, the meeting of the International Network for Contemporary Performing Arts (IETM) set an agenda for ‘greening the performing arts’ that was positioned both within the ongoing climate emergency, and within the pandemic itself. Their report, Climate Action and the Performing Arts, evoked the landscape in which it was written:

This vivid picture of a cessation of normal polluting activity invites a reconsideration of the current Zeitgeist moment in performance tech (and Metaverse hype) through a more socially pragmatic lens, framing it through the possibility of mitigating the need for environmentally damaging touring practices. In addition, could this moment also instigate the lowering of barriers to access into collaborative digital spaces, so that people, regardless of geography, disability or social status, can come together over great distances to move and create with a convincing sense of embodied presence, interaction and communication? We now move to ask: could ‘dancing in the metaverse’ come to be seen, not only as legitimate, but even as the ethical form of future dance performance?

As we have discovered in our research, technical, aesthetic and environmental innovation goals are not necessarily in opposition to each other. In contemplating the historical lineage of remote, networked, distributed or telematic performance practice, developing as a form since the 1960s through the work of artists such as Roy Ascott, Paul Sermon (see Chapter 3 in this volume), Laurie Anderson, Joan Jonas and ORLAN, we can see that practical concerns around embodied presence and liveness, collaboration and interaction have often coexisted with more philosophical questions of conscious evolution, empathy and ethics. This has usually involved speculative and utopian ideas around altered states of extended embodiment, what it means to be ‘together’ or virtually present, and of harmony and care for the other. However, this philosophical approach to immersive networked technology may also frame solutions to ecological and other societal concerns. Such ethical practice might generate an understanding about the role of the performing arts in future visions of digital society, and of a social and cultural metaverse.

In this chapter I work towards a genealogically supported vision of telematic dance and performance work as driving the sort of innovation that might address some of these ecological and other societal concerns. To do this I draw together some historical and then institutional discourses before turning to a model of analysis of ‘user’ experience, to frame the research of Goldsmiths Mocap Streamer through the words of the dancers themselves. In doing this, I approach the speculative ‘promise of the digital’ through not only short-term technical and aesthetic innovation concerns, but also through affective, ethical and ‘technoetic’ philosophical frameworks.⁴

From telematic to metaverse futurity

In 1966, cybernetic visionary and artist Roy Ascott

In what is effectively an alternative origin for the type of shared virtual space now dubbed the ‘metaverse’, Ascott imagined a non-linear, processual interaction between participant and artwork that ‘is not primarily a thing, an object, but a set of behaviours, system, actually a system of systems’ (1990: 243). He emphasised the emotional and embodied nature of this vision, in a relationship of mutual becoming – where the artwork ‘embraces the viewer in the creation of meaning’ (1990: 242) in a ‘spiritual interchange’ imbued with a sense of intimacy and affection (1990: 247).

First framed by cybernetic theory, Ascott’s artworks and writings later became primarily positioned through the idea of the ‘telematic’, a term introduced in 1978 by Simon Nora and Alain Minc (Shanken, 2000). As a concept, ‘telematic’ simply refers to remote subjects brought together through computer-mediated communication. But even early on, it gained, particularly for Ascott, a decidedly utopian nuance, regarding the evolution of human consciousness:

While aspects of this utopianism seem somewhat anachronistic today, much of Ascott’s telematic vision does still remain relevant at a time when tangible interfaces are disappearing and we are becoming increasingly immersed in virtual worlds. Ascott offered a cyborg vision of conscious fusion with the computer that clearly suggests itself to new immersive technologies: ‘As we come to see more, we shall see the computer less and less. It will become invisible in its immanence, but its presence will be palpable to the artist engaged telematically in the world process of autopoiesis, planetary self-creation’ (Ascott, 1990: 247).

Current metaverse visions can be seen as simply the next iteration of this same idealistic imperative, now framed by the convergence of new VR and XR interfaces, real-time generative graphics and motion capture systems. However, it should be noted that when we arrive at current simulation technologies, a decisively new dynamic relationship between felt presence and embodiment emerges. Performance theorist and practitioner Johannes Birringer describes this as a ‘historical shift … between video dance (choreography for the camera) and interactions systems’, described as ‘sensing systems worn on the body which generate data in real time or produce data that can be digitally processed and transcoded’ (2008: 25). We now see simulation computation rendering complex models in real time and on immersive interfaces, with open-ended, generative images forming in response to participants’ actions. This, for Birringer, yields a significant disruption of previous practices, even a new ontological landscape:

However, while this technology is no doubt disruptive to the structures of representation that have developed through media technology over the last 150 years, and while it may indeed shift our understanding of humanity, in a more pragmatic sense can we truly say that it addresses our contemporary and urgent real-world needs?

Contemporary concerns and realities

What twenty-first-century developments in virtual and immersive media mean in relation to ‘physical’ reality, consciousness and experience is hotly debated.⁵ The quasi-spiritual futurist vision of a ‘technoetic’ evolution of consciousness that Roy Ascott still pursues to this day can seem naïve when tech corporations manipulate global democracy through the harvesting of sensor data, and when fresh hardware with built-in obsolescence is released daily while a climate disaster looms (Chua, 2006: notes 12–14). What seems called for is a more context-specific and pragmatic approach to assessing the value of metaverse technologies, including our own Mocap Streamer tool, within our present-day eco-political context. This might mean exploring whether the new immersive technologies for remote collaboration and performance offer a viable alternative to touring that might help address the climate impact of the performing arts. While lockdowns rather forced the issue of maintaining arts and culture in online forms, can we now consider the more long-term ecological benefits and potentialities of this shift to virtualised media arts?

The environmental sustainability of arts, culture and the performing arts was an issue that was being institutionally foregrounded prior to the pandemic. The 2019 Arts Council England Sustaining Great Art and Culture Report acknowledged that ‘culture has a carbon footprint, and data matters … There should be no opposition between what art is and how art does: reducing emissions prompts creativity and activism, and vice versa’ (ACE, 2019). This report offered figures on the most common impacts that can be meaningfully tracked – of energy, water, waste, business travel and touring – across 747 arts and culture organisations in England. Travel and touring were found to equate to 114,547 tonnes of CO₂, with dance and theatre together generating about 20% of this. While this damaging activity pretty much ceased during the lockdowns, at the time of writing, and in our haste and desperation to get back to ‘normal’ regardless of how damaging that ‘normal’ might have been, we seem to have backslid in terms of progressive environmental policy without learning any lessons from the pandemic.

The 2020 IETM report gathered many ideas, suggestions, tools and experiences that considered the role of performing arts in addressing climate change.⁶ The report includes a section on ‘the promise of the digital’ which asks a set of technical and aesthetic questions to frame the potential of digitally streamed performance, motion capture and gaming tech, for sustainability and reduced environmental impact:

It is worth noting that even these digital modes of access to performance through streaming and cloud computing are not carbon neutral of course, and in fact got progressively more polluting during the pandemic. Media philosopher and activist Laura Marks, creator of the Small File Media Festival (which challenges the need for high-definition media images), speaks to the real impact of our media streaming culture:

The main impact of this activity is from the power consumption of media streaming giants such as YouTube and Netflix, where the pursuit of ever higher quality images directly compounds carbon output. Videoconferencing, due to the standard lower resolutions, pollutes considerably less than streaming video. Furthermore, the streaming of motion capture data rather than of whole images, with tiny packages of lightweight data, is less energy consuming than this again.⁷ This does not mean, however, that our own Goldsmiths Mocap Streamer tool, and this mode of virtual production more broadly, is without impact. We still extensively use video conferencing platforms adjunct to the streamer itself, to meet, plan, share the image back to remote dancers and provide a simulcast video feed of the physical dancer(s) for audiences (to allow them to understand the performance’s liveness). We must acknowledge that technological innovation in the performing arts is not simply a quick solution to climate realities. The promise of the digital, as much as it is about technical and aesthetic innovation, must also be guided by an ethical approach of moderation and sensitivity towards ecology, embedded in day-to-day practices, that weighs audience experience and artistic aspiration against environmental impact. We can, however, think about how this sensitivity can catalyse innovation, creativity and activism in ways that might allow the arts to shape new technologies for social good, rather than merely adopting them.

Goldsmiths Mocap Streamer: User experience

It is perhaps easy to accept in principle that the new metaverse-type iterations of telematic performance work are a good thing, in terms of their potential not only to connect performers and audiences remotely but also to diminish the climate impact of the performing arts. However, in the same vein as the stated goal of the Small File Media Festival – ‘Movies don’t have to be big to be bingeworthy!’ (Marks 2021, section ‘Call for work, Second Annual Small File Media Festival’) – can we yet say that good live performance doesn’t require the physical presence of either audience or performers? Or confidently state that we can deliver experiences which are aesthetically satisfying, and no longer merely a substitute for live theatrical performing arts?

Much recent research has turned to audience research to address these questions. A recent UK government policy POSTnote (May 2022) acknowledged that, while there is a relative abundance of quantitative research ‘mostly focusing on audiences or organisations’, … ‘there is limited evidence on how performers and artists are affected by digital technology’ (Cîrstea and Mutebi, 2022: 4). Acknowledging this dearth of research on how performers actually feel about digital technologies within their disciplinary practice, it is perhaps also key to note, as performance theorist Johannes Birringer does (2008: 35), that in interactive performance systems, the distinction between audience and performer, or between ‘viewer’, ‘participant’, and ‘user’ can become complexified, or even break down. Understanding the new kinds of interactions, actors and interfaces within non-linear, interactive performative events can mean that it becomes difficult to ascertain what ‘audience’ even means, as all participants effectively become co-creators.

In what follows, I seek to explore this convergence of experience by bringing a structured model of audience research to my own qualitative and often informal research into performer experience. I focus on the experience of our project’s collaborating practitioners, who have led or participated in the workshops and showcases in which we have connected dancers from Singapore, New York and Hong Kong to those in the UK. In these sessions, the dancers wore Noitom ‘Perception Neuron’ inertial motion sensor (IMU) capture kits, allowing them to move and interact in a virtual landscape in real time. The virtual performance spaces were generated by real-time games engines Unity and Unreal Engine, with live interaction design through sets of digital tools such as shaders, colliders, and GPU particle effect. These tools permit reactions between avatars and objects, and give dancers the feeling of virtual touch, proximity and gravity, that allows a sense of kinesthetic embodiment within the virtual scene. More than this, it allows a kind of augmentation of both kinetic force and proprioception which can both extend and distort normative modes of dance embodiment (Figure 2.2). Throughout the research we asked dancers about how this mode of practice feels, the kinds of obstacles that they face both cognitively and physically, and if they can see themselves productively and creatively engaging with these systems in the future.

Figure 2.2Dancers Alexander Whitley, Tia May Hockey and Nicola Henshall dance together in May 2021 in a scene called ‘Bubblegum’ – designed to explore altered feelings of embodiment and proximity in the virtual environment (image author’s own).

The IMPACT audience research model

In 2018, Nesta and i2 Media Research group released a report on Evaluating Immersive User Experience and Audience Impact, which drew attention to the ‘challenges of understanding the value of creative content while … traditional quantitative measures of measuring impact are not always available or reliable’ (Nesta and i2, 2018: 1). They proposed to ‘develop a research methodology for testing and evaluating the experimental immersive content that is being made now, in a way that will help us predict the potential audience appetite, cultural impact, and commercial opportunity in the future’. This report identified some key predictors of experience in immersive experiences through concepts of positive affect, engagement, and sense of presence, drawn from psychological models which render these states quantifiable. i2 Research then honed their own model as a toolkit for both the creation and evaluation of immersive experiences, offering what they call quantifiable ‘high level categories’ as ‘helpful thinking aids to stimulate conversation and deliberation during design sprints’ (Kurta and Freeman, 2022: 209).⁸ These categories of experience, or ‘states’, are offered with the acronym IMPACT (Kurta and Freeman, 2022: 200):

• Interesting: Capturing initial interest through automatic and personalised attentional cues.
• Meaningful: Connecting to user values and cultural or universal goals. Meaning goes beyond initial interest.
• Personalised: Personalising design features. Personalisation focuses on individual relevance, e.g. achieves personal ‘to be goals’. [sic]
• Affective: Emotionally arousing. The affective dimension acknowledges how changes in arousal level are important for maintaining interest and engagement.
• Collective: Enabling social experience or ability to connect and share with others. Features may connect individually or demonstrate collective action or community.
• Transportive: Feeling presence, sustained attentional engagement. The transportive dimension guides design which resolves user friction and minimises disengagement via distraction.

While these categories are deduced as insights into the key domains of audience experience, they suggest themselves as useful modes of thinking about the relative quality or success also from the performer’s point of view. Below, I adopt the IMPACT model as a thinking aid to reflect in a structured way on the experience of our own project’s dancers and choreographers in their collaborative practice with our 3D interaction designers and creative technologist. This interview material was gathered through question-and-answer sessions and personal interviews with some of the dancers, choreographers and project partners that we worked with in the course of the project. The dancers quoted here are Mavin Khoo, Creative Associate of Akram Khan Company, and Alexander Whitley and Tia May Hockey of Alexander Whitley Dance Company.

The performer’s experience

 Interesting/meaningful

The first two IMPACT states of ‘interesting’ and ‘meaningful’ are intimately connected. The first ‘captures initial interest’ through a measure of novelty. The second is concerned with sustaining interest and attention through embedding technological innovations within a meaningful structure (Kurta and Freeman, 2022). The i2 researchers note that the ‘psychological impact of content can be “broken” or diminished by poor technology implementation’ (Nesta and i2, 2018: 10), referring to the possibility of tech failure or a glitch that would draw the participant out of the experience. They also note how perceived gimmickry may cause the same ‘break’, as the initial point of interest proves to be depthless. Mavin Khoo spoke eloquently on this point in relation to his company’s storytelling ethos:

In conversation with the author, Khoo expanded on this point of striving to avoid gimmickry. Having just extensively used a form of motion capture animation (rotoscoped) in Akram Khan Company’s production of The Jungle Book (2022), he mused:

In many ways, Khoo’s issue of the technology being a ‘poetic collaborator’ became the guiding mantra throughout our research. We are acutely aware that the technology must serve the intention of the dancer if it is to become embedded meaningfully within practice. On this point, Khoo had some critical thoughts about some of our 3D avatar and interaction design (see Figure 2.3).

Figure 2.3Dancers Jack Thomson and Hannah Burfield in a performance designed as a dance battle called ‘Bird of Prey’. With complex gravity, inertial and elaborate costume effects built into the scene, Mavin Khoo felt that this design did not adequately serve the dance (image author’s own).

Khoo’s comment highlights how important it is that the whole team, from software developer to animator, to dancer, and then, ultimately, to the audience, focuses on the poetics of the performance, which needs to lead (if not limit) the technological marvel. There is, then, a magical moment when this human/technology collaboration clicks. As dancer Tia May Hockey observed:

Khoo and Hockey’s comments give insight into the way that, from the perspective of both dancer and audience, the technical and aesthetic innovation of the work must be in the service of artistic expression and poetry. To be interesting and meaningful it must mesh with both the intention of the performer and the broader discipline in which they work – to generate meaning (embedded within a dramaturgical framework or ‘journey’), to exhibit a disciplined artistry and to ‘allow the work to speak’.

 Personalised

For dancers within the mocap streaming framework, the question of personalisation as ‘individual relevance’ seems, to a large extent, to come down to the feeling of embodiment within the avatar figure. Our dance partner Alexander Whitley spoke about the negotiation involved in achieving this sense of personal investment with the virtual body, particularly when the expression of the body is not exactly figurative:

It is this process of ‘taking on’ and internalising the traits of these avatars that seems to enhance the personalisation of the experience, where one starts to identify, and to feel and act differently. In a second interview, Whitley explained:

This real-time feedback, and the practice of feeling-out the kinaesthetic possibilities of the avatar, are a process through which we come to ‘recognise’ it as an extension of self. Moreover, this is a modified self, structured by a set of intentions and concepts made ‘flesh’ and with which the dancer can explore alternative or augmented subjective and personal experiences of embodiment.

 Affective

The affective dimension of experience seems to follow from and extend this sense of personalised embodied investment in the avatar and its potentialities into a deeper emotional sense of ‘presence’, such that the dancer can feel that they are expressing something of their interiority through it: to ‘reveal ideas of choreographic intention or performance intention through the kind of visuals that are generated’ (Whitley, 2022). For Whitley, this kind of ‘deep’ connection is really the purpose of continuing to engage and experiment in this realm. He states, ‘I guess facilitating those more deeply embodied experiences through this kind of technology is what really excites me. And it obviously presents some great solutions to the challenges we’ve been experiencing over the last year [of the pandemic]’ (Whitley, 2021).

For Khoo, however, there were always reservations. He confessed, early in our research process in 2019: ‘I was genuinely concerned that the presence of motion capture in my process would be problematic in generating a kind of emotive human ownership to the work’ (Khoo, 2022, in Strutt, 2022). Later, he again noted the aesthetic and design element as a major factor in being able to achieve this sense of investment: ‘whatever the creation of that avatar is, if you like the nature of that, plays a huge importance in terms of how far one is able to invest in it from an emotional perspective’ (Khoo, 2022). He describes:

Our research attests that it is certainly possible to experience a kind of affective immersion or ‘merging’ with the system through an iterative process of exploration and experimentation. While Khoo and Whitley’s remarks show how this ‘merger’ is initiated through a dancer’s exploration of their own avatar form, one of the most significant findings of our research is how emotional ownership is dramatically heightened when the participant starts to move and ‘merge’ with other avatar performers within the remote streaming ‘metaverse’ framework. Ironically perhaps, it is in the displacement of the attention on self, through focusing on moving with the other avatar figures, that the potent, transcendent affect of ‘deep embodiment’ seems to occur. This collective sharing of experience, with real-time sensory feedback from a partner, enhances the sensation of deep kinaesthetic embodiment within the avatar form.

 Collective

Fitting neatly into the IMPACT model’s categorisation of collective experience, the primary purpose for the development of our own mocap streamer tool from the outset has been to ‘enabl[e]‌ social experience or the ability to connect and share with others’ (Kurta and Freeman, 2022: 200). The dancers’ reflections then show how the collective aspect of the work takes on an even more affective and almost ethical nuance of ‘taking responsibility’ for the remotely connected other. Khoo described: ‘There’s a sense for the dancers of having to first take responsibility in terms of what their own potential is, and then of course the minute there is the engagement with the second or third avatars, to take responsibility for the others as well’ (Khoo, 2021).

Of course, this is something that would happen in the physical studio quite normally. However, as Whitley observes, the remoteness of the streaming tool is not too much of an obstacle to this, though there is a need to develop a set of task-based, semi-improvisational techniques to lean into this sensitivity, honing attunement with the rhythm, intention and nuanced expressivity of the other, remote, dancers through their avatars.⁹

Tia May Hockey adds nuance to this idea of responsivity, and responsibility, in an interesting analogy between the avatar and an animal (Figure 2.4):

Figure 2.4Nicola Henshall and Tia May Hockey explored a hide and seek dynamic in a fantasy woodland scene in May 2021 (image author’s own).

There is a complexity to this work that a dancer, perhaps intuitively, understands more than most, due to their already disciplined practice of sensing, anticipating, predicting and responding to the movement of another. However, what we have noted is that there is implicit ethical inflection to these practices when the ‘other’ is remote and not even within their own body. We could perhaps call this an ‘alien phenomenology’ which can yield transformative experiences of connection and interaction through exploration of alternative forms and ‘augmented’ embodiments.¹⁰

 Transportive

With her expression of the experience of ‘connection through to her being’, Tia May Hockey nicely leads into the IMPACT model’s notion of the transportive as a sense of flow or deep unbroken attentiveness. In the same interview, she explained:

Similarly, Khoo noted that there is a kind of altered state that needs to be reached, where ‘psychologically the dancer has to come to a space where they are able to let go of what their habitual pattern would be’. This led him to reflect that ‘[y]‌ou get quite easily in this state like “I’m in another realm and so much new is happening”. … It’s so interesting and there is something in that psychology, I think, that is actually bigger than we consider’ (Khoo, 2022). It seems that here, in the collective moment that is facilitated by the motion capture streamer as much as by the elements of avatar and interaction design, there can be an intense and transportive sense of almost ‘out-of-body’ engagement that fixes our attention on our attunement and sensitivity to the other figures. This, as Khoo notes in terms that seem close to Roy Ascott’s utopian dreams of global harmonisation, is perhaps ‘actually bigger than we consider’.

Conclusion

This chapter set out to question whether, within our current moment, we are truly at a point of sufficient technological advancement in which remote connection can become a legitimate, mainstream and carbon-conscious alternative medium for dance practice and performance. It is clear, however, that there are no quick or direct fixes in terms of technological or aesthetic innovation. We need to look a bit deeper, and more speculatively, at the ‘promise of the digital’ (here, specifically, of metaverse-adjacent tech) as a performance medium that can meaningfully connect people toward shared experience. Indeed, this is not a new critical approach, but one that is broadly in line with a longer genealogy of telematic connection in the arts.

Though in one way these questions about consciousness and rationality might still seem a little disconnected from the clear and present climate emergency, there is a burgeoning literature on how this more philosophical approach does in fact speak to these concerns. While there is a fair amount of good critical work on VR and immersive media as an ethical ‘empathy machine’,¹¹ it is particularly interesting to think about Laura Marks’ recent work,¹² where she speaks of a ‘talismanic image’ that cultivates an active form of sensory engagement with images, and which makes us palpably feel the ‘interconnectedness’ of things. She says:

She asserts that this kind of image can not only alert us to our responsibility to other entities, but also stimulate activism and action: ‘When images make contact with us, we recognise that we are strung in webs of causality. By cultivating affective response and indexical awareness, we can draw the universe close, even rearrange it’ (Marks, 2020: 255).

Ending on this point, I suggest, drawing Roy Ascott close to Laura Marks, that the promise of ‘dance in the metaverse’ is not only that it might just mitigate the carbon footprint of the performing arts, but moreover that telematic performance has the potential to cultivate structures of empathy and awareness in ways that are not just abstract messages. Rather they are embodied and connected practices and processes which might lead into action (the modification of existing damaging behaviours) and even activism (spreading the word and creating new kinds of content). Returning to Tia May Hockey’s words, ‘with shared intentions, heightened awareness, conscious effort and sensitivity’, we might get there yet.