More details on IWMM keynote and panel of experts from academia and industry are pending.
Tuesday, 1 October 2024
Poster Session | 10:00 - 10:55 and 15:00 - 15:55 Local Time | Lobby
Ethical Considerations in the Production and Consumption of Music in the Metaverse
Author(s): Gareth W. Young
Abstract
Volumetric music videos (VMVs) transform immersive entertainment to create 3D musical performances using volumetric video technology. These videos, accessible via 2D screens or extended reality (XR) platforms, offer novel interactive experiences in the interconnected worlds of the metaverse. As VMVs rapidly integrate into immersive virtual environments (IVEs), they raise critical ethical concerns about data privacy, content authenticity, and user well-being. Here, we show that addressing these ethical issues is crucial for the responsible development of VMVs. Our findings reveal that robust privacy protection, accurate representation, and inclusive design are essential to safeguard user rights and promote a positive user experience. This study builds upon previous work and highlights the need for industry-wide standards and collaborative efforts to ensure ethical practices in VMV production and consumption, fostering trust and sustainability in the emergent metaverse.
Immersive Io3MT Environments: Design Guidelines, Use Cases and Future Directions
Author(s): Rômulo Vieira, Shu Wei, Thomas Röggla, Débora C. Muchaluat-Saade and Pablo César
Abstract
The Internet of Multisensory, Multimedia, and Musical Things (Io3MT) is an emerging field at the intersection of computer science, arts, and humanities. It focuses on integrating technologies and datasets to explore human sensory perception, multimedia elements, and musical compositions for artistic and entertainment purposes.
This position paper advocates for merging Io3MT with Extended Reality (XR) in creative contexts. Through literature review and expert focus group discussions, we identify five key design guidelines for developing immersive environments for artistic creation within the Io3MT framework. We developed PhysioDrum as a practical demonstration of these guidelines. This work provides insights into the infrastructure, tools, opportunities, and research challenges arising from integrating Io3MT and XR.
Remote Orchestral Conduction via a Virtual Reality System
Author(s): Leonardo Severi, Matteo Sacchetto, Andrea Bianco, Cristina Rottondi, Gabriele Abbate, Antonio Paolillo and Alessandro Giusti
Abstract
This paper envisions the adoption of a Virtual Reality (VR)-based approach to provide visual feedback to remote performers without the acquisition and transmission of a video stream, in a Networked Music Performance scenario. Focusing on a remote orchestral conduction setup, the VR headset tracks the conductor's gestures to convey them to remotely connected musicians, where the conductor's hands and head pose are displayed by means of an avatar. Quantitative results suggest that the Motion-to-Photon latency introduced by the system is tolerable for NMP applications, while the proposed system achieves a substantial reduction of bit-rate requirements in comparison to traditional video streaming.
Paper Session | 11:00 - 12:30 Local Time | Main Room
“It Takes Two” - Shared and Collaborative Virtual Musical Instruments in the Musical Metaverse
Author(s): Alberto Boem, Damian Dziwis, Matteo Tomasetti, Sascha Etezazi and Luca Turchet
Abstract
The relevance and technical possibilities of Shared Virtual Environments (SVEs) are constantly growing as part of what is known as the Metaverse. This includes software and web platforms for creating SVEs and the availability of hardware for experiencing these environments. SVEs offer many possibilities and unique capabilities that have yet to be explored. One aspect is music. In this paper we explore the concept of networked Virtual Musical Instruments (VMIs) for the Musical Metaverse, where virtual spaces are specifically designed for musical collaboration and social interaction.
We describe three prototypes for shared, collaborative VMIs that incorporate specific features of SVEs, such as spatial audio, data sonification, and embodied avatar-based interactions. We conducted a user study to investigate how these instruments can support creativity and usability and to what extent they can deliver a sense of social presence and mutual engagement between users. Finally, we critically discuss how the three implementations of the proposed shared and collaborative instruments provide novel avenues for music-making in the Metaverse. Our results show that our instruments exhibit varying degrees of creativity and usability. However, instruments that employ symmetrical and embodied interactions better support social presence and interdependence among users.
Using Web Audio Modules for Immersive Audio Collaboration
Author(s): Michel Buffa, Dorian Girard and Ayoub Hofr
Abstract
Web standards established by the W3C now allow the design of applications unthinkable just three or four years ago. In 2020, the Web Audio plug-in standard (aka "VSTs for the web") called "Web Audio Modules V2" or "WAM" was a major contribution to the world of computer music on the web. In addition to the collaborative aspects naturally associated with web applications, Web Audio Modules have enabled the creation of host applications capable of exploiting reusable web components such as note generators, virtual instruments and audio effects. Today, the WAM ecosystem offers over a hundred plugins of all types via the WAM Community initiative, online tutorials and several host applications. One of these applications, Sequencer Party, is a real-time collaborative audio/visual platform built entirely from WAMs.
In this position paper the concepts of sequencer.party have been transposed into a web application for the Musical Metaverse. By demonstrating that it is possible to reuse existing WAMs without modifying them, by automatically generating, through introspection, a new 3D graphical interface adapted to VR/XR manipulation, or, in some cases, by manually recreating a new dedicated interface, the potential of the WAM standard, usually confined to traditional 2D web applications, is clearly highlighted. For the 3D rendering of the virtual world, the BabylonJS framework was used, which provides strong support for WebXR. We also choose a CRDT approach for “state synchronization” between clients. The software is open source and hosted online.
Key considerations of duo singing in virtual reality and videoconferencing: A mixed methods study
Author(s): Ben Loveridge
Abstract
This study explored the key considerations of duo singing in virtual reality (VR) and videoconferencing (VC) settings. The COVID-19 pandemic significantly impacted in-person musical collaborations, pushing performers to seek online alternatives. While VR may offer a novel visual medium for musical collaboration, limited research has examined its affordances alongside VC in networked music performance contexts. This mixed-methods study involved ten participants from Australia and the United States, equipped with internet access and VR headsets. Participants engaged in singing sessions in both VR (Bigscreen) and VC (Zoom) environments, followed by semi-structured interviews and online questionnaires. Thematic analysis revealed five key themes: presence, performance, affect, usability, and usefulness. Findings indicated that participants felt more co-present and less inhibited in VR, leading to a greater sense of fun and creativity, despite challenges with the fidelity of facial expressions and physical comfort. VC, with its more detailed and realistic visual cues, facilitated better timing and synchronization but provoked self-consciousness among less experienced singers. Overall, participants expressed a preference for singing in VR, citing the novelty and reduced inhibition. These results suggest that platform suitability for online singing is context-dependent, with stylised VR being appropriate for casual collaboration and VC useful for performances requiring more realistic visual fidelity. This study broadens our understanding of visual communication in online music collaboration, emphasising the role of self-awareness and inhibition.
The Musical Metaverse: Advancements and Applications in Networked Immersive Audio
Author(s): Claudia Rinaldi and Carlo Centofanti
Abstract
The convergence of the Sound and Music Computing (SMC) domain with the Internet of Things (IoT) paradigm has given rise to the burgeoning field of the Internet of Sounds (IoS). Within this context, the concept of the Musical Metaverse emerges as a pivotal area of exploration, offering unprecedented opportunities for immersive audio experiences and networked musical interactions. This paper provides a comprehensive examination of the Musical Metaverse, delving into the technological innovations, methodologies, and applications that underpin this dynamic domain.
We explore the integration of networked immersive audio technologies, including the use of Web Audio for real-time, interactive sound experiences, and the role of Networked Musical Extended Reality (XR) in creating collaborative virtual environments for music performance and education. The paper highlights the significance of semantic audio applications, intelligent music production, and participatory live music performances within the IoS framework.
Furthermore, we discuss the advancements in smart musical instruments and ubiquitous music, examining their implications for accessibility, inclusiveness, and sustainability in the digital music landscape. The challenges and solutions related to privacy, security, and the management of large audio databases are also addressed, emphasizing the need for robust protocols and exchange formats.
Our investigation extends to the potential of cloud-based services and wireless acoustic sensor networks in enhancing the efficiency and reach of musical and audio applications. Through the lens of ecoacoustics and sonification, we consider the environmental and ethical dimensions of the IoS.
By synthesizing current research and practical applications, this paper aims to provide a foundational understanding of the Musical Metaverse, offering insights into its future directions and potential impact on the global music industry.
Presence and Flow in Virtual and Mixed Realities for Music-Related Educational Settings
Author(s): Leonard Bruns, Benedict Saurbier, Tray Minh Voong and Michael Oehler
Abstract
Music in Extended Reality (XR) is increasingly present in both academic and industrial research. While XR applications are more prevalent in STEM education, there is growing interest in the use of XR for music education. This study investigates how multi-user XR scenarios affect perceived social presence and flow in music-related tasks, as both dimensions positively correlate with learning satisfaction and perceived learning effectiveness. Participants played interactive musical memory games using the multi-user applications PatchWorld and Fortnite under different Virtual and Mixed Reality conditions. Initial results indicate significant differences in presence but not in flow perception depending on the XR modality, providing valuable insights for the design of effective XR-based music teaching tools.
Holodeck: A Research Framework for Distributed Multimedia Concert Performances
Author(s): Andrea Genovese, Marta Gospodarek, Zack Nguyen, Robert Pahle, Agnieszka Roginska and Corinne Brenner
Abstract
This paper presents the Holodeck project, a multi-room research framework designed to support distributed multimedia concert performances. Developed through an inter-lab collaboration at New York University, the Holodeck platform leverages the Corelink engine, a flexible and unitary data routing system that facilitates immersive, interactive experiences across diverse networked environments. The framework enables real-time streaming and synchronization of various data types, including audio, video, and motion-capture data, thus supporting the design and implementation of augmented Network Music Performances (NMPs). Discussion around case studies of large-scale distributed concerts illustrate the system's capabilities and potential applications. In a pilot research exploration, the platform has been used to gather preliminary data on the quality of experience from both audiences and musicians. This research aims to enhance telepresence and realism in remote collaborations, contributing to the development of new methodologies and artistic practices in multimedia performances. The paper also discusses the technical challenges and solutions associated with implementing such a flexible and adaptive system and future work regarding the platform and its usage in multimedia immersive NMPs.
Lunch | 12:30 - 13:30 Local Time
Pieter-Jan Maes Keynote | 13:30 - 14:15 Local Time
Panel Discussion | 14:15 - 15:00 Local Time
Featuring experts from academia and industry: Michel Buffa, Cristina Rottondi, Gareth W. Young, Edo Fouilloux, Alberto Boem, and Pieter-Jan Maes