IEEE IWMM Program

Volumetric music videos (VMVs) transform immersive entertainment to create 3D musical performances using volumetric video technology. These videos, accessible via 2D screens or extended reality (XR) platforms, offer novel interactive experiences in the interconnected worlds of the metaverse. As VMVs rapidly integrate into immersive virtual environments (IVEs), they raise critical ethical concerns about data privacy, content authenticity, and user well-being. Here, we show that addressing these ethical issues is crucial for the responsible development of VMVs. Our findings reveal that robust privacy protection, accurate representation, and inclusive design are essential to safeguard user rights and promote a positive user experience. This study builds upon previous work and highlights the need for industry-wide standards and collaborative efforts to ensure ethical practices in VMV production and consumption, fostering trust and sustainability in the emergent metaverse.

The Internet of Multisensory, Multimedia, and Musical Things (Io3MT) is an emerging field at the intersection of computer science, arts, and humanities. It focuses on integrating technologies and datasets to explore human sensory perception, multimedia elements, and musical compositions for artistic and entertainment purposes.

This position paper advocates for merging Io3MT with Extended Reality (XR) in creative contexts. Through literature review and expert focus group discussions, we identify five key design guidelines for developing immersive environments for artistic creation within the Io3MT framework. We developed PhysioDrum as a practical demonstration of these guidelines. This work provides insights into the infrastructure, tools, opportunities, and research challenges arising from integrating Io3MT and XR.

This paper envisions the adoption of a Virtual Reality (VR)-based approach to provide visual feedback to remote performers without the acquisition and transmission of a video stream, in a Networked Music Performance scenario. Focusing on a remote orchestral conduction setup, the VR headset tracks the conductor's gestures to convey them to remotely connected musicians, where the conductor's hands and head pose are displayed by means of an avatar. Quantitative results suggest that the Motion-to-Photon latency introduced by the system is tolerable for NMP applications, while the proposed system achieves a substantial reduction of bit-rate requirements in comparison to traditional video streaming.

The relevance and technical possibilities of Shared Virtual Environments (SVEs) are constantly growing as part of what is known as the Metaverse. This includes software and web platforms for creating SVEs and the availability of hardware for experiencing these environments. SVEs offer many possibilities and unique capabilities that have yet to be explored. One aspect is music. In this paper we explore the concept of networked Virtual Musical Instruments (VMIs) for the Musical Metaverse, where virtual spaces are specifically designed for musical collaboration and social interaction.

We describe three prototypes for shared, collaborative VMIs that incorporate specific features of SVEs, such as spatial audio, data sonification, and embodied avatar-based interactions. We conducted a user study to investigate how these instruments can support creativity and usability and to what extent they can deliver a sense of social presence and mutual engagement between users. Finally, we critically discuss how the three implementations of the proposed shared and collaborative instruments provide novel avenues for music-making in the Metaverse. Our results show that our instruments exhibit varying degrees of creativity and usability. However, instruments that employ symmetrical and embodied interactions better support social presence and interdependence among users.

Web standards established by the W3C now allow the design of applications unthinkable just three or four years ago. In 2020, the Web Audio plug-in standard (aka "VSTs for the web") called "Web Audio Modules V2" or "WAM" was a major contribution to the world of computer music on the web. In addition to the collaborative aspects naturally associated with web applications, Web Audio Modules have enabled the creation of host applications capable of exploiting reusable web components such as note generators, virtual instruments and audio effects. Today, the WAM ecosystem offers over a hundred plugins of all types via the WAM Community initiative, online tutorials and several host applications. One of these applications, Sequencer Party, is a real-time collaborative audio/visual platform built entirely from WAMs.

In this position paper the concepts of sequencer.party have been transposed into a web application for the Musical Metaverse. By demonstrating that it is possible to reuse existing WAMs without modifying them, by automatically generating, through introspection, a new 3D graphical interface adapted to VR/XR manipulation, or, in some cases, by manually recreating a new dedicated interface, the potential of the WAM standard, usually confined to traditional 2D web applications, is clearly highlighted. For the 3D rendering of the virtual world, the BabylonJS framework was used, which provides strong support for WebXR. We also choose a CRDT approach for “state synchronization” between clients. The software is open source and hosted online.

This study explored the key considerations of duo singing in virtual reality (VR) and videoconferencing (VC) settings. The COVID-19 pandemic significantly impacted in-person musical collaborations, pushing performers to seek online alternatives. While VR may offer a novel visual medium for musical collaboration, limited research has examined its affordances alongside VC in networked music performance contexts. This mixed-methods study involved ten participants from Australia and the United States, equipped with internet access and VR headsets. Participants engaged in singing sessions in both VR (Bigscreen) and VC (Zoom) environments, followed by semi-structured interviews and online questionnaires. Thematic analysis revealed five key themes: presence, performance, affect, usability, and usefulness. Findings indicated that participants felt more co-present and less inhibited in VR, leading to a greater sense of fun and creativity, despite challenges with the fidelity of facial expressions and physical comfort. VC, with its more detailed and realistic visual cues, facilitated better timing and synchronization but provoked self-consciousness among less experienced singers. Overall, participants expressed a preference for singing in VR, citing the novelty and reduced inhibition. These results suggest that platform suitability for online singing is context-dependent, with stylised VR being appropriate for casual collaboration and VC useful for performances requiring more realistic visual fidelity. This study broadens our understanding of visual communication in online music collaboration, emphasising the role of self-awareness and inhibition.

The convergence of the Sound and Music Computing (SMC) domain with the Internet of Things (IoT) paradigm has given rise to the burgeoning field of the Internet of Sounds (IoS). Within this context, the concept of the Musical Metaverse emerges as a pivotal area of exploration, offering unprecedented opportunities for immersive audio experiences and networked musical interactions. This paper provides a comprehensive examination of the Musical Metaverse, delving into the technological innovations, methodologies, and applications that underpin this dynamic domain.

We explore the integration of networked immersive audio technologies, including the use of Web Audio for real-time, interactive sound experiences, and the role of Networked Musical Extended Reality (XR) in creating collaborative virtual environments for music performance and education. The paper highlights the significance of semantic audio applications, intelligent music production, and participatory live music performances within the IoS framework.

Furthermore, we discuss the advancements in smart musical instruments and ubiquitous music, examining their implications for accessibility, inclusiveness, and sustainability in the digital music landscape. The challenges and solutions related to privacy, security, and the management of large audio databases are also addressed, emphasizing the need for robust protocols and exchange formats.

Our investigation extends to the potential of cloud-based services and wireless acoustic sensor networks in enhancing the efficiency and reach of musical and audio applications. Through the lens of ecoacoustics and sonification, we consider the environmental and ethical dimensions of the IoS.

By synthesizing current research and practical applications, this paper aims to provide a foundational understanding of the Musical Metaverse, offering insights into its future directions and potential impact on the global music industry.

Music in Extended Reality (XR) is increasingly present in both academic and industrial research. While XR applications are more prevalent in STEM education, there is growing interest in the use of XR for music education. This study investigates how multi-user XR scenarios affect perceived social presence and flow in music-related tasks, as both dimensions positively correlate with learning satisfaction and perceived learning effectiveness. Participants played interactive musical memory games using the multi-user applications PatchWorld and Fortnite under different Virtual and Mixed Reality conditions. Initial results indicate significant differences in presence but not in flow perception depending on the XR modality, providing valuable insights for the design of effective XR-based music teaching tools.

This paper presents the Holodeck project, a multi-room research framework designed to support distributed multimedia concert performances. Developed through an inter-lab collaboration at New York University, the Holodeck platform leverages the Corelink engine, a flexible and unitary data routing system that facilitates immersive, interactive experiences across diverse networked environments. The framework enables real-time streaming and synchronization of various data types, including audio, video, and motion-capture data, thus supporting the design and implementation of augmented Network Music Performances (NMPs). Discussion around case studies of large-scale distributed concerts illustrate the system's capabilities and potential applications. In a pilot research exploration, the platform has been used to gather preliminary data on the quality of experience from both audiences and musicians. This research aims to enhance telepresence and realism in remote collaborations, contributing to the development of new methodologies and artistic practices in multimedia performances. The paper also discusses the technical challenges and solutions associated with implementing such a flexible and adaptive system and future work regarding the platform and its usage in multimedia immersive NMPs.