AI in Computational Arts, Music, and Games

Notating electroacoustic music can be challenging due to the uniqueness of the instruments employed. Electronic instruments can include generative components that can manipulate sound at different time levels, in which parameter variations can correlate non-linearly to changes in the instrument’s timbre. The way compositions for electronic instruments are notated depends on their interfaces and the parameter controls available to performers, which determine the state of their sound-generating system. In this article, we propose a notation system for generative synthesis based on a projection from its parameter space to a timbre space, allowing to organise synthesiser states based on their timbral characteristics. To investigate this approach, we introduce the Meta-Benjolin, a state-based notation system for chaotic sound synthesis employing a three-dimensional, navigable timbre space and a composition timeline. The Meta-Benjolin was developed as a control structure for the Benjolin, a chaotic synthesiser. Framing chaotic synthesis as a specific instance of generative synthesis, we discuss the advantages and drawbacks of the state- and timbre-based representation we designed based on the thematic analysis of an interview study with 19 musicians, who composed a piece using the Meta-Benjolin notational interface.

Neural Audio is a category of deep learning pipelines which output audio signals directly, in real-time scenarios of action-sound interactions. In this work, we examine how neural audio-based artificial intelligence, when embedded in digital musical instruments (DMIs), shapes embodied musical interaction. While DMIs have long struggled to match the physical immediacy of acoustic instruments, neural audio methods can magnify this challenge, requiring data collection, model training and deep theoretical knowledge that appear to push musicians toward symbolic or conceptual modes of engagement. Paradoxically, these same methods can also foster more embodied practices, by introducing opaque yet expressive behaviors that free performers from rigid technical models and encourage discovery through tactile, real-time experimentation. Drawing on established perspectives in DMI embodiment literature, as well as emerging neural-audio-focused efforts within the community, we highlight two seemingly conflicting aspects of these instruments: on one side, they inherit many “disembodying” traits known from DMIs; on the other, they open pathways reminiscent of acoustic phenomenology and soma, potentially restoring the close physical interplay often missed in digital performance.

The advancement of AI-tools for musical performance has inspired exciting opportunities for interaction with musical-AI-agents. Interactions between humans and AI-agents in musical settings entail dynamic exchanges of control and power, and framings of AI-agents’ roles by human performers. We probe these framings and power-control exchanges through qualitative thematic lenses, drawing from post-phenomenology, matters of fact and concern and feminist science and technology studies. We contribute with a novel interdisciplinary analytical method as a tool for developers and designers of AI systems to help visibilise and examine the implicit, the wider connections and entangled filaments in Human–AI musical interactions.

Technological advancements in deep learning for speech and voice have contributed to a recent expansion in applications for voice cloning, synthesis and generation. Invisibilised stakeholders in this expansion are numerous absent bodies, whose voices and voice data have been integral to the development and refinement of these speech technologies. This position paper probes current working practices for voice and speech in machine learning and AI, in which the bodies of voices are “invisibilised". We examine the facts and concerns about the voice-Body in applications of AI-voice technology. We do this through probing the wider connections between voice data and Schaefferian listening; speculating on the consequences of missing Bodies in AI-Voice; and by examining how vocalists and artists working with synthetic Bodies and AI-voices are ‘bringing the Body back’ in their own practices. We contribute with a series of considerations for how practitioners and researchers may help to ‘bring the Body back’ into AI-voice technologies.

The explosion of content generated by artificial intelligence (AI) models has initiated a cultural shift in arts, music, and media, whereby roles are changing, values are shifting, and conventions are challenged. The vast, readily available dataset of the Internet has created an environment for AI models to be trained on any content on the Web. With AI models shared openly and used by many globally, how does this new paradigm shift challenge the status quo in artistic practices? What kind of changes will AI technology bring to music, arts, and new media?