Latent Timbre Synthesis

Fast Deep Learning tools for experimental electronic music

Description

Latent Timbre Synthesis is a new audio synthesis method using deep learning . The synthesis method allows composers and sound designers to interpolate and extrapolate between the timbre of multiple sounds using the latent space of audio frames. The implementation includes a fully working application with a graphical user interface, called interpolate_two, which enables practitioners to generate timbres between two audio excerpts of their selection using interpolation and extrapolation in the latent space of audio frames. Our implementation is open source, and we aim to improve the accessibility of this technology by providing a guide for users with any technical background. Our study includes a qualitative analysis where nine composers evaluated the Latent Timbre Synthesis and the interpolate_two application within their practices.

The visualizations below illustrate a latent space generated by a Variational Autoencoder. On the right, the green and red dots represent latent vectors of audio frames of two audio examples from the dataset, while the green is the latent vectors of a synthesized sound generated by interpolating the latent vectors of two original audio files.

Visalization of a latent space generated using our approach.

Resources

Code

https://github.com/ktatar/latent-timbre-synthesis

Compilation Album

💿️ Nine composers joined our qualitative study and composed short pieces using Latent Timbre Synthesis. The album is available here: https://medienarchiv.zhdk.ch/entries/376e81a2-a6b9-4b74-91a3-9144c192f8e1

Acknowledgements

This work has been supported by the Swiss National Science Foundation, the Natural Sciences and Engineering Research Council of Canada, and Social Sciences and Humanities Research Council of Canada.

Ce travail est supporté par le Fonds national Suisse de la recherche scientifique, le Conseil national des sciences et de l’ingénieurie du Canada, et le Conseil national des sciences humaines et sociales du Canada.