Ganis, F., Knudesn, E. F., Lyster, S. V. K., Otterbein, R., Südholt, D., & Erkut, C. (2021). Real-time Timbre Transfer and Sound Synthesis using DDSP. Retrieved from http://arxiv.org/abs/2103.07220

Overview - 何がすごい?

Google MagentaのDDSPをリアルタイムに動かせるプラグイン！と聞いただけですごいが....

Abstract

Neural audio synthesis is an actively researched topic, having yielded a wide range of techniques that leverages machine learning architectures. Google Magenta elaborated a novel approach called Differ- ential Digital Signal Processing (DDSP) that incorporates deep neural networks with preconditioned digital signal processing techniques, reaching state-of-the-art results especially in timbre transfer applications. However, most of these techniques, including the DDSP, are generally not applicable in real-time constraints, making them ineligible in a musical workflow. In this paper, we present a real-time implementation of the DDSP library embedded in a virtual synthesizer as a plug-in that can be used in a Digital Audio Workstation. We focused on timbre transfer from learned representations of real instruments to arbitrary sound inputs as well as controlling these models by MIDI. Furthermore, we developed a GUI for intuitive high-level controls which can be used for post-processing and manipulating the parameters estimated by the neural network. We have conducted a user experience test with seven participants online. The results indicated that our users found the interface appealing, easy to understand, and worth exploring further. At the same time, we have identified issues in the timbre transfer quality, in some components we did not implement, and in installation and distribution of our plugin. The next iteration of our design will address these issues. Our real-time MATLAB and JUCE implementations are available at https://github.com/SMC704/juce-ddsp and https://github.com/SMC704/matlab-ddsp, respectively

Motivation

• DDSPをリアルタイムに動かせたら便利だよね
• プラグインのデザインの検証

Architecture

• JUCEベースでVST3のプラグインとして実装したバージョンとMATLABで実装したバージョンの二つ
• MATLABのコードを C++に自動的に変換する仕組みも使っている

Results

• VST3のバージョンでマイクで入力した音をリアルタイムに変換している例

うーん、音質的にだいぶ微妙

Further Thoughts

• アルゴリズムの論文ではなく、あくまでプラグインのデザイン、ユーザビリティの評価にフォーカスが当たっている感じ。
• Noise/Mod(modulation?)というセクションがプラグインにあり、論文でも触れられているがどうやっても有効に出来なかった
• プラグイン自体 めちゃくちゃ不安定。一時間に20回ほどクラッシュ...
• ノイズの方が面白い

Links

MATLABバージョン

SMC704/matlab-ddsp

Contribute to SMC704/matlab-ddsp development by creating an account on GitHub.

github.com