Overview - 何がすごい?

Abstract

Sensory substitution can help persons with perceptual deficits. In this work, we attempt to visualize audio with video. Our long-term goal is to create sound perception for hearing impaired people, for instance, to facilitate feedback for training deafspeech. Different from existing models that translate between speech and text or text and images, we target an immediate and low-level translation that applies to generic environment sounds and human speech without delay. No canonical mapping is known for this artificial transla- tion task. Our design is to translate from audio to video by compressing both into a common latent space with shared structure. Our core contribution is the development and eval- uation oflearned mappings that respect human perception limits and maximize user comfort by enforcing priors and combining strategies from unpaired image translation and disentanglement. We demonstrate qualitatively and quanti- tatively that our AudioViewer model maintains important audio features in the generated video and that generated videos offaces and numbers are well suited for visualizing high-dimensional audio features since they can easily be parsed by humans to match and distinguish between sounds, words, and speakers.

Motivation

1. 人は自然環境、特に人の顔の変化に敏感 → マッピングされる画像として、顔の画像を使う
2. 人は対象の急激な変化に慣れない → スムーズな変化を実現するための制約の導入
3. 頻出する音に対応する画像も頻出してほしい → CycleGANようなCycle consistencyを導入。音と画像の関係性を学習
4. 音(言葉)の内容とスタイル(話者、トーン etc)を切り離す

Architecture

前提4.のスタイルとコンテンツを切り離すために d次元の潜在ベクトル $z$ を話者を表す $z_s$ と コンテンツを表す $z_c$ の二つにわけ、これらをつなげて $z$ とする。　

$\mathbf{z}_{s}=\left[z_{1}, \cdots, z_{m}\right]^{T}$

$\mathbf{z}_{c}=\left[z_{m+1}, \cdots, z_{d}\right]^{T}$

Results

結果、どちらもランダムよりも遥かに良い成績に

顔のモデルの方(CelebA)が概ね結果が良いことに

Further Thoughts

• VJ的な要素として音と映像の潜在空間をつなげる方法はこれまでも提案されてきたが、聴覚障害者のための...という切り口は新しい
• コンテンツとスタイルの切り分け、潜在空間間の構造を近づけるためのcycle lossの設定などは非常に勉強になる！！
• 画像を大量にみることで果たして音を識別できるように人は学習できるものなのだろうか

Links

Learning Latent Representations for Speech Generation and Transformation

An ability to model a generative process and learn a latent representation for speech in an unsupervised fashion will be crucial to process vast quantities of unlabelled speech data. Recently, deep probabilistic generative models such as Variational Autoencoders (VAEs) have achieved tremendous success in modeling natural images.

arxiv.org

Learning Latent Representations for Speech Generation and Transformation

(this work is to appear in Interspeech 2017. the paper can be found here) apply a convolutional VAE to model the generative process of natural speech. derive latent space arithmetic operations to disentangle learned latent representations. demonstrate the capability to randomly generate short speech segments (200ms).

people.csail.mit.edu