A Vector Quantized Approach for Text to Speech Synthesis on Real-World Spontaneous Speech

Recent Text-to-Speech (TTS) systems trained on reading or acted corpora have achieved near human-level naturalness. The diversity of human speech, however, often goes beyond the coverage of these corpora. We believe the ability to handle such diversity is crucial for AI systems to achieve human-level communication. Our work explores the use of more abundant real-world data for building speech synthesizers. We train TTS systems using real-world speech from YouTube and podcasts. We observe the mismatch between training and inference alignments in mel-spectrogram based autoregressive models, leading to unintelligible synthesis, and demonstrate that learned discrete codes within multiple code groups effectively resolves this issue. We introduce our MQTTS system whose architecture is designed for multiple code generation and monotonic alignment, along with the use of a clean silence prompt to improve synthesis quality. We conduct ablation analyses to identify the efficacy of our methods. We show that MQTTS outperforms existing TTS systems in several objective and subjective measures.
Accepted to AAAI 2023
Code
Paper

Reference Speaker	Transformer TTS	VITS (40M)	VITS (100M)	MQTTS (40M)	MQTTS (100M)	MQTTS (200M)	Text————————————————-
							in two thousand and eighteen there was so much um expectation on messi can he take them to the final and win it this time .
							maybe don’t have enough room in front of the truck to ah get the truck out where you need it to to get you straight .
							so same as with an apple watch series two or newer , you can take this ah with you when you’re swimming or even snorkeling , yes up to fifty meters .
							yeah so ah all of the a i conferences are open to anyone who is capable of ah you know make you know paying for the trip and the the ticket .
							but one that stands out to me was , ah , concluding the , ah , montreal protocol , ah , as a treaty that dealt with a extraordinarily significant environmental issue .
							my guitar we just wanted to give a very timeless feeling with this ah album cover when i’m in my real life i’m basically always very unsophisticated .
							i think it was very well planned very well executed and ah to solve those big bombs in this closed circle around and on the ship .