Lip shape-aware word selection for lyric translation

Authors: Kotaro Ikeda, Chihaya Matsuhira, Hirotaka Kato, Marc A. Kastner, Takatsugu Hirayama, Takahiro Komamizu, Ichiro Ide

Abstract:

The pronunciation of a language and corresponding lip shapes have a strong relationship. As such, technologies that align the lip movements of a sentence in the translated language with that in the original language can be useful for scenarios such as movie dubbing, particularly in singing scenes, where consistency between a character’s speech and their visual lip movements is desirable. In this paper, we define lip shape similarity based on the International Phonetic Alphabet (IPA) chart which is a knowledge on phonetics, and integrate it into a word selection algorithm for general machine translation. We propose an automatic lyric translation method that balances the semantics and the lip shape similarity when translating the source lyrics into a target language. For quantitative evaluation, by using professionally translated lyrics as a reference, we optimize the proposed method to best preserve both semantics and lip shape similarity. Experimental results demonstrate that the generated translations achieve higher lip shape similarity than that by baseline translations.

Type: 8th Asian Conference on Pattern Recognition (ACPR 2025)

Publication date: To be published in Nov 2025