Síntesis de audio cantado mediante modelos de aprendizaje profundo: implementación y análisis de una arquitectura U-Net asimétrica

Gabriel Pena; Ricardo A. Veiga

Autores/as

Gabriel Pena Universidad Nacional de Tres de Febrero, Universidad de Buenos Aires, Argentina https://orcid.org/0000-0001-9222-8640
Ricardo A. Veiga Universidad de Buenos Aires, Argentina https://orcid.org/0000-0003-1278-6007

Palabras clave:

aprendizaje, redes neuronales, síntesis de audio, espectrogramas, unet

Resumen

La síntesis de audio cantado constituye un problema desafiante en el campo del procesamiento de señales, dada la necesidad de preservar no solo los armónicos principales sino también información tonal fina. En particular, el problema de conversión de habla en canto resulta especialmente complejo pues requiere separar y recombinar distintas fuentes de información: el contenido fonético y tímbrico del habla y la estructura tonal y rítmica de una melodía. En este trabajo se implementa una arquitectura de tipo U-net asimétrica previamente utilizada en este campo. Se proponen siete modelos y se evalúa su desempeño mediante métricas objetivas. Los experimentos se realizaron utilizando el mismo conjunto de datos y bajo las mismas condiciones reportadas en el trabajo original de referencia. En este contexto, varios de los modelos propuestos obtuvieron una puntuación comparable a la del original en las métricas de comparación utilizando una arquitectura más simple.

Descargas

Los datos de descarga aún no están disponibles.

Referencias

Agarwal, S., Ganapathy, S., & Takahashi, N. (2022). Leveraging Symmetrical Convolutional Transformer Networks for Speech to Singing Voice Style Transfer. INTERSPEECH 2022 - Voice Conversion and Adaption II. https://doi.org/10.48550/arXiv.2208.12410

Bai, P., Zheng, M., & Shi, X. (2025). A Survey of Singing Voice Synthesis. Proceedings of the 10th Conference on Sound and Music Technology.

Bonada, J., & Serra, X. (2007). Synthesis of the Singing Voice by Performance Sampling and Spectral Models. IEEE Signal Processing Magazine, 24(2), 67-79. https://doi.org/10.1109/MSP.2007.323266

Brum, L. A. Z., & Moreno, E. D. (2020). Challenges and Perspectives on Realtime Singing Voice Synthesis. Revista de Informática Teórica e Aplicada, 27(4), 118-126. https://doi.org/10.22456/2175-2745.107292

Cho, Y.-P., Yang, F.-R., Chang, Y.-C., Cheng, C.-T., Wang, X.-H., & Liu, Y.-W. (2021). A Survey on Recent Deep Learning-driven Singing Voice Synthesis Systems. Proceedings of the IEEE International Conference on Artificial Intelligence and Virtual Reality (AIVR).

Dreamtonics Co., Ltd. (s.f.). Synthesizer V [Accessed: 2026-03-16]. https://dreamtonics.com/synthesizerv/

Duan, Z., Fang, H., Li, B., Sim, K. C., & Wang, Y. (2013). The NUS sung and spoken lyrics corpus: A quantitative comparison of singing and speech. 2013 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference, 1-9. https://doi.org/10.1109/APSIPA.2013.6694316

Griffin, D. W., & Lim, J. (1984). Signal estimation from modified short-time Fourier transform. IEEE Transactions on Acoustics, Speech, and Signal Processing.

Kelleher, L. (2023). What is PESQ? - Cyara.

Kenmochi, H., & Oshita, H. (2007). VOCALOID — Commercial singing synthesizer based on sample concatenation. Proceedings of the Interspeech 2007, 4009-4010.

Kim, J. W., Salamon, J., Qi Li, P., & Bello, J. P. (2018). Crepe: A Convolutional Representation for Pitch Estimation. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 161-165.

Librosa Development Team. (s.f.). Librosa [Accessed: 2026-03-31]. https://librosa.org/doc/main/index.html

Liu, L., Jiang, H., He, P., Chen, W., Liu, X., Gao, J., & Han, J. (2019). On the Variance of the Adaptive Learning Rate and Beyond. https://doi.org/10.48550/arXiv.1908.03265

Pan, C., Yao, D., Zhang, Y., Guo, W., Lu, J., Zhu, Z., & Zhao, Z. (2025). Synthetic Singers: A Review of Deep-Learning-based Singing Voice Synthesis Approaches. Proceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics

Parekh, J. (2020). Speech-to-Singing Conversion in an Encoder-Decoder Framework - ICASSP 2020 Project Website. https://jayneelparekh.github.io/icassp20

Parekh, J., Rao, P., & Yang, Y.-H. (2020). Speech-To-Singing Conversion in an Encoder-Decoder Framework. ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 261-265. https://doi.org/10.1109/ICASSP40776.2020.9054473

Pena, G. (2025). SkyVoiceNet. 10.5281/zenodo.17119012

Rix, A. W., Beerends, J. G., Hollier, M. P., & Hekstra, A. P. (2001). Perceptual evaluation of speech quality (PESQ)-a new method for speech quality assessment of telephone networks and codecs. Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing, 2, 749-752. https://doi.org/10.1109/ICASSP.2001.941023

Ronneberger, O., Fischer, P., & Brox, T. (2015). U-Net: Convolutional Networks for Biomedical Image Segmentation. Medical Image Computing and Computer-Assisted Intervention (MICCAI 2015), 234-241.

Shen, C., Zhao, L., Fu, C., Gan, B., & Du, Z. (2025). Transinger: Cross-Lingual Singing Voice Synthesis via IPA-Based Phonetic Alignment. Sensors, 25(13). https://doi.org/10.3390/s25133973

Taal, C. H., Hendriks, R. C., Heusdens, R., & Jensen, J. (2010). A short-time objective intelligibility measure for time-frequency weighted noisy speech. 2010 IEEE International Conference on Acoustics, Speech and Signal Processing.

Taal, C. H., Hendriks, R. C., Heusdens, R., & Jensen, J. (2011). An Algorithm for Intelligibility Prediction of Time–Frequency Weighted Noisy Speech. IEEE Transactions on Audio, Speech, and Language Processing, 19(7), 2125-2136. https://doi.org/10.1109/TASL.2011.2114881

Wu, D.-Y., & Yang, Y.-H. (2020). Speech-To-Singing Conversion based on Boundary Equilibrium GAN. INTERSPEECH 2020 - Singing and Multimodal Systems. https://doi.org/10.48550/arXiv.2005.13835

Yamaha Corporation. (s.f.). VOCALOID [Accessed: 2026-03-16]. https://www.vocaloid.com

Zhang, Y., Guo, W., Pan, C., Yao, D., Zhu, Z., Jiang, Z., Wang, Y., Jin, T., & Zhao, Z. (2025). TCSinger 2: Customizable Multilingual Zero-shot Singing Voice Synthesis. Findings of the Association for Computational Linguistics: ACL 2025, 13280-13294. https://doi.org/10.18653/v1/2025.findings-acl.687