Diffrhythm+: Controllable and Flexible Full-Length Song Generation with Preference Optimization

Abstract

Songs, as a central form of musical art, exemplify the richness of human intelligence and creativity. While recent advances in generative modeling have enabled notable progress in long-form song generation, current systems for full-length song synthesis still face major challenges, including data imbalance, insufficient controllability, and inconsistent musical quality. DiffRhythm, a pioneering diffusion-based model, advanced the field by generating full-length songs with expressive vocals and accompaniment. However, its performance was constrained by an unbalanced dataset and limited controllability over musical style, resulting in noticeable quality disparities and restricted creative flexibility.
To address these limitations, we propose DiffRhythm+, an enhanced diffusion-based framework for controllable and flexible full-length song generation. DiffRhythm+ leverages a substantially expanded and balanced training dataset to mitigate issues such as repetition and omission of lyrics, while also fostering the emergence of richer musical skills and expressiveness. The framework introduces a multi-modal style conditioning strategy, enabling users to precisely specify musical styles through both descriptive text and reference audio, thereby significantly enhancing creative control and diversity. We further introduce a direct performance optimization approach aligned with user preferences, guiding the model toward consistently preferred outputs across evaluation metrics. Extensive experiments demonstrate that DiffRhythm+ achieves significant improvements in naturalness, arrangement complexity, and listener satisfaction over previous systems.

Text Prompt

[00:16.74]You get it easy
[00:19.21]When I gonna be alone
[00:25.13]But evening when my life to contain your love
[00:32.84]Cause when the life goes alone
[00:35.78]You want it all
[00:37.31]And you back yourself and drove
[00:41.87]And when your heart is close
[00:44.04]Do you wanna in most
[00:46.18]Ease to make it
[00:47.20]Just to make it oh oh oh
[00:50.41]Because all my love he want this love
[00:54.57]And we'll take into anyone
[00:59.00]We should back then start
[01:00.93]We don't mind you had
[01:02.99]Cause your loving is so where you go
[01:07.75]oh oh
[01:11.70]So one our way is in for
[01:16.10]oh oh
[01:20.20]Loving you is all I want
[01:24.43]And you with me just put you for the reason
[01:32.35]Cause beginning so please just let it go
[01:40.62]Cause when the life goes alone
[01:43.02]You want it all
[01:45.11]And you back yourself and drove
[01:49.12]And when your heart is close
[01:51.44]Do you wanna is most
[01:53.58]Ease to make it
[01:54.57]Just to make it oh oh oh
[01:58.30]Maybe it's time to and bring for its God
[02:02.31]Cause she let me straight to your heart
[02:06.66]Maybe but ready to save what we want
[02:10.52]Dist I'm coming for you
[02:14.54]Because all my love he want this love
[02:18.74]And we'll take into anyone
[02:23.06]We should back then start
[02:25.17]We don't mind you had
[02:27.22]Cause your loving is so where you go
[02:31.94]oh oh
[02:36.09]Someone are we waiting for
[02:40.36]oh oh
[02:44.54]Loving you is all I want
[02:53.02]Our love she is walking line
[03:01.03]We should stay this time
[03:03.15]We don't love you had
              

Audio Prompt

Musical Skills

Ethics Statement

DiffRhythm+ enables the creation of original music across diverse genres, supporting applications in artistic creation, education, and entertainment. While designed for positive use cases, potential risks include unintentional copyright infringement through stylistic similarities, inappropriate blending of cultural musical elements, and misuse for generating harmful content. To ensure responsible deployment, users must implement verification mechanisms to confirm musical originality, disclose AI involvement in generated works, and obtain permissions when adapting protected styles.