Length Desensitization in Directed Preference Optimization

Abstract

Direct Preference Optimization (DPO) is widely utilized in the Reinforcement Learning from Human Feedback (RLHF) phase to align Large Language Models (LLMs) with human preferences, thereby enhancing both their harmlessness and efficacy. However, it has been observed that DPO tends to over-optimize for verbosity, which can detrimentally affect both performance and user experience. In this paper, we conduct an in-depth theoretical analysis of DPO’s optimization objective and reveal a strong correlation between its implicit reward and data length. This correlation misguides the optimization direction, resulting in length sensitivity during the DPO training and leading to verbosity. To address this issue, we propose a length-desensitization improvement method for DPO, termed LD-DPO. The proposed method aims to desensitize DPO to data length by decoupling explicit length preference, which is relatively insignificant, from the other implicit preferences, thereby enabling more effective learning of the intrinsic preferences. We utilized two settings (Base and Instruct) of Llama2-13B, Llama3-8B, and Qwen2-7B for experimental validation on various benchmarks including MT-Bench and AlpacaEval 2. The experimental results indicate that LD-DPO consistently outperforms DPO and other baseline methods, achieving more concise responses with a 10-40% reduction in length compared to DPO. We conducted in-depth experimental analyses to demonstrate that LD-DPO can indeed achieve length desensitization and align the model more closely with human-real preferences.

“Brevity is the Soul of Wit.”

—William Shakespeare

1 Introduction

Human preference alignment is crucial to enable large language models (LLMs) to be helpful, honest, and harmless. Among the various methods to achieve effective alignment (Dai et al., 2024; Yuan et al., 2024a), Directed Preference Optimization (DPO) has emerged as a promising technique (Rafailov et al., 2024), giving rise to numerous derivative algorithms (Hong et al., 2024; Chen et al., 2024b; Ethayarajh et al., 2024). DPO eliminates the reliance on online Reward Models (RMs) by reparameterizing the reward function in Reinforcement Learning from Human Feedback (RLHF), thereby implementing a simple and stable offline preference learning paradigm. Among the dimensions of human language preferences, detailedness is one of the most straightforward categories that current alignment algorithms can effortlessly capture, as longer texts tend to be richer in content. However, it has been demonstrated that DPO is susceptible to an over-optimization issue in this particular preference dimension (Xu et al., 2024). As shown in Fig.1, this overemphasis results in models that produce excessively verbose responses, which can compromise their instruction-following and reasoning capabilities (Ding et al., 2023; Yuan et al., 2024b).

Figure 1: Average response length of multi-iteration DPO Models (Chen et al., 2024d) on AlpacaEval 2.

The phenomenon of verbose response caused by DPO is often attributed to the presence of length bias in the training data (Park et al., 2024; Singhal et al., 2023). This bias arises from an inherent length preference in offline RMs (Wang et al., 2024; Chen et al., 2024c), which results in most preferred responses (chosen) being significantly longer than the dispreferred ones (rejected). Based on this assumption, Yuan et al. (2024b) proposed LIFT-DPO to mitigate the length bias in the training data through a prompt enhancement strategy. Recently, more researchers have questioned the efficacy of the DPO algorithm itself. Park et al. (2024) introduce a regularization term in the optimization objective to adjust the weight of the gradient according to the length difference between the preference pairs. Meng et al. (2024) propose a reference-model-free method SimPO, which used the likelihood averaged by length to eliminate the effect of data length. Lu et al. (2024) introduce a down-sampling approach on KL divergence to eliminate the length reliance of DPO. Though Lu et al. (2024) have conducted a statistical analysis of the implicit rewards during the DPO process and found that the rewards might be overestimated or underestimated due to length, the theoretical explanation of why DPO encounters this issue remains inadequately explored. And experimental results demonstrate that these methods either fail to achieve significant length control or compromise the performance to some extent.

In this paper, we attribute the verbosity problem to the length sensitivity of DPO. Specifically, the partial derivatives of the optimization objective of DPO with respect to the chosen and rejected responses are inversely proportional to their respective likelihood (Feng et al., 2024a). Since the likelihood, which is calculated as the product of the conditional probabilities of each token, decreases rapidly with increasing sequence length, longer chosen or rejected responses are disproportionately favored in the optimization process. Moreover, the length disparity between the chosen and rejected responses will substantially skew the optimization objective, ultimately biasing the direction of optimization. While decreasing the likelihood of any rejected may not affect response length, the primary cause of the verbosity problem is the model’s tendency to increase the likelihood of longer chosen responses.

To address this issue, we propose an offline optimization algorithm for Length Desentsitization of DPO, termed LD-DPO. In this approach, we decompose the likelihood of the longer response in a preference pair into the product of the likelihood of the public-length portion and the likelihood of the excessive portion. The excessive portion is further broken down into human-like preference (implicit and intrinsic) and verbosity preference (due to excess length). LD-DPO aims to mitigate the verbosity preference caused by excessively long responses, thereby smoothing the relationship between the likelihood and response length. This adjustment reduces the influence of length on the optimization direction in DPO, effectively achieving length desensitization.

We employ two settings (Base and Instruct) of Llama2-13B (Touvron et al., 2023), Llama3-8B (AI@Meta, 2024), and Qwen2-7B (Yang et al., 2024) for experimental validation on MT-Bench (Zheng et al., 2024) and AlpacaEval 2 (Dubois et al., 2024). The experimental results indicate that LD-DPO consistently outperforms DPO and other baseline methods, achieving more concise responses with a 10-40% reduction in length compared to DPO. Moreover, experiment on MT-Bench and ProofWriter (Tafjord et al., 2021) shows that LD-DPO significantly improves models’ reasoning performance. An interesting phenomenon is also observed: the length sensitivity during DPO training exhibits a negative correlation with the underlying model capability. we then define $\gamma$ as the length sensitivity coefficient and conduct a detailed analysis of the DPO length sensitivity across models of varying capabilities, we believe $\gamma$ is instructive for the entire preference optimization process. Our contributions are summarized as follows:

• To the best of our knowledge, we are the first to propose that DPO exhibits length sensitivity and provide theoretical validation for this phenomenon.

• We propose LD-DPO, a length-desensitization preference optimization algorithm that mitigates length sensitivity by decoupling length preference from the reward.

• We experimentally verify that LD-DPO enables the model to achieve superior results with more concise responses, reducing response length by 10-40% compared to DPO.

2 Preliminaries

In this section, we will outline the standard pipeline of Reinforcement Learning From Human Feedback (RLHF) (Bai et al., 2022; Ziegler et al., 2019)and the Direct Preference Optimization (DPO) algorithm (Rafailov et al., 2024), which is essential for the analysis of the length sensitivity of DPO and the motivation of our method.

2.1 Reinforcement Learning From Human Feedback

The standard pipeline of RLHF aligns LLMs with human preferences in three stages:
Supervised Fine-tuning (SFT) stage: In this stage, labeled data is used to fine-tune the pre-trained model so that it acquires a basic ability to follow commands and carry on a fluent conversation, to obtain model $\pi^{SFT}(y|x)$.
Reward Model (RM) Training stage: In the second stage, $\pi^{SFT}(y|x)$ is utilized by prompts $x$ to generate pairs of responses $(y_{1},y_{2})\sim\pi^{SFT}(y|x)$, which are then labeled by human annotators as a preferred answer $y_{w}$ and a dispreferred answer $y_{l}$, denoted as $y_{w}>y_{l}$. To predict these preferences, previous works employ the Bradley-Terry (BT) RM (Bradley & Terry, 1952), which essentially constructs a pairwise contrast:

$$\begin{split}\mathcal{L}_{RM}=-\log\frac{exp(r_{\phi}(x,y_{w}))}{exp(r_{\phi}(x,y_{w}))+exp(r_{\phi}(x,y_{l}))}.\end{split}$$ (1)

Reinforcement Learning (RL) stage: In the Final Stage, the reward function is used to provide feedback to the language model. The optimization objective is formulated as:

$$\begin{aligned} \max_{\pi_{\theta}}\mathbb{E}_{x\sim\mathcal{D},y\sim\pi_{\theta}(y|x)}[r_{\phi}(x,y)]-\beta\mathbb{D}_{KL}[\pi_{\theta}(y|x)\|\pi_{ref}(y|x)],\end{aligned}$$ (2)

where $\beta$ is a parameter controlling the deviation from the reference model $\pi_{ref}$, namely the initial SFT model $\pi^{SFT}(y|x)$, and in practice, the language model $\pi_{\theta}$ is also initialized to $\pi^{SFT}(y|x)$. This objective is optimized using a general purpose RL algorithm, such as PPO (Wu et al., 2023).

2.2 Direct Preference Optimization

Direct Preference Optimization (DPO) is one of the most popular offline preference optimization methods, which starts with the same objective as Eq.2, reparameterizes the reward function $r$ using a closed-form expression with the optimal policy:

$$r(x,y)=\beta\log\frac{\pi_{\theta}(y|x)}{\pi_{ref}(y|x)}+\beta\log Z(x),$$ (3)

where $Z(x)=\sum_{y}\pi_{ref}(y|x)exp(\frac{1}{\beta}r(x,y))$ is the partition function, which is only relevant for $\pi_{ref}$ and $\pi_{\theta}$, no additional training of the RM is required. By incorporating Eq.3 into the BT ranking objective, $p(y_{w}>y_{l}|x)=\sigma(r(x,y_{w})-r(x,y_{l}))$, therefore, the optimization objective becomes:

$$\begin{aligned} \mathcal{L}_{DPO}(\pi_{\theta};\pi_{ref})=-\mathbb{E}_{(x,y_{w},y_{l})\sim\mathcal{D}}[\log\sigma(\beta\log\frac{\pi_{\theta}(y_{w}|x)}{\pi_{ref}(y_{w}|x)}-\beta\log\frac{\pi_{\theta}(y_{l}|x)}{\pi_{ref}(y_{l}|x)})].\end{aligned}$$ (4)

DPO replaces the reward model (RM) with an implicit reward, offering enhanced stability and ease of training compared to traditional reinforcement learning methods such as PPO. Several related works have validated the effectiveness of this paradigm.

3 Methodology

In this section, we first conduct a theoretical analysis of the optimization object of DPO and verify that differences in data length significantly affect the optimization direction during the training process, demonstrating that DPO is length-sensitive. We then derive our LD-DPO algorithm, which addresses the length sensitivity problem by reparameterizing the likelihood, thereby preventing the generation of verbose responses and aligning the model more closely with human-like preferences.

3.1 Length Sensitivity of DPO

According to the optimization objective of DPO in Eq.4, we know that the purpose of DPO is to make the likelihood of human-preferred response $y_{w}$ given $x$ greater than that of human-dispreferred response $y_{l}$, denoted as $\pi_{\theta}(y_{w}|x)>\pi_{\theta}(y_{l}|x)$. Additionally, $\pi_{\theta}$ serves as the actor model, while $\pi_{ref}$ is introduced to prevent the model from deviating from the reference model. Both models are tpyically products of the Supervised Fine-Tuning (SFT) phase.

Following Feng et al. (2024a), we provide the theoretical derivation of the optimization objective for DPO. Firstly, we denote $\mathcal{X}_{1}=\pi_{\theta}(y_{w}|x)$, $\mathcal{X}_{2}=\pi_{\theta}(y_{l}|x)$ and $\mathcal{K}_{1}=\pi_{ref}(y_{w}|x)$, $\mathcal{K}_{2}=\pi_{ref}(y_{l}|x)$, the loss function of DPO can be written as:

$$\begin{aligned} \mathcal{L}_{DPO}(\mathcal{X}_{1};\mathcal{X}_{2})=-\log(\frac{(\mathcal{K}_{2}\mathcal{X}_{1})^{\beta}}{(\mathcal{K}_{2}\mathcal{X}_{1})^{\beta}+(\mathcal{K}_{1}\mathcal{X}_{2})^{\beta}}).\end{aligned}$$ (5)

We calculate the partial derivatives of $\mathcal{L}_{DPO}$ with respect to $\mathcal{X}_{1}$, and $\mathcal{X}_{2}$:

$$\left\{\begin{array}[]{l}\dfrac{\partial\mathcal{L}_{DPO}(\mathcal{X}_{1};\mathcal{X}_{2})}{\partial\mathcal{X}_{1}}=-\dfrac{\beta(\mathcal{K}_{1}\mathcal{X}_{2})^{\beta}}{\mathcal{X}_{1}((\mathcal{K}_{2}\mathcal{X}_{1})^{\beta}+(\mathcal{K}_{1}\mathcal{X}_{2})^{\beta})},\\ \dfrac{\partial\mathcal{L}_{DPO}(\mathcal{X}_{1};\mathcal{X}_{2})}{\partial\mathcal{X}_{2}}=\dfrac{\beta(\mathcal{K}_{1}\mathcal{X}_{2})^{\beta-1}}{(\mathcal{K}_{2}\mathcal{X}_{1})^{\beta}+(\mathcal{K}_{1}\mathcal{X}_{2})^{\beta}},\end{array}\right.$$ (6)

then leading to the following result:

$$\begin{aligned} \left\lvert\frac{\partial\mathcal{L}_{DPO}(\mathcal{X}_{1};\mathcal{X}_{2})}{\partial\mathcal{X}_{1}}/\frac{\partial\mathcal{L}_{DPO}(\mathcal{X}_{1};\mathcal{X}_{2})}{\partial\mathcal{X}_{2}}\right\rvert=\frac{\mathcal{X}_{2}}{\mathcal{X}_{1}}=\frac{\pi_{\theta}(y_{l}|x)}{\pi_{\theta}(y_{w}|x)}.\end{aligned}$$ (7)

Therefore, the partial derivatives of the optimization objective with respect to $\pi_{\theta}(y_{w}|x)$ and $\pi_{\theta}(y_{l}|x)$ are inversely proportional to their respective values. Furthermore, the derivation process of Eq.7and a detailed analysis of the absolute magnitude of the gradient is provided in Appendix A.1. When $\pi_{\theta}(y_{w}|x)$ is less than $\pi_{\theta}(y_{l}|x)$, DPO tends to increase the likelihood of generating human-preferred response $y_{w}$. Conversely, DPO tends to avoid generating human-dispreferred response $y_{l}$. Based on this conclusion, we will analyze the length sensitivity of DPO as follows.

(a) DPO ($\alpha=1$)

(b) $\alpha=0.5$

(c) $\alpha=0$

Figure 2: Comparison of the relationship between the length of preference data and $\pi_{\theta}(y_{l}|x)/\pi_{\theta}(y_{w}|x)$ under both DPO and LD-DPO. Measured on Llama3-8B-Instruct with UltraFeedback dataset (Cui et al., 2023), and the heatmap values represent $\log\pi_{\theta}(y_{l}|x)-\log\pi_{\theta}(y_{w}|x)$.

In DPO process, the likelihood $\pi_{\theta}(y|x)$ for sequence-level output is obtained by cumulatively multiplying the conditional probability of each token $p(y_{t}|x,y_{<t})$ as shown in Eq.8.

$$\pi_{\theta}(y|x)=\prod_{i=1}^{len(y)}p(y_{i}|x,y_{<i}).$$ (8)

As the conditional probability of the current token from the policy $p(y_{t}|x,y_{<t})$ lies within the range $[0,1]$, it follows that as the sentence $y$ consists of more tokens, $\pi_{\theta}(y|x)$ will obviously be smaller. According to Eq.7, we know that if $len(y_{w})>len(y_{l})$, then it is highly likely that $\pi_{\theta}(y_{w}|x)<\pi_{\theta}(y_{l}|x)$, so the language model tends to generate the longer response $y_{w}$ after DPO; Conversely, if $len(y_{w})<len(y_{l})$, DPO prevents the output of the longer answer $y_{l}$, but this does not imply that the shorter answer $y_{w}$ will be preferred, then verbosity arises.

As shown in Fig.2(a) and based on the above analysis, DPO is more sensitive to data pairs with large differences in length. Therefore, it tends to guide the model to prioritize length preferences in the data, ignoring other human-like preferences that are more important.

3.2 Derivation of LD-DPO

Based on the analysis conducted in the preceding section, it is evident that the length sensitivity of DPO primarily originates from the substantial influence text length exerts on the likelihood $\pi_{\theta}(y|x)$. This influence consequently biases the optimization process towards favoring data with a length advantage. To address this issue, Length-Desensitization DPO (LD-DPO) is employed to diminish the impact of length on the likelihood. This adjustment allows the optimization process to focus more on the substantive content of the text, thereby better aligning with human preference.

For a pair of preference data ($y_{w},y_{l}$) with lengths $(l_{w},l_{l})$, we denote $l_{p}=min(l_{w},l_{l})$ as the public length. Then the likelihood of response in DPO can be rewritten as:

$$\pi_{\theta}(y|x)=\prod_{i=1}^{l_{p}}p(y_{i}|x,y_{<i})\prod_{i=l_{p}+1}^{l}p(y_{i}|x,y_{<i}),$$ (9)

where $l$ is the length of $y$. The second term contains extensive length information, which directly decreases the reward and further biases the optimization objective. This bias contributes to the overall length sensitivity of DPO.

In LD-DPO, our objective is to attenuate the sensitivity of DPO by eliminating the verbosity preferences induced by the excessively long portions, while concurrently maintaining the human-like preferences, which include a certain degree of length preference. Initially, as demonstrated in Eq.10, we disassociate the verbosity preferences from the likelihood of over-long portion (second term in Eq.9) by introducing a hyperparameter $\alpha\in[0,1]$.

$$\prod_{i=l_{p}+1}^{l}\underbrace{p^{\alpha}(y_{i}|x,y_{<i})}_{\text{{human-like preferences}}}\underbrace{p^{1-\alpha}(y_{i}|x,y_{<i})}_{\text{{verbosity preference}}}.$$ (10)

We then diminish the length sensitivity of DPO by removing verbosity preference from $\pi_{\theta}(y|x)$, obtaining the modified likelihood $\hat{\pi}_{\theta}(y|x)$ in LD-DPO:

$$\hat{\pi}_{\theta}(y|x)=\prod_{i=1}^{l_{p}}p(y_{i}|x,y_{<i})\prod_{i=l_{p}+1}^{l}p^{\alpha}(y_{i}|x,y_{<i}).$$ (11)

When $\alpha=1$, $\hat{\pi}_{\theta}(y|x)=\pi_{\theta}(y|x)$, which is consistent with vanilla DPO. Conversely, when $\alpha=0$, the likelihood of over-length part is equal to $1$, meaning that only the public-length part will be considered. Ultimately, we reformulate $\hat{\pi}_{\theta}(y_{k}|x)$ in Eq.12 to present it in a more elegant form, with the detailed derivation provided in Appendix A.2.

$$\begin{aligned} \hat{\pi}_{\theta}(y|x)=\prod_{i=1}^{l}p^{\alpha}(y_{i}|x,y_{<i})\prod_{i=1}^{l_{p}}p^{1-\alpha}(y_{i}|x,y_{<i}).\end{aligned}$$ (12)

It is observable that $\hat{\pi}_{\theta}(y|x)$ is constituted by the complete sequence and the public-length component of the preference data pair. The proportion between these two components can be modulated by adjusting the hyperparameter $\alpha$. In scenarios where the length sensitivity during the DPO training process is relatively pronounced, a smaller $\alpha$ should be opted for in order to decouple the verbosity preference. Conversely, a larger $\alpha$ should be selected to avert the loss of genuine human preferences.

As shown in Fig.2, compared to DPO (Fig.2(a)), LD-DPO (Fig.2(b), 2(c)) can smooth $\pi_{\theta}(y_{l}|x)/\pi_{\theta}(y_{w}|x)$, and this effect is markedly amplified as $\alpha$ diminishes. Based on the prior analysis and Eq.7, it is evident that the optimization direction of LD-DPO is less affected by the length disparity within the preferred data pairs. This indicates that, relative to DPO, LD-DPO has achieved a measure of length desensitization.

4 Experimental Setup

We follow the experimental setup of SimPO (Meng et al., 2024) to objectively demonstrate the validity of our method.

Models and training settings. We perform preference optimization using three families of models: Llama2-13B (Touvron et al., 2023), Llama3-8B (AI@Meta, 2024) and Qwen2-7B (Yang et al., 2024) under two setups: Base and Instruct/Chat.

For the Base setup, we train a base language model on the UltraChat-200k dataset (Ding et al., 2023) to obtain an SFT model, which possesses a basic capability for conversation. For the Instruct setup, we select their corresponding instruct models (i.e., Llama2-13B-Chat, Llama3-8B-Instruct, and Qwen2-7B-Instruct) as initial models. These models are more powerful and robust compared to the base models. Both setups ensure a high level of transparency as the models and training data are open source.

In the preference optimization phase, we utilize UltraFeedback(Cui et al., 2023) as the human preference dataset. This dataset consists of 60,000 high-quality data pairs $(x,y_{w},y_{l})$ designed to align with human conversational preferences and emphasize helpfulness.

Evaluation benchmarks. We primarily evaluate our models using two of the most popular open-ended evaluation benchmarks: MT-Bench (Zheng et al., 2024)and AlpacaEval 2 (Dubois et al., 2024). These benchmarks assess the model’s versatile session capabilities across a wide range of queries and have been widely adopted by the community.

MT-Bench comprises 80 questions spanning 8 categories, whereas AlpacaEval 2 encompasses 805 questions derived from 5 datasets. We present the results in accordance with the evaluation protocol designated for each benchmark. For MT-Bench, we present the average score and provide a detailed breakdown of the scores for each capability item in Appendix B.2. In the case of AlpacaEval 2, we report the length-controlled (LC) win rate against GPT-4-preview-1106, a metric specifically engineered to be resistant to model verbosity. All our evaluations are executed utilizing GPT4-turbo-0409 as the adjudicating model. Furthermore, we calculate the average response length on each benchmark to compare the effects of different methods on response length.

Baselines. We compare LD-DPO with five other offline preference optimization techniques. Among these, DPO serves as our most crucial comparison. R-DPO revises DPO by incorporating a length regularity term to control the response length. SamPO avoides reward overestimation or underestimation due to length by downsampling the KL dispersion. WPO simulates the on-policy learning process by adding weights to the optimization objective of DPO. SimPO introduces an optimization objective that does not rely on a reference model, and mitigates the impact of data length by utilizing average likelihood.

Phase	LR	BS	Epoch	LS	WP
SFT	2e-5	128	3	cosine	10%
PO	5e-7	32	1	cosine	10%

Table 1: General training hyperparameters settings for SFT phase and preference optimization (PO) phase, including Leaning Rate (LR), Batch Size (BS), Epoch, Learning rate Schedule (LS), Warmup Phase (WP).

General Training Hyperparameters. The training hyperparameters are shown in Table.1. Additionally, to ensure the performance of the offline preference optimization algorithms, we set the fitting tuning hyperparameters for all methods. In general, we set $\beta=0.1$ for DPO, R-DPO, SamPO and LD-DPO. Specifically, for SimPO, setting $\beta=2.0$ and $\gamma=1.0$, for R-DPO, setting $\alpha=0.05$, and for LD-DPO, we set $\alpha=\{0.1,0.2,...,1.0\}$ to explore its effect on generation length and model performance. Finally, all preference optimization training was conducted on 16 A100-80G GPUs.

5 Experimental Results

Method	Llama2-Base (13B)				Llama2-Chat (13B)
	MT-Bench		AlpacaEval 2		MT-Bench		AlpacaEval 2
	Score	Avg. Len	LC (%)	Avg. Len	Score	Avg. Len	LC (%)	Avg. Len
SFT	5.51	170	6.56	220	6.35	326	23.46	452
DPO	5.67	191	6.70	266	6.33	368	25.52	487
R-DPO	5.45	150	7.64	198	6.32	346	26.27	461
SimPO	5.45	180	7.31	246	6.40	351	26.38	471
WPO	5.76	185	9.65	244	6.40	401	26.81	486
SamPO	5.78	183	8.80	259	6.21	390	26.09	484
LD-DPO	5.83	154	10.37	208	6.55	329	28.20	449
Method	Llama3-Base (8B)				Llama3-Instruct (8B)
	MT-Bench		AlpacaEval 2		MT-Bench		AlpacaEval 2
	Score	Avg. Len	LC (%)	Avg. Len	Score	Avg. Len	LC (%)	Avg. Len
SFT	6.08	156	8.40	167	7.36	255	38.28	326
DPO	6.38	178	12.58	235	7.61	323	40.21	393
R-DPO	6.18	137	12.15	155	7.54	248	41.07	318
SimPO	6.24	142	9.96	194	7.36	266	39.14	374
WPO	6.42	179	12.99	226	7.60	320	39.77	386
SamPO	6.12	162	14.62	200	7.50	294	40.77	368
LD-DPO	6.45	153	16.82	144	7.74	247	44.00	308
Method	Qwen2-Base (7B)				Qwen2-Instruct (7B)
	MT-Bench		AlpacaEval 2		MT-Bench		AlpacaEval 2
	Score	Avg. Len	LC (%)	Avg. Len	Score	Avg. Len	LC (%)	Avg. Len
SFT	6.30	160	7.62	173	7.95	359	34.09	373
DPO	6.73	181	10.20	204	7.79	321	35.63	437
R-DPO	6.16	137	8.79	168	7.94	314	38.85	365
SimPO	6.61	154	12.08	181	7.88	352	35.10	430
WPO	6.71	167	11.02	193	7.72	361	37.53	433
SamPO	6.79	180	10.89	187	7.78	343	37.05	399
LD-DPO	6.80	163	12.14	155	8.03	303	40.88	356

Table 2: MT-Bench and AlpacaEval 2 results under six model settings. LC-winrate denotes length-controlled win rate against the baseline model (GPT-4-1106-preview), which can mitigate the length preference of the judge model (GPT-4-turbo-0409) compared to the raw win rate. Avg. Token denotes the average length of the model’s answers. For the Base settings, we train SFT models on the UltraChat dataset. For the Instruct settings, we use off-the-shelf models as the SFT model.

In this section, we present the main results of our experiments, demonstrating that LD-DPO achieves state-of-the-art (SOTA) performance on both MT-Bench and AlpacaEval for all six settings through effective length control. Building on these results, we further analyze the sensitivity of different models to data length. Additionally, our findings show that our method significantly enhances the model’s reasoning ability. Finally, we conduct ablation studys and hyperparameter analysis.

5.1 Main Results

As shown in Table.2, LD-DPO exhibits significant improvements in both MT-Bench and AlpacaEval 2 compared to all other baselines. In addition, the average response length is reduced by 7.8% to 37.9% relative to DPO, suggesting higher quality and more concise model outputs after LD-DPO.

In the Base setting, we observe that the overall model performance is suboptimal, with responses tending to be shorter. This phenomenon may be attributed to the model’s performance not being fully realized during the SFT phase. Conversely, in the Instruct setting, the model demonstrates greater competence and generates much longer responses than the base model, due to extensive SFT and RLHF conducted by their publishers. However, in both settings, it is clear that DPO consistently encourages the model to produce more verbose outputs, with experiments showing an increase ranging from 10% to 40%, while LD-DPO can significantly alleviate this issue. This verbosity can potentially impair model performance, as we will illustrate with several case studies in Appendix C.

As illustrated in Table.2, we examine the average response length and LC-winrate (which may vary from public results due to different judge model) of the models on AlpacaEval 2 under six different settings. Our method achieves the state-of-the-art (SOTA) performance in LC-winrate across all settings. When comparing the three length control methods (R-DPO, SimPO, and SamPO), We find that R-DPO demonstrates superior length control under the Base setting, but its overall performance is suboptimal in terms of LC-winrate. SimPO does not show consistent performance across different settings, likely due to the absence of a reference model, which impacts its stability. SamPO’s performance fluctuates less compared to DPO.

5.2 Comparative Analysis of Varied Model

(a) Llama2-13B-Chat (SOTA: $\alpha=0.3$)

(b) Llama3-8B-Instruct (SOTA: $\alpha=0.5$)

Figure 3: Exploring the distribution of reward differences $r(x,y_{w})-r(x,y_{l})$ in different settings: (a) Llama2-13B-Chat (SOTA: $\alpha=0.3$); (b) Llama3-8B-Instruct (SOTA: $\alpha=0.5$). In each subplot, the left image represents data where the chosen is longer, and the right image represents data where the rejected is longer. SOTA indicates the alpha setting where LD-DPO performs optimally. The images depict the true distribution on the UltraFeedback dataset during training.

The models selected for our experiments vary in their capabilities and, consequently, in their length sensitivity during the DPO process. Their performance based on the choice of the hyperparameter alpha is as follows:

In the Instruct setting, Llama2-13B achieves optimal performance at $\alpha=0.3$, Llama3-8B at $\alpha=0.5$, and Qwen2-7B at $\alpha=0.6$. In the Base setting, the optimal $\alpha$ values for these models are approximately 0.1 to 0.2 lower compared to the Instruct setting.

Due to the similarity in performance between Llama3-8B and Qwen2-7B, we subsequently just conducted an in-depth analysis of the Llama2-13B and Llama3-8B models to explore the performance differences between these two sizes. This selection is representative of varying model capacities.

In Fig.3, we plot the distribution of reward bias during preference optimization for Llama2-13B-Chat (Fig.3) and Llama3-8B-Instruct (Fig.3), respectively. The data is differentiated based on the length relationship between the chosen and rejected responses. We will analyze the training process of the LLMs with different hyperparameter settings:

Under the DPO setting, We can clearly observe that the distribution of reward differences is influenced by the length of the data. When the chosen is longer, the reward difference is almost always less than 0. Based on our previous theoretical analysis, we can infer that it is easier to optimize in the direction of the chosen under these conditions, leading to lengthy outputs. Conversely, when the rejected is longer, the situation is reversed.

Under the $\bm{\alpha=0}$ setting, where only the public length portions of the $y_{w}$ and $y_{l}$ are considered, we find that the reward difference is greater than 0 for a larger proportion of the data. This indicates that the models prefer outputting $y_{w}$, which suggests that both types of models possess sizable base capabilities, with Llama3-8B-Instruct being stronger than Llama2-13B-Chat. Additionally, compared to the DPO setting, the average reward difference of Llama2-13B-Chat increases (decreases) by 19.66 (14.01) and Llama3-8B-Instruct increases (decreases) by 18.37 (13.68), indicating that the former is more significantly affected by the data length.

Under the SOTA setting, The fact that LD-DPO cannot achieve optimal results in the extreme case of $\alpha=0$ suggests that longer responses are necessary. This is because additional text can convey more human-like preferences. Furthermore, compared to Llama2-13B-Chat, Llama3-8B-Instruct is more powerful and can capture more human-like preferences from the text. This capability can mitigate the negative effects of response length, indicating that setting $\alpha$ to an extreme value may not be appropriate.

From the above anaylsis, we know that $\alpha$ is actually the result of a compromise to achieve desensitization of DPO based on model capabilities and to prevent the loss of human-like preferences. In addition, we know that different models have different length sensitivities in the DPO process. We define $\gamma$ as the length sensitivity coefficient, where $\gamma=1-\alpha_{s}$ and $\alpha_{s}$ represents the $\alpha$ that yields the best LD-DPO results. A smaller value of $\gamma$ in more capable models indicates that such models are more likely to capture genuine human preferences rather than being influenced by text length. For example, the length sensitivity coefficient of Llama3-8B-Instruct is 0.5, whereas that of Llama2-13B-Chat is 0.7. This suggests that the latter is more sensitive to length during DPO.

5.3 Improvement of Reasoning Capability

In further analysis of the MT-Bench, we found that the reasoning ability of the model significantly improved after applying LD-DPO, compared to both the SFT and DPO models. To further validate this conclusion, we conducted experiments on ProofWriter using Llama3-8B-Instruct. ProofWriter is a specialized dataset designed to evaluate the reasoning capabilities of large language models. It comprises a broad range of problems, from direct reasoning tasks to those requiring more than five steps, and distinguishes between open-world assumptions (OWA) and closed-world assumptions (CWA), resulting in a total of 14 data subsets.

Figure 4: Performance of various methods on 14 subsets of ProofWriter dataset for the Llama3-8B-Instruct setting.

We conducted experiments on ProofWriter and the results are shown in Fig.4. The scatter plot in the figure indicates the scores for each data subset. After applying LD-DPO, the model shows an overall improvement across 14 data subsets. Compared to Llama3-8B-Instruct, the average score increases from 55.19 to 58.72, outperforming five classes of preference optimization algorithms, including DPO, indicating a significant improvement in reasoning. Detailed results of MT-Bench shown in the Appendix B.2 also prove this point, and a case study of the reasoning problems in Appendix C.

In fact, verbose responses negatively impact the reasoning abilities of language models, particularly smaller models. Unlike Chain of Thought (CoT), which is considered an excellent reasoning paradigm (Feng et al., 2024b), overly lengthy responses tend to include incorrect derivations or meaningless descriptions, which interfere with the model’s ability to make the next step in the reasoning process or to reach the correct conclusion. Our approach enables the model to learn a concise CoT style while preventing overly lengthy answers, thereby improving the model’s reasoning capabilities.

5.4 Ablation Study

To verify the effect of the relative lengths of $y_{w}$(chosen) and $y_{l}$(rejected) in the training data on LD-DPO, we constructed ablation experiments: The length decoupling strategy, as shown in Eq.12, was applied to $y_{w}$ and $y_{l}$ separately or simultaneously.

As shown in Table.3, the performance of the LLMs on MT-Bench and AlpacaEval, as well as the length control effect, is inferior to that of LD-DPO under both the chosen and rejected setups. This suggests that length decoupling is necessary for both chosen and rejected. For more detail, we have:

• If $y_{w}$ is longer: According to the previous analysis, it is known that DPO is more inclined to optimize in the direction of $y_{w}$ under the effect of length bias, which results in redundant output. At this point, length decoupling for $y_{w}$ can alleviate this tendency.

• If $y_{l}$ is longer: DPO tends to block the output of $y_{l}$. In this case, the decoupling of the length of $y_{l}$ may redirect the optimization towards a shorter $y_{w}$, thus reducing redundancy and improving the quality of the model’s answers.

Therefore, our length desensitization strategy can encourage the model to identify and leverage more human-real preference gaps in the data samples, rather than optimizing the training process based on superficial length differences.

Method	MT-Bench		AlpacaEval 2
	Score	Avg. Token	LC (%)	Avg. Token
SFT	7.36	255	38.28	326
DPO	7.61	323	40.21	393
chosen	7.67	302($\downarrow$21)	43.39	352($\downarrow$41)
rejected	7.62	283($\downarrow$40)	42.23	351($\downarrow$42)
LD-DPO	7.74	247($\downarrow$76)	44.00	308($\downarrow$85)

Table 3: Ablation study on Llama3-8B-Instruct: chosen denotes length decoupling (Eq.12) applied only to $y_{w}$, rejected denotes only to $y_{l}$, and $\downarrow$tokens denotes the Avg.Token drop compared to DPO.

5.5 Hyperparameter Analysis

In this subsection, we verify the effect of the hyperparameter $\alpha$ on the performance of LD-DPO. We present the experimental results of Llama3-8B-Instruct. In all experiments, we set $\beta=0.1$, though we encourage researchers to explore the effects of different $\alpha$ selected at various $\beta$ settings.

When $\alpha=1$, LD-DPO is equivalent to DPO. As $\alpha$ gradually decreases, the degree of length decoupling increases. At this point, as shown in Fig.5, the sensitivity of the training process to length begins to decline, resulting in a subsequent reduction in the average output length.

Figure 5: Hyperparametric analysis on $\alpha$ with Llama3-8B-Instruct on AlpacaEval 2. Left: Mean response length (tokens) Right: LC-Winrate on AlpacaEval 2

We find that the decrease in average response length is pronounced as the parameter $\alpha$ decreases from 1 to 0.5. However, this tendency becomes less significant as $\alpha$ further decreases from 0.5 to 0. This phenomenon can likely be attributed to the fact that verbosity preference decoupling is effectively complete when $\alpha$ reaches 0.5.

In terms of LC-winrate, the trend initially increases and then decreases with the value of $\alpha$. This suggests that when $\alpha$ is too large, the model performance is constrained by length sensitivity. Conversely, when $\alpha$ is too small, excessive length decoupling leads to a loss of human-like preferences in the text, thereby reducing the optimization effectiveness.

6 Related Works

6.1 Offline Preference Optimization

In practice, traditional RLHF paradigms are more complex in terms of coding and hyperparameter tuning, requiring four models simultaneously, which makes them more resource-intensive and difficult to train stably. Due to the lack of online reward models, DPO needs to construct artificial preference datasets in advance, and many works have proposed different data construction strategies to enable the model to better learn human preferences (An et al., 2023; Gallego, 2024; Khaki et al., 2024). Meanwhile, another research direction is to improve the preference optimization objective, including the necessity of the reference model, the selection of the reward fitting function, and the adjustment of the update weights, and derive a variety of offline optimization strategies, including ORPO (Hong et al., 2024), KTO (Ethayarajh et al., 2024), NCA (Chen et al., 2024b), IPO (Azar et al., 2024), WPO Zhou et al. (2024), etc.

6.2 Length Control Strategy

Recent research has shown that DPO may lead to biased results, such as models producing lengthy outputs, which affects the model’s ability to follow instructions and reasoning. To address this problem, Park et al. (2024) proposed R-DPO, which suppresses the model from producing excessively long answers by introducing a rule term, which is a intuitive scheme followed by several algorithms (Zhou et al., 2024; Chen et al., 2024a). Meng et al. (2024) proposed an optimization strategy that does not rely on the reference model, called SimPO, which uses length-normalized rewards to prevent the model from generating excessively long but low-quality answers. Lu et al. (2024) argued that the phenomenon of lengthy outputs in DPO is due to the overestimation or underestimation of implicit rewards caused by the length of the training data. Based on this, they proposed SamPO, which mitigates the length bias by down-sampling the KL divergence to ensure that implicit rewards are not affected by length.

7 Discussion

7.1 Conclusion

In this work, we propose for the first time that the optimization process of DPO is length-sensitive and provide a theoretical proof. Based on this, we design a length-desensitization algorithm based on DPO: LD-DPO, which achieves length desensitization by reparameterizing the likelihood to decouple verbosity preferences from complete information while preserving human-like preferences. Through extensive experimental analysis, LD-DPO consistently outperforms existing algorithms in various training settings, achieving performance improvements with a 10-40% reduction in output length, especially in reasoning ability. This suggests that previous optimization algorithms have overemphasized length at the expense of quality, validating the value of our work. Furthermore, we perform a comparative analysis of length sensitivity across models with different capabilities, which may provide new insights into the preference optimization process.

7.2 Limitations and Future Works

First, despite the empirical success and intuitive motivation of LD-DPO, the length-sensitive coefficient $\gamma$ for different models requires manual and experimental exploration. Future work could investigate methods to determine the optimal margins automatically. Second, length preference is among the most readily captured human preferences by models, we have not yet examined the decoupling of other preferences such as format preference and morphology preference during the training process. There is significant value in decoupling all of these preferences, and we will continue to explore along this path.

Appendix A Mathematical Derivations

A.1 Derivation of Optimization Direction in DPO

$$\begin{aligned} \mathcal{L}_{DPO}(\pi_{\theta};\pi_{ref})=-\mathbb{E}_{(x,y_{w},y_{l})\sim\mathcal{D}}[\log\sigma(\beta\log\frac{\pi_{\theta}(y_{w}|x)}{\pi_{ref}(y_{w}|x)}-\beta\log\frac{\pi_{\theta}(y_{l}|x)}{\pi_{ref}(y_{l}|x)})]\end{aligned}$$ (13)

(a) $\mathcal{L}_{DPO}(\mathcal{X}_{1};\mathcal{X}_{2})$

(b) $\partial\mathcal{L}_{DPO}(\mathcal{X}_{1};\mathcal{X}_{2})/\partial\mathcal{X}_{1}$

(c) $\partial\mathcal{L}_{DPO}(\mathcal{X}_{1};\mathcal{X}_{2})/\partial\mathcal{X}_{2}$

Figure 6: (a)The optimization objective of DPO (b)The partial derivative of $\mathcal{L}_{DPO}(\mathcal{X}_{1};\mathcal{X}_{2})$ with respect to $\mathcal{X}_{1}$ (c)The partial derivative of $\mathcal{L}_{DPO}(\mathcal{X}_{1};\mathcal{X}_{2})$ with respect to $\mathcal{X}_{2}$, where we denote $\pi_{\theta}(y_{w}|x)$ by $\mathcal{X}_{1}$ and $\pi_{\theta}(y_{l}|x)$ by $\mathcal{X}_{2}$.

The optimization objective of DPO is presented in Eq.13 and its image is shown in Fig.6(a). We define $\mathcal{X}_{1}=\pi_{\theta}(y_{w}|x),\mathcal{X}_{2}=\pi_{\theta}(y_{w}|x)$ and $\mathcal{K}_{1}=\pi_{\theta}(y_{w}|x),\mathcal{K}_{2}=\pi_{\theta}(y_{w}|x)$, where $\mathcal{K}_{1}$ and $\mathcal{K}_{2}$ can be regarded as constants and the optimization process of DPO is only related to $\mathcal{X}_{1},\mathcal{X}_{2}$. We can rewrite the loss of DPO as:

$$\begin{aligned} \mathcal{L}_{DPO}(\mathcal{X}_{1};\mathcal{X}_{2})=&-\log\sigma(\beta\log\frac{\mathcal{X}_{1}}{\mathcal{K}_{1}}-\beta\log\frac{\mathcal{X}_{2}}{\mathcal{K}_{2}})\\ =&-\log\frac{1}{1+exp\{-\beta\log(\mathcal{K}_{2}\mathcal{X}_{1}/\mathcal{K}_{1}\mathcal{X}_{2}\})}\\ =&-\log\frac{1}{1+(\mathcal{K}_{1}\mathcal{X}_{2}/\mathcal{K}_{2}\mathcal{X}_{1})^{\beta}}\\ =&-\log\frac{(\mathcal{K}_{2}\mathcal{X}_{1})^{\beta}}{(\mathcal{K}_{2}\mathcal{X}_{1})^{\beta}+(\mathcal{K}_{1}\mathcal{X}_{2})^{\beta}}.\end{aligned}$$ (14)

As shown in the Eq.15, the partial differentiation of $\mathcal{L}_{DPO}(\mathcal{X}_{1};\mathcal{X}_{2})$ with respect to $\mathcal{X}_{1}$ indicates that the model is oriented in the chosen direction, the function image is showen in Fig.6(b).

$$\begin{aligned} \frac{\partial\mathcal{L}_{DPO}(\mathcal{X}_{1};\mathcal{X}_{2})}{\partial\mathcal{X}_{1}}=&-\frac{\beta\mathcal{K}_{2}^{\beta}\mathcal{X}_{1}^{\beta-1}[(\mathcal{K}_{2}\mathcal{X}_{1})^{\beta}+(\mathcal{K}_{1}\mathcal{X}_{2})^{\beta}]-(\mathcal{K}_{2}\mathcal{X}_{1})^{\beta}\beta\mathcal{K}_{2}^{\beta}\mathcal{X}_{1}^{\beta-1}}{[(\mathcal{K}_{2}\mathcal{X}_{1})^{\beta}+(\mathcal{K}_{1}\mathcal{X}_{2})^{\beta}]^{2}}\frac{(\mathcal{K}_{2}\mathcal{X}_{1})^{\beta}+(\mathcal{K}_{1}\mathcal{X}_{2})^{\beta}}{(\mathcal{K}_{2}\mathcal{X}_{1})^{\beta}}\\ =&-\frac{\beta\mathcal{K}_{2}^{\beta}\mathcal{X}_{1}^{\beta-1}(\mathcal{K}_{2}\mathcal{X}_{1})^{\beta}+\beta\mathcal{K}_{2}^{\beta}\mathcal{X}_{1}^{\beta-1}(\mathcal{K}_{1}\mathcal{X}_{2})^{\beta}-(\mathcal{K}_{2}\mathcal{X}_{1})^{\beta}\beta\mathcal{K}_{2}^{\beta}\mathcal{X}_{1}^{\beta-1}}{[(\mathcal{K}_{2}\mathcal{X}_{1})^{\beta}+(\mathcal{K}_{1}\mathcal{X}_{2})^{\beta}](\mathcal{K}_{2}\mathcal{X}_{1})^{\beta}}\\ =&-\frac{\beta\mathcal{K}_{2}^{\beta}\mathcal{X}_{1}^{\beta-1}(\mathcal{K}_{1}\mathcal{X}_{2})^{\beta}}{[(\mathcal{K}_{2}\mathcal{X}_{1})^{\beta}+(\mathcal{K}_{1}\mathcal{X}_{2})^{\beta}](\mathcal{K}_{2}\mathcal{X}_{1})^{\beta}}\\ =&-\frac{\beta(\mathcal{K}_{1}\mathcal{X}_{2})^{\beta}}{\mathcal{X}_{1}[(\mathcal{K}_{2}\mathcal{X}_{1})^{\beta}+(\mathcal{K}_{1}\mathcal{X}_{2})^{\beta}]}\end{aligned}$$ (15)

As shown in the Eq.16, the partial differentiation of $\mathcal{L}_{DPO}(\mathcal{X}_{1};\mathcal{X}_{2})$ with respect to $\mathcal{X}_{2}$ indicates that the model is oriented in the rejected direction, the function image is showen in Fig.6(c).

$$\begin{aligned} \frac{\partial\mathcal{L}_{DPO}(\mathcal{X}_{1};\mathcal{X}_{2})}{\partial\mathcal{X}_{2}}=&-\frac{-(\mathcal{K}_{2}\mathcal{X}_{1})^{\beta}\beta\mathcal{K}_{1}^{\beta}\mathcal{X}_{2}^{\beta-1}}{[(\mathcal{K}_{2}\mathcal{X}_{1})^{\beta}+(\mathcal{K}_{1}\mathcal{X}_{2})^{\beta}]^{2}}\frac{(\mathcal{K}_{2}\mathcal{X}_{1})^{\beta}+(\mathcal{K}_{1}\mathcal{X}_{2})^{\beta}}{(\mathcal{K}_{2}\mathcal{X}_{1})^{\beta}}\\ =&\frac{\beta\mathcal{K}_{1}^{\beta}\mathcal{X}_{2}^{\beta-1}}{(\mathcal{K}_{2}\mathcal{X}_{1})^{\beta}+(\mathcal{K}_{1}\mathcal{X}_{2})^{\beta}}\end{aligned}$$ (16)

According to the Eq.15 and Eq.16, we can obtain Eq.17, which implies that the value of the gradient of the DPO loss function in the preference direction, compared to the non-preference direction, is inversely proportional to its prediction probability.

$$\begin{aligned} \left\lvert\frac{\partial\mathcal{L}_{DPO}(\mathcal{X}_{1};\mathcal{X}_{2})}{\partial\mathcal{X}_{1}}/\frac{\partial\mathcal{L}_{DPO}(\mathcal{X}_{1};\mathcal{X}_{2})}{\partial\mathcal{X}_{2}}\right\rvert=\frac{\mathcal{X}_{2}}{\mathcal{X}_{1}}=\frac{\pi_{\theta}(y_{l}|x)}{\pi_{\theta}(y_{w}|x)}.\end{aligned}$$ (17)

Next, we analyze the relationship between the partial gradient values $\frac{\partial\mathcal{L}_{DPO}(\mathcal{X}_{1};\mathcal{X}_{2})}{\partial\mathcal{X}_{1}}$,$\frac{\partial\mathcal{L}_{DPO}(\mathcal{X}_{1};\mathcal{X}_{2})}{\partial\mathcal{X}_{2}}$ and $\mathcal{X}_{1},\mathcal{X}_{2}$, and we denote $\frac{\partial\mathcal{L}_{DPO}(\mathcal{X}_{1};\mathcal{X}_{2})}{\partial\mathcal{X}_{1}}$ as $\mathcal{G}_{1}(\mathcal{X}_{1};\mathcal{X}_{2})$ and $\frac{\partial\mathcal{L}_{DPO}(\mathcal{X}_{1};\mathcal{X}_{2})}{\partial\mathcal{X}_{2}}$ as $\mathcal{G}_{2}(\mathcal{X}_{1};\mathcal{X}_{2})$. In addition, it is known that $\mathcal{X}_{1},\mathcal{X}_{2},\mathcal{K}_{1},\mathcal{K}_{2}$ all represent likelihood, each taking values in the range $(0,1)$, and $\beta$ is a hyperparameter that is range from $(0,1)$ in DPO. We solve for the partial derivatives with respect to $\mathcal{X}_{1}$ and $\mathcal{X}_{2}$ for $\mathcal{G}_{1}(\mathcal{X}_{1};\mathcal{X}_{2})$ and $\mathcal{G}_{2}(\mathcal{X}_{1};\mathcal{X}_{2})$ between Eq.18 and Eq.21, and their images are shown in Fig.7.

(a) $\frac{\partial\mathcal{G}_{1}(\mathcal{X}_{1};\mathcal{X}_{2})}{\partial\mathcal{X}_{1}}$

(b) $\frac{\partial\mathcal{G}_{1}(\mathcal{X}_{1};\mathcal{X}_{2})}{\partial\mathcal{X}_{2}}$

(c) $\frac{\partial\mathcal{G}_{2}(\mathcal{X}_{1};\mathcal{X}_{2})}{\partial\mathcal{X}_{1}}$

(d) $\frac{\partial\mathcal{G}_{2}(\mathcal{X}_{1};\mathcal{X}_{2})}{\partial\mathcal{X}_{1}}$

Figure 7: (a)The partial derivative of $\mathcal{G}_{1}(\mathcal{X}_{1};\mathcal{X}_{2})$ with respect to $\mathcal{X}_{1}$; (b)The partial derivative of $\mathcal{G}_{1}(\mathcal{X}_{1};\mathcal{X}_{2})$ with respect to $\mathcal{X}_{2}$; (c)The partial derivative of $\mathcal{G}_{2}(\mathcal{X}_{1};\mathcal{X}_{2})$ with respect to $\mathcal{X}_{1}$; (d)The partial derivative of $\mathcal{G}_{2}(\mathcal{X}_{1};\mathcal{X}_{2})$ with respect to $\mathcal{X}_{2}$.

$$\begin{aligned} \frac{\partial{\mathcal{G}_{1}(\mathcal{X}_{1};\mathcal{X}_{2})}}{\partial{\mathcal{X}_{1}}}=&\frac{\beta(\mathcal{K}_{1}\mathcal{X}_{2})^{\beta}[(\mathcal{K}_{2}\mathcal{X}_{1})^{\beta}+(\mathcal{K}_{1}\mathcal{X}_{2})^{\beta}+\mathcal{X}_{1}\beta\mathcal{K}_{2}^{\beta}\mathcal{X}_{1}^{\beta-1}]}{\mathcal{X}_{1}^{2}[(\mathcal{K}_{2}\mathcal{X}_{1})^{\beta}+(\mathcal{K}_{1}\mathcal{X}_{2})^{\beta}]^{2}}\\ =&\frac{\beta(\mathcal{K}_{1}\mathcal{X}_{2})^{2\beta}+\beta(\beta+1)(\mathcal{K}_{1}\mathcal{X}_{2})^{\beta}(\mathcal{K}_{2}\mathcal{X}_{1})^{\beta}}{\mathcal{X}_{1}^{2}[(\mathcal{K}_{2}\mathcal{X}_{1})^{\beta}+(\mathcal{K}_{1}\mathcal{X}_{2})^{\beta}]^{2}}>0\end{aligned}$$ (18)

$$\begin{aligned} \frac{\partial{\mathcal{G}_{1}(\mathcal{X}_{1};\mathcal{X}_{2})}}{\partial{\mathcal{X}_{2}}}=&-\frac{\beta^{2}\mathcal{K}_{1}^{\beta}\mathcal{X}_{2}^{\beta-1}\mathcal{X}_{1}[(\mathcal{K}_{2}\mathcal{X}_{1})^{\beta}+(\mathcal{K}_{1}\mathcal{X}_{2})^{\beta}]-\beta(\mathcal{K}_{1}\mathcal{X}_{2})^{\beta}\beta\mathcal{K}_{1}^{\beta}\mathcal{X}_{1}\mathcal{X}_{2}^{\beta-1}}{\mathcal{X}_{1}^{2}[(\mathcal{K}_{2}\mathcal{X}_{1})^{\beta}+(\mathcal{K}_{1}\mathcal{X}_{2})^{\beta}]^{2}}\\ =&-\frac{\beta^{2}\mathcal{K}_{1}^{\beta}\mathcal{X}_{2}^{\beta-1}\mathcal{X}_{1}(\mathcal{K}_{2}\mathcal{X}_{1})^{\beta}}{\mathcal{X}_{1}^{2}[(\mathcal{K}_{2}\mathcal{X}_{1})^{\beta}+(\mathcal{K}_{1}\mathcal{X}_{2})^{\beta}]^{2}}<0\end{aligned}$$ (19)

$$\begin{aligned} \frac{\partial{\mathcal{G}_{2}(\mathcal{X}_{1};\mathcal{X}_{2})}}{\partial{\mathcal{X}_{1}}}=&-\frac{\beta\mathcal{K}_{1}^{\beta}\mathcal{X}_{2}^{\beta-1}\beta\mathcal{K}_{2}^{\beta}\mathcal{X}_{1}^{\beta-1}}{[(\mathcal{K}_{2}\mathcal{X}_{1})^{\beta}+(\mathcal{K}_{1}\mathcal{X}_{2})^{\beta}]^{2}}\\ =&-\frac{\beta^{2}\mathcal{K}_{1}^{\beta}\mathcal{X}_{2}^{\beta-1}\mathcal{K}_{2}^{\beta}\mathcal{X}_{1}^{\beta-1}}{[(\mathcal{K}_{2}\mathcal{X}_{1})^{\beta}+(\mathcal{K}_{1}\mathcal{X}_{2})^{\beta}]^{2}}<0\end{aligned}$$ (20)

$$\begin{aligned} \frac{\partial{\mathcal{G}_{2}(\mathcal{X}_{1};\mathcal{X}_{2})}}{\partial{\mathcal{X}_{2}}}=&\frac{\beta(\beta-1)\mathcal{K}_{1}^{\beta}\mathcal{X}_{2}^{\beta-2}[(\mathcal{K}_{2}\mathcal{X}_{1})^{\beta}+(\mathcal{K}_{1}\mathcal{X}_{2})^{\beta}]-\beta\mathcal{K}_{1}^{\beta}\mathcal{X}_{2}^{\beta-1}\beta\mathcal{K}_{1}^{\beta}\mathcal{X}_{2}^{\beta-1}}{[(\mathcal{K}_{2}\mathcal{X}_{1})^{\beta}+(\mathcal{K}_{1}\mathcal{X}_{2})^{\beta}]^{2}}\\ =&\frac{\beta(\beta-1)\mathcal{K}_{1}^{\beta}\mathcal{X}_{2}^{\beta-2}\mathcal{K}_{2}^{\beta}\mathcal{X}_{1}^{\beta}-\beta\mathcal{K}_{1}^{2\beta}\mathcal{K}_{2}^{2\beta-2}}{[(\mathcal{K}_{2}\mathcal{X}_{1})^{\beta}+(\mathcal{K}_{1}\mathcal{X}_{2})^{\beta}]^{2}}<0\end{aligned}$$ (21)

Based on the analysis of the aforementioned function trend, we can draw the following conclusion: When $\mathcal{X}_{1}$ decreases, both $\left\lvert\frac{\partial\mathcal{L}_{DPO}(\mathcal{X}_{1};\mathcal{X}_{2})}{\partial\mathcal{X}_{1}}\right\lvert$ and $\left\lvert\frac{\partial\mathcal{L}_{DPO}(\mathcal{X}_{1};\mathcal{X}_{2})}{\partial\mathcal{X}_{2}}\right\lvert$ increase. When $\mathcal{X}_{2}$ decreases, $\left\lvert\frac{\partial\mathcal{L}_{DPO}(\mathcal{X}_{1};\mathcal{X}_{2})}{\partial\mathcal{X}_{1}}\right\lvert$ decreases and $\left\lvert\frac{\partial\mathcal{L}_{DPO}(\mathcal{X}_{1};\mathcal{X}_{2})}{\partial\mathcal{X}_{2}}\right\lvert$ increases. According to the analysis in Subsection 2, when the training data is more extensive, the parameters will be updated at a faster rate, thereby exacerbating the sensitivity of the DPO.

A.2 Derivation of the Modified Likelihood in LD-DPO

$$\begin{aligned} \hat{\pi}_{\theta}(y|x)=&\prod_{i=1}^{l_{p}}p(y_{i}|x,y_{<i})\prod_{i=l_{p}+1}^{l}p^{\alpha}(y_{i}|x,y_{<i})\\ =&\prod_{i=1}^{l_{p}}p^{\alpha}(y_{i}|x,y_{<i})p^{1-\alpha}(y_{i}|x,y_{<i})\prod_{i=l_{p}+1}^{l}p^{\alpha}(y_{i}|x,y_{<i})\\ =&\prod_{i=1}^{l_{p}}p^{\alpha}(y_{i}|x,y_{<i})\prod_{i=l_{p}+1}^{l}p^{\alpha}(y_{i}|x,y_{<i})\prod_{i=1}^{l_{p}}p^{1-\alpha}(y_{i}|x,y_{<i})\\ =&\prod_{i=1}^{l}p^{\alpha}(y_{i}|x,y_{<i})\prod_{i=1}^{l_{p}}p^{1-\alpha}(y_{i}|x,y_{<i})\end{aligned}$$ (22)

Method	BBH	WinoGrande	CSQA	ARC	MMLU	HellaSwag	ProofWriter	Average
Llama2-13B-Base
SFT	37.27	55.25	66.25	74.40	50.79	39.00	48.69	53.09
DPO	37.55	54.75	66.00	74.68	50.62	38.75	47.89	52.89
R-DPO	37.14	53.75	65.75	73.68	50.59	43.50	48.19	53.23
SimPO	32.47	52.00	65.00	74.04	50.42	40.25	47.92	51.73
WPO	36.83	55.25	66.25	74.22	50.52	38.00	47.67	52.68
SamPO	36.85	54.25	65.50	74.58	50.31	39.50	48.50	52.78
LD-DPO	37.63	54.75	66.50	74.76	50.93	44.25	48.83	53.95
Llama2-13B-Chat
SFT	44.88	48.75	65.75	74.26	55.39	50.25	47.08	55.19
DPO	45.00	49.00	66.25	73.97	55.50	50.75	47.72	55.46
R-DPO	44.41	51.00	65.25	73.65	54.78	52.25	47.42	55.54
SimPO	44.85	50.00	64.25	72.04	54.74	49.50	47.50	54.70
WPO	44.63	49.50	65.50	74.08	55.19	49.75	47.56	55.17
SamPO	44.77	49.00	65.25	73.90	55.45	51.00	47.94	55.33
LD-DPO	44.83	52.00	65.75	73.97	55.35	54.00	47.89	56.23
Llama3-8B-Base
SFT	45.05	54.75	72.75	84.79	61.70	41.50	52.75	59.04
DPO	45.73	55.50	71.75	84.72	62.41	45.25	48.89	59.18
R-DPO	45.27	53.75	71.50	84.68	62.52	48.00	50.61	59.47
SimPO	45.18	55.25	70.00	84.54	62.14	45.50	51.56	59.18
WPO	45.43	54.00	72.25	84.68	62.38	45.25	50.50	59.21
SamPO	45.05	55.00	72.25	84.58	62.57	46.25	49.61	59.33
LD-DPO	45.72	55.25	73.00	84.68	62.30	49.75	51.39	60.30
Llama3-8B-Instruct
SFT	61.56	60.00	76.50	88.43	69.27	63.00	55.19	67.71
DPO	60.80	59.00	76.25	87.68	67.50	63.25	56.14	67.23
R-DPO	60.03	55.75	76.25	87.15	66.54	63.00	57.19	66.56
SimPO	58.55	56.00	76.00	86.90	66.77	63.00	53.69	65.84
WPO	61.09	59.00	76.75	88.00	68.95	63.00	56.47	67.61
SamPO	61.16	57.75	76.75	87.72	67.66	62.50	57.42	67.28
LD-DPO	61.75	58.75	77.50	87.83	68.42	64.50	58.72	68.21
Qwen2-7B-Base
SFT	49.51	65.25	79.00	89.68	69.55	57.25	54.50	66.40
DPO	50.08	64.75	79.25	89.90	68.92	58.75	51.64	66.18
R-DPO	48.94	64.75	78.50	89.36	68.91	59.25	51.58	65.91
SimPO	47.94	64.00	79.00	89.57	68.77	58.25	53.42	65.85
WPO	49.66	64.75	79.25	89.75	68.72	57.75	54.06	66.28
SamPO	49.62	65.00	79.50	89.97	68.83	58.75	52.97	66.38
LD-DPO	49.79	65.25	79.50	90.43	69.05	60.50	54.61	67.02
Qwen2-7B-Instruct
SFT	57.56	68.50	80.50	90.79	71.15	71.75	58.14	71.11
DPO	57.85	68.00	80.25	90.89	71.41	71.00	59.11	71.22
R-DPO	56.26	68.50	80.75	90.61	70.74	70.75	58.31	70.85
SimPO	58.10	68.25	80.75	90.65	71.11	74.00	58.03	71.64
WPO	58.15	68.75	80.75	90.72	71.10	71.75	57.97	71.31
SamPO	58.33	68.00	80.25	90.79	71.41	71.00	59.58	71.34
LD-DPO	59.18	68.75	81.75	91.07	71.16	71.25	59.69	71.84

Table 4: Results on downstream tasks on the Huggingface OpenLLM Leaderboard.

Appendix B Additional Experimental Results

B.1 Results on Downstream Tasks

We evaluate the performances of LD-DPO and all baselines on various tasks on OpenLLM leaderboard (Beeching et al., 2023), including MMLU (Hendrycks et al., 2021), ARC (Clark et al., 2018), BBH (Suzgun et al., 2023), CommonsenseQA (Talmor et al., 2019), WinoGrande (Sakaguchi et al., 2021), HellaSwag (Zellers et al., 2019) and ProofWriter (Tafjord et al., 2021). The results are shown in Table 4, from where we find that:

• LD-DPO outperforms SFT, DPO, and all other baselines on the average score across all benchmarks in all settings.

• Preference optimization, whose goal is to align LLMs with human preference, may not significantly improve the performance on all downstream tasks.

• All preference optimization methods perform comparably to SFT model on MMLU and CSQA, with slight fluctuations, showing that knowledge is maintained during the preference optimization stage.

• Compared to SFT, DPO, and other baselines, LD-DPO significantly improves the performance on HellaSwag and ProofWriter, indicating that LD-DPO can enhance the reasoning capability of LLMs.

Method	Writing	Roleplay	Reasoning	Math	Coding	Extraction	STEM	Humanities	Average
Llama2-13B-Base
SFT	8.15	6.20	4.35	1.65	2.95	6.75	5.90	8.15	5.51
DPO	7.55	7.05	4.95	1.70	3.00	6.55	6.30	8.30	5.67
R-DPO	8.05	5.90	3.80	2.10	2.90	6.35	6.10	8.40	5.45
SimPO	7.95	6.30	4.60	1.70	3.05	6.60	5.75	8.05	5.50
WPO	7.75	6.70	4.60	1.80	2.85	7.55	6.30	8.55	5.76
SamPO	7.85	6.55	4.75	1.35	3.05	8.05	6.40	8.25	5.78
LD-DPO	7.80	6.50	4.75	1.60	3.65	7.40	6.55	8.45	5.83
Llama2-13B-Chat
SFT	8.45	7.20	5.35	3.30	2.75	7.30	7.50	9.00	6.35
DPO	7.90	7.50	5.40	2.95	3.15	7.05	7.30	9.40	6.33
R-DPO	8.75	7.05	5.65	3.05	3.00	7.25	6.80	9.05	6.32
SimPO	8.90	7.25	5.55	3.05	3.50	6.75	7.15	9.05	6.40
WPO	8.50	6.65	5.60	2.60	3.05	7.95	7.60	9.25	6.40
SamPO	8.20	7.05	5.45	2.70	3.00	7.85	6.55	8.90	6.21
LD-DPO	8.60	7.30	6.05	3.20	3.25	7.20	7.65	9.15	6.55
Llama3-8B-Base
SFT	7.80	6.25	3.90	3.05	4.25	8.40	6.55	8.50	6.08
DPO	7.95	6.80	4.05	3.20	4.45	8.75	7.25	8.65	6.38
R-DPO	8.30	6.30	4.00	2.65	4.10	8.30	7.45	8.35	6.18
SimPO	8.10	6.30	4.40	3.05	4.35	8.35	7.15	8.20	6.24
WPO	8.45	6.60	4.10	3.45	4.60	8.50	7.10	8.50	6.42
SamPO	8.15	6.40	4.40	2.75	4.30	8.50	6.40	8.20	6.12
LD-DPO	8.15	7.40	4.45	3.15	4.40	8.55	7.15	8.35	6.45
Llama3-8B-Instruct
SFT	8.95	8.45	4.60	5.05	5.35	9.10	8.10	9.30	7.36
DPO	9.05	8.55	5.55	5.30	5.45	9.00	8.70	9.35	7.61
R-DPO	8.85	8.00	5.75	5.50	6.15	8.75	7.70	9.65	7.54
SimPO	9.05	7.40	5.45	5.30	5.75	8.60	7.90	9.45	7.36
WPO	8.20	8.55	5.50	5.20	6.20	9.25	8.50	9.40	7.60
SamPO	9.20	8.65	5.30	3.55	6.15	9.25	8.45	9.50	7.50
LD-DPO	8.95	8.35	5.90	5.35	6.75	8.70	8.55	9.40	7.74
Qwen2-7B-Base
SFT	7.45	6.35	4.65	6.05	4.45	6.85	6.50	8.15	6.30
DPO	7.65	7.30	4.70	6.95	5.00	7.35	6.65	8.25	6.73
R-DPO	7.40	6.30	4.50	6.30	3.95	6.70	6.15	8.00	6.16
SimPO	7.75	6.60	4.70	5.95	5.45	6.60	7.60	8.30	6.61
WPO	7.85	6.75	4.85	6.75	5.10	6.75	7.25	8.45	6.71
SamPO	7.95	7.35	4.85	6.70	4.50	7.15	7.20	8.65	6.79
LD-DPO	8.20	6.85	5.05	7.45	4.85	6.70	6.80	8.50	6.80
Qwen2-7B-Instruct
SFT	9.10	8.95	6.30	6.35	6.45	8.05	9.05	9.40	7.95
DPO	9.15	9.05	6.45	6.40	4.85	7.90	9.05	9.45	7.79
R-DPO	8.80	8.50	6.80	6.25	6.10	8.60	8.90	9.60	7.94
SimPO	8.85	8.90	6.20	6.55	6.00	8.00	9.05	9.50	7.88
WPO	9.00	8.80	6.80	6.40	4.90	8.10	8.20	9.55	7.72
SamPO	8.90	8.80	6.50	6.15	5.70	8.45	8.35	9.40	7.78
LD-DPO	8.90	8.55	7.25	6.25	5.90	8.75	9.05	9.55	8.03

Table 5: Scores for each capability item on the MT-Bench.

B.2 MT-Bench Result

We provide a detailed presentation of the MT-Bench results in Table 5. MT-Bench comprises 80 questions specifically designed to evaluate the model’s proficiency across 8 dimensions: writing, roleplay, reasoning, math, coding, extraction, stem, humanities. To further assess the model’s capability in multi-round interactions and bolster the validity of the results, each question in MT-Bench undergoes two rounds of Q&A. The scores reported for each dimension are the averages derived from these two rounds.

Generally, we find that LD-DPO outperforms SFT, DPO, and all other baselines on average in all settings. In the case of slight fluctuations in performance across other dimensions, LD-DPO significantly outperforms all baselines in reasoning, indicating that LD-DPO can enhance the reasoning capability of LLMs.

Appendix C Case Studies

In Fig.8, we present two examples from AlpacaEval 2, where LD-DPO generates comparably accurate outputs with less tokens compared to vanilla DPO, showing that LD-DPO can generate more concise outputs without sacrificing performance. Since longer outputs are not necessarily more accurate or informative, sometimes they may contain more redundant and unimportant information, making the outputs verbose and lengthy. We believe that providing concise and clear outputs can also reflect the expressive capability of LLMs.

We also present two examples from ProofWriter in Fig.9, where LD-DPO generates more concise and direct chain-of-thought (CoT). From the cases we find that DPO generates complex and verbose CoTs, ultimately resulting in incorrect answers. By contrast, LD-DPO provides more direct and clearer CoTs, resulting in correct answers. The experimental results show that overly lengthy CoTs may detract from the accuracy of reasoning, whereas LD-DPO can improve the reasoning capability of LLMs by producing more concise and clearer CoTs.

Figure 8: Comparing generations of AlpacaEval 2 prompts from Llama3-8B trained based on DPO and LD-DPO.

Figure 9: Comparing generations of ProofWriter prompts from Llama3-8B trained based on DPO and LD-DPO.