New Desiderata for Direct Preference Optimization
Abstract
Large language models in the past have typically relied on some form of reinforcement learning with human feedback (RLHF) to better align model responses with human preferences. However, because of oft-observed instabilities when implementing these RLHF pipelines, various reparameterization techniques have recently been introduced to sidestep the need for separately learning an RL reward model. Instead, directly fine-tuning for human preferences is achieved via the minimization of a single closed-form training objective, a process originally referred to as direct preference optimization (DPO) and followed by several notable descendants. Although effective in certain real-world settings, we introduce new evaluation criteria that serve to highlight unresolved shortcomings in the ability of existing DPO methods to interpolate between a pre-trained reference model and empirical measures of human preferences, as well as unavoidable trade-offs in how low- and high-quality responses are regularized and constraints are handled. Our insights then motivate an alternative DPO-like loss that provably mitigates these limitations. Empirical results serve to corroborate notable aspects of our analyses111A preliminary version of this work appears in the ICML 2024 Workshop on Models of Human Feedback for AI Alignment: https://openreview.net/pdf?id=Fgf0iAOb22.
1 Introduction
Although pre-trained large language models (LLMs) often display remarkable capabilities [9, 11, 26, 44], it is well-established that they are prone to responding in ways that may be at odds with human preferences for rationale discourse [5, 17]. To this end, after an initial supervised fine-tuning phase that produces a reference model or policy , it is now commonplace to apply reinforcement learning with human feedback (RLHF) to further refine the LLM responses to input prompts [46, 38, 4, 27]. This multi-step process involves first learning a reward model that reflects human inclinations culled from labeled preference data, and then subsequently training a new policy that balances reward maximization with proximity to .
Because RLHF introduces additional complexity, computational overhead, and entry points for instability, clever reparameterization techniques have recently been proposed that sidestep the need for separately learning a reward model altogether. Instead, increased alignment with human preferences is achieved via the minimization of a single closed-form training objective, a process originally referred to as direct preference optimization (DPO) [33] followed by several notable descendants and generalizations [3, 40, 41, 45]. These alternatives dramatically economize model development; however, with recency comes the potential that the consequences of less obvious properties of DPO-based objectives may still be under-explored. It is along these lines that our attention herein lies, with the end goal of quantifying and steering model behavior in transparently beneficial directions.
After introducing basic concepts and the details of existing preference optimization models in Section 2, the remainder of the paper devoted to our technical contributions can be distilled as follows:
-
•
We introduce new evaluation desiderata that comport with intuition regarding how a preference model ideally should behave, and yet (somewhat surprisingly) are provably not satisfied by a broad class of existing DPO-based approaches. In particular, we show that because of uniform regularization effects, the minimizers of commonly-used preference optimization objectives like DPO are at times unable to preserve performance in regions where the reference model is strong while simultaneously improving upon the reference model elsewhere (Section 3.1). Moreover, we also elucidate limitations in the ability to interpolate between ideal endpoints as model trade-off parameters are varied (Section 3.2).
-
•
We prove that once inevitable learning constraints are introduced (explicitly or implicitly, e.g., early-stopping, weight decay, etc.), the core reparameterizations that underpin certain DPO models no longer strictly hold (Section 3.3). This motivates alternative justifications based solely on properties of the final loss functions involved (Appendices C and D).
-
•
Based on the above, we introduce a new preference optimization loss called that, by design, satisfies our evaluation desiderata while avoiding any dependency on constraint-dependent reparameterizations (Section 4). Properties of this loss relative to its precursors are also corroborated using Monte-Carlo simulations (Section 5 and Appendix A.2).
2 Background
We adopt to denote an input prompt drawn from some distribution . From here, conditioned on such prompts we may then generate responses using a pre-trained reference language model/policy . Moreover, given a pair of such responses , we adopt the binary indicator variable to convey that is preferred over by a human evaluator when , or else if instead . Given a population of such evaluators, we express the ground-truth human preference distribution as . And finally, we define a set of human labeled tuples drawn from a training distribution as
(1) |
where (subscripts here stand for ‘win’ and ‘lose’).222We generally assume that ; however, the case can nonetheless be handled by simply assigning , inclusion of which does not effect the analysis that follows. In particular, such cases merely introduce an irrelevant constant into the human preference loss functions under consideration. In other words, each training tuple is generated by drawing from , from the reference policy , and finally is produced by human labelers that operate according to . Note that per convention in prior work and ease of presentation, we will often abbreviate the preference distribution notation as when the context is sufficiently clear.
2.1 Reinforcement Learning with Human Feedback (RLHF)
Reward Function Estimation:
Given two candidate responses sampled using prompt , the Bradley-Terry (BT) model [8] for human preferences stipulates that
(2) |
where is a so-called latent reward model and is the logistic function. Because is unobservable, it is not possible to directly compute ; however, we can train an approximation (equivalent to as before) defined by a parameterized proxy reward . Specifically, we can minimize the loss
(3) |
The optimized reward can then be applied to fine-tuning the pre-trained reference model as described next.
RL Fine-Tuning with Estimated Reward Function:
The goal here is to improve upon a given using a separate trainable model , the high-level desiderata being: (i) Maximize the previously-estimated reward function when following , while (ii) Minimizing some measure of distance between and to avoid overfitting merely to preference rewards. These objectives typically materialize through the minimization of
(4) |
where is a trade-off parameter. Although not differentiable, starting from an initialization such as , the loss can be optimized over using various forms of RL [37, 34]
2.2 Direct Preference Optimization (DPO)
Consider now the reward-dependent RLHF loss from (4) defined w.r.t. and arbitrary reward function . DPO [33] is based on the observation that, provided is sufficiently flexible such that we may treat it as an arbitrary function for optimization purposes,333This is a key assumption with non-trivial consequences; Section 3.3 will explore this issue in further detail. the minimum of w.r.t. can be directly computed as
(5) |
where is the partition function ensuring that forms a proper distribution [31, 32]. From here, assuming , we can rearrange (5) to equivalently establish that
(6) |
Because thus far has remained unspecified, it naturally follows that these policy/reward relationships hold even for the ground-truth reward and the associated optimal policy . Hence instead of approximating with as in (2), we may equivalently approximate with some leading to the DPO loss
(7) |
noting that the partition function conveniently cancels out and can be excluded from further consideration. It is now possible to directly optimize (7) over using SGD without the need for any challenging RLHF procedure. The basic intuition here is that the parameterized policy induces an implicit reward that is being optimized via the original BT preference model. Moreover this equivalence is exact assuming data distributed as in (1).
2.3 Identity Preference Optimization (IPO)
Similar to DPO, the identity preference optimization (IPO) formulation [3] avoids both a 2-step learning process and cumbersome, potentially unstable RL training. To accomplish this, IPO is predicated on minimizing the original RLHF loss from (4) but with an alternative reward function. Specifically, the motivating IPO objective is to minimize , where
(8) |
over .444Note that in principle the distribution used to draw samples in defining need not be set to ; however, in practice is a typical choice, which we adopt throughout for simplicity. Because of the special structure of this particular reward function, it turns out that it is possible to minimize over without RL. In brief, this is accomplished by first noting that for any pair of responses the existence of an optimal IPO policy, denoted , evaluated at these responses can be computed as a function of the reward using (5). Combining and dependent terms, after a few algebraic manipulations this then leads to the equivalence relation
(9) |
However, unlike DPO where an analogous expression is inverted to create an implicit reward for integration within the BT model, IPO instead attempts to approximate this equivalence relation by replacing the unknown with some . Although technically is also unknown, given samples , it is nicely shown in [3] that
(10) |
provided follows from (1). Note that this closed-form consistency is a direct consequence of how is defined in (8) and will not generally hold for other choices of the reward function. Regardless, it is straightforward to minimize in its present form via SGD as with DPO.
2.4 Flexible Quasi-Convex Generalizations
From the expressions above, it is clear that both DPO and IPO reduce to functions of and a tunable hyperparameter . As such, it is natural to consider extensions to broader choices in the form
(11) |
where is a monotonically increasing function (which generalizes the logarithm), and the function influences the overall loss shape. We stipulate that is a differentiable quasi-convex function [20]; hence the chosen loss notation for quasi-convex preference optimization. By definition of quasi-convexity, monotonically increases to the right or left away from the minimum.
These specifications cover DPO and IPO as representative special cases, and include essentially all reasonable choices for a loss within this family, e.g., it is nonsensical to include multi-modal losses. The generalized preference optimization (GPO) [40] and f-DPO [41] frameworks are also special cases of QPO as defined herein. With GPO, is a logarithm and is chosen as an arbitrary convex function (such as used by SLiC [45]). Meanwhile f-DPO involves analogous to DPO but with , where denotes the derivative of an -divergence [36]; given that must be convex, its derivative will necessarily be monotonically increasing. In this way, the RLHF objective from (4) is still optimized via -DPO, but with an -divergence replacing the KL term.
3 Comparative Analysis of Existing Approaches
We now turn to comparative analysis of existing approaches, which all have ties relating back to the BT preference model. Throughout this section we say that a policy is BT-optimal at prompt if implies that for all response pairs with nonzero probability (as determined by the reference policy generating the preference data). Appendix F.1 introduces how can be formed.
3.1 Selective Preservation of Optimal Policies
Consider the following plausible scenario, variations of which are likely to occur (at least in varying degrees) with real-world data. Suppose the support of prompts generated by partitions as , with . Furthermore, assume we have access to a reference policy such that for and for , where is an arbitrary distance measure. In other words, when evaluated w.r.t. a policy that proportionally reflects human preferences, performs ideally on a subset of prompts but not on others.
This dichotomy provides a useful lens for examining certain loss function properties. In particular, we would like any policy that minimizes a candidate loss to preserve for prompts , while pushing away from towards for prompts . However, because of uniform regularization effects intrinsic to the QPO loss, it is not actually possible to achieve even this modest objective.
Theorem 1
(Informal version) Given the prompt partitioning, reference policy, and optimal policy described above, define for any fixed selection of . Then under relatively mild assumptions on the labeled responses in , if for , then for .
The proof and formal version are provided in Appendix E.1, while Figure 1(left) below provides an illustration. The somewhat unexpected implication here is that if we minimize any possible QPO loss in the form of (11) and improve the policy quality in areas where performs poorly w.r.t. , then it must also be the case that performance becomes worse in areas where was originally optimal. This phenomena represents an unavoidable trade-off when we restrict ourselves to using a QPO loss, of which DPO and IPO (as well as GPO and -DPO) are special cases inheriting the same limitation. The core issue here is that QPO losses unselectively apply the same regularization, starting from the same initialization point, to both good and bad cases relative to .
3.2 Interpolation Capabilities
As the underlying goal shared by all approaches is to balance proximity to a reference policy with respect for the human preference model , a non-negative trade-off parameter that allows for interpolating between these competing objectives is inevitable, where and are lower and upper bounds respectively.555Depending on the method, if or we may replace the range with an open set. In this section we examine more closely the nature of loss function minimizers as is varied, zooming in on their behavior in the limit as and . To this end, we first introduce the following definitions :
Definition 1
We say that an arbitrary preference optimization loss satisfies the strong interpolation criteria (SIC) if the following conditions hold:
-
1.
;
-
2.
;
-
3.
For all other , the optimal policy interpolates between the above two extremes.
Definition 2
For any prompt and response define666See Appendix F.1 for the derivation of the right-hand equality in (12).
(12) |
In this way, assigns probability one to the mode of , i.e., akin to a delta function with no generation diversity. We then say that a loss satisfies the weak interpolation criteria (WIC) analogously to the SIC, only for the lower bound we instead require .
In summary, the only difference between these interpolation criteria is their limiting behavior w.r.t. the lower bounding ; for the SIC we approach the BT-optimal policy, while for the WIC we approach a degenerate policy with all probability mass restricted to the mode of the BT-optimal policy. We remark that both the SIC and WIC cannot be simultaneously satisfied unless itself is a degenerate delta function. We now explore how these distinctions are reflected in the behavior of DPO and IPO loss minimizers, with Figure 1(middle) illustrating the basic concepts.
Proposition 1
In terms of practical applicability of this result, there exists one important caveat: the empirical distribution of a finite set of labeled preference data need not actually satisfy the conditions of Proposition 1. For example, suppose for each prompt we collect only two responses along with a single preference label , which together produce the tuple . In this scenario, which reflects certain publicly-available human preference datasets [4, 18], the empirical distribution of preferences will be for all . Notably, Proposition 1 will not hold, and in particular, it can be easily shown that minimizers of any valid -DPO loss will be completely independent of for all ; in other words, no interpolation occurs at all; see Appendix F.2 for the derivation. A similar observation specific to DPO (but not -DPO) can be found in [1]. The fact that DPO-based solutions may still reflect in practice, and more-so as increases, relates to implicit constraints and subtle regularization effects as discussed further in Section 3.3 and Appendix C.
Proposition 2
Comparing Proposition 2 with Proposition 1, we observe that IPO maintains its ability to interpolate under broader conditions than DPO, particularly in the empirical sampling regime involving binary probability values. That being said, neither DPO nor IPO satisfy the SIC, which motivates consideration of alternative losses that do, at least if our priority is to actually achieve the SIC (which of course may depend on the application scenario). For this purpose, it turns out that selections beyond the family of QPO objectives (which includes DPO, -DPO, and IPO) are necessary per the following:
Theorem 2
Section 4 will consider objectives outside of the QPO family which circumvent this limitation.
3.3 Impact of Optimization Constraints
Originally in [33], and later supported by follow-up analysis [3], it has been shown that minimizing the DPO loss is effectively the same as minimizing the RLHF loss with optimal reward model . But there is a pivotal assumption underlying this association which previous analysis has not rigorously accounted for. Specifically, the key equalities that facilitate the DPO and IPO reparameterizations, namely (6) and (9) (and the analogous for -DPO), are all predicated on the solution of an uncononstrained optimization problem over an arbitrary policy .
However, when actually training models in real-world settings, constraints will always exist, whether implicitly or explicitly. Such constraints stem from any number of factors including the model architecture/capacity limitations, early stopping, weight decay, drop-out regularization, machine precision, and so on. Hence in reality we are never exactly minimizing some preference loss over any possible (as assumed by DPO, IPO, and -DPO derivations). Instead, we must consider properties of the constrained problem , where is a constraint set. For example, if we restrict training to a single epoch with a fixed learning rate, then can be viewed as the set of all points reachable within a limited number of SGD updates.
Theorem 3
Let denote a constraint set on the learnable policy . Then we can have that
(13) |
As can be observed by the proof in Appendix E.5, the difference between the two is akin to the difference between applying a constraint to a trainable policy with respect to either the forward or backward KL divergence, which are generally quite distinct [7]; see also Figure 1(right). There are several important consequences of this result worth considering:
-
•
As discussed in Section 3.2, the DPO-based losses can have degenerate unconstrained minimizers that completely ignore on certain real-world datasets; therefore counter-measures like early stopping are imposed that effectively introduce a that dramatically alters the estimated policy. But in doing so, the inequality from (13) is introduced and so we can no longer say that DPO provides an optimal implicit reward for the original RLHF problem, i.e., the original connection is now ambiguous.
-
•
As such, the value of DPO in practice (and indeed it often does work well) cannot be unreservedly attributed to its original affiliation with an optimal RLHF solution, and instead, should be evaluated based on properties of . See Appendix C for one step in this direction.
-
•
To further illustrate the above points, in Appendix D we rederive the DPO loss from scratch based solely on a Gaussian estimation perspective that is completely unrelated to RLHF. But of course we do not actually believe that binary human preference data are really Gaussian. Instead, this exercise serves to highlight that what matters are properties of the underlying loss when deployed in practice, not necessarily the assumptions made in deriving the loss in the first place.
-
•
Other losses based on unconstrained RLHF-based reparameterizations in the -DPO and IPO families may be similarly influenced by the inevitable introduction of policy constraints.
![Refer to caption](x1.png)
4 New Objectives for Human Preference Optimization
Motivated by the analysis in Section 3 and illustrated in Figure 1, we next examine alternative objective functions adhering to the following desiderata:
-
1.
Perservation: Capable of selectively preserving an optimal policy in ideal regimes, while simultaneously improving the policy in regions of poor performance (from Section 3.1);
-
2.
Interpolation: Smoothly interpolates between the BT-optimal policy and the reference policy, i.e., it achieves the SIC (from Section 3.2);
-
3.
Constraints: Independent of any derivation or required equivalence/reparameterization that no longer holds upon the introduction of constraints (from Section 3.3).
We label the our new objective to highlight the potential ability to “tame your preference optimization” (and “lower typos”) by explicitly targeting these desiderata.
4.1 TYPO Objective Function
Consider a loss, composed of separable supervised and unsupervised factors, in the general form
(14) | |||
where serves as a supervised penalty over labeled training tuples while represents an additional regularization term independent of labeled preferences. We remark that objectives in the form of (14) are natural candidates for SGD given that all sampling is independent of , unlike the typical regularized loss adopted by RLHF, which requires samples from .
Supervised Term:
After first defining
(15) |
we then consider the supervised term
(16) | |||||
Please see Appendix F.3 for the derivation of this equivalence. Importantly here, because the KL-divergence is minimized iff , unlike an arbitrary reward, the optimal solution to will necessarily recover the BT-optimal distribution as will be analyzed below.
Unsupervised Term:
For the unsupervised term in (14) we simply adopt
(17) |
ignoring terms independent of . Like (16), this expression also does not require sampling from . That being said, (17) can exploit out-of-preference data (meaning unlabeled responses), and prior work [25] has argued for the merits of using such data in broader RLHF contexts. (It may also be reasonable to consider switching to a reverse-KL term and optimize with REINFORCE per general observations from [1]; however, we do not pursue this direction further here.)
4.2 Properties
Notable attributes of w.r.t. the three desiderata from above are as follows:
Proposition 3
Per this result, minimizers of are capable of preserving in regions where performance is strong relative to , while concurrently improving performance in other areas where it is not. Figure 1(left) visualizes this unique TYPO capability.
Figure 1(middle) contrasts this property with the WIC achieved by prior methods. We also remark that none of the derivations used to motivate rely on unconstrained optimization to form a reparameterized objective function as with DPO, -DPO, and IPO. As such, the inevitable introduction of such constraints in practice does not compromise the TYPO origin story. In other words, since TYPO is not based on any implicit association with RLHF in the first place, adding constraints that might otherwise compromise such an association pose no issue.
5 Empirical Validation
Although more of an analysis-driven contribution, our core insights from Sections 3 and 4 can nonetheless benefit from empirical corroboration. To this end, we first present a series of experiments adapted from [3] to highlight aspects of TYPO behavior vis-à-vis our proposed desiderata. As the most relevant published points of reference, we contrast with DPO, IPO, and -DPO; for the latter we choose the Jensen–Shannon divergence, which next to the reverse-KL implicitly assumed by DPO, performed well in prior experiments [41]. Later we test using the Anthropic Helpfulness and Harmlessness (HH) real-world preference dataset [4, 18]. For space considerations, some experiment details, including hyperparameters and training setups, are deferred to Appendix A.
Interpolation Tests:
As in [3] we consider the bandit setting with a discrete space of three responses/actions and create a dataset of labeled pairs as , i.e., a total ordering consistent with the BT model. Preferences are assigned via computed using (55) with , , and . Furthermore, again following [3] we form our trainable policy as with optimized using Adam over each different preference loss. Results using a small are shown in Figure 2, where we observe that TYPO closely converges to the BT-optimal solution, while DPO and IPO converge to (the mode of ) consistent with Propositions 1 (DPO), 2 (IPO), and 4 (TYPO), as well as Theorem 2 which applies to -DPO. Additional interpolation results traversing different towards the upper limit are presented in Appendix A.2.
![Refer to caption](x2.png)
Preservation Tests:
We next modify the setting from above to include two input prompts chosen such that and sampled with equal probability. We then specify the corresponding response space and prompt-dependent probabilities (see Appendix A.1). For the reference policy we set and . We generate pair-wise preference data as before, only now with prompt-dependent responses. Results shown in Figure 3(left & middle) are in direct accordance with Theorem 1 and Proposition 3, whereby TYPO is the only approach that preserves a strong policy with prompt while at the same time improving performance relative to for over all .
![Refer to caption](extracted/5726789/figures/preservation_constraint.png)
Constraint Tests:
We probe the extent to which learning constraints can interfere with the equivalence between DPO and RLHF implemented with an optimal reward function. To this end, we adopted the same data generation setup as in the interpolation experiments from above. We then train policies to separately minimize the right- and left-hand sides of (13), but with one key modification: we added an identical penalty function to both models to instantiate weight decay (a typical form of constraint used in practice), where is a tunable hyperparameter. Figure 3(right) plots the distance (-axis) between learned policies from RLHF and DPO as is varied. Consistent with the original DPO derivations and analysis from [33], we observe negligible error when given that unconstrained DPO is explicitly designed to mimic RLHF with an optimal reward . However, in accordance with our Theorem 3, as increases, the distance between RLHF and DPO grows considerably, and their relationship is no longer clear-cut.
![Refer to caption](x3.png)
Testing on Anthropic HH Preference Data:
Finally, to explore TYPO capabilities in a real-world scenario, we train a Pythia 2.8B model [6] on the Anthropic Helpfulness and Harmlessness (HH) preference dataset [4, 18] as previously used in [33]. Following their settings, we first execute supervised fine-tuning (SFT) on the Pythia model using values as the target response. We then use this SFT model as for training DPO, IPO and TYPO. Given that alignment results (our focus) from [41] already show that reverse KL (i.e., DPO) works best among -divergences, we do not compare with other -DPO selections here. We use GPT-4 to evaluate the win rate of the generated responses from each model against the chosen on the test set for single turn dialogues. We emphasize that our comparisons cover both helpfulness and harmlessness (see Appendix A.3), whereas the original DPO paper [33] only tests the former.
6 Conclusions
In this work we have proposed multiple desiderata that existing methodology for human preference optimization does not satisfy and yet our proposed TYPO approach does.
References
- [1] Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Ahmet Üstün, and Sara Hooker. Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms. arXiv preprint arXiv:2402.14740, 2024.
- [2] Afra Amini, Tim Vieira, and Ryan Cotterell. Direct preference optimization with an offset. arXiv preprint arXiv:2402.10571, 2024.
- [3] Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm to understand learning from human preferences. In International Conference on Artificial Intelligence and Statistics, pages 4447–4455. PMLR, 2024.
- [4] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022.
- [5] Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073, 2022.
- [6] Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pages 2397–2430. PMLR, 2023.
- [7] C.M. Bishop. Pattern recognition and machine learning. Springer, New York, 2006.
- [8] Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324–345, 1952.
- [9] Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang. Sparks of artificial general intelligence: Early experiments with gpt-4, 2023.
- [10] Emmanuel J Candes, Michael B Wakin, and Stephen P Boyd. Enhancing sparsity by reweighted l1 minimization. Journal of Fourier analysis and applications, 14:877–905, 2008.
- [11] Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. A survey on evaluation of large language models. ACM Transactions on Intelligent Systems and Technology, 15(3):1–45, 2024.
- [12] Rick Chartrand and Wotao Yin. Iteratively reweighted algorithms for compressive sensing. International Conference on Accoustics, Speech, and Signal Processing, 2008.
- [13] Yichen Chen, Dongdong Ge, Mengdi Wang, Zizhuo Wang, Yinyu Ye, and Hao Yin. Strong NP-hardness for sparse optimization with concave penalty functions. In International Confernece on Machine Learning, 2017.
- [14] Bin Dai, Chen Zhu, Baining Guo, and David Wipf. Compressing neural networks using the variational information bottleneck. In International Conference on Machine Learning, pages 1135–1144. PMLR, 2018.
- [15] Jianqing Fan and Runze Li. Variable selection via nonconcave penalized likelihood and its oracle properties. JASTA, 96(456):1348–1360, 2001.
- [16] Duanyu Feng, Bowen Qin, Chen Huang, Zheng Zhang, and Wenqiang Lei. Towards analyzing and understanding the limitations of dpo: A theoretical perspective. arXiv preprint arXiv:2404.04626, 2024.
- [17] Isabel O Gallegos, Ryan A Rossi, Joe Barrow, Md Mehrab Tanjim, Sungchul Kim, Franck Dernoncourt, Tong Yu, Ruiyi Zhang, and Nesreen K Ahmed. Bias and fairness in large language models: A survey. arXiv preprint arXiv:2309.00770, 2023.
- [18] Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, et al. Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned. arXiv preprint arXiv:2209.07858, 2022.
- [19] Alexey Gorbatovski, Boris Shaposhnikov, Alexey Malakhov, Nikita Surnachev, Yaroslav Aksenov, Ian Maksimov, Nikita Balagansky, and Daniil Gavrilov. Learn your reference model for real good alignment. arXiv preprint arXiv:2404.09656, 2024.
- [20] Harvey Greenberg and William Pierskalla. A review of quasi-convex functions. Operations research, 19(7):1553–1570, 1971.
- [21] Jiwoo Hong, Noah Lee, and James Thorne. Orpo: Monolithic preference optimization without reference model. arXiv preprint arXiv:2403.07691, 2024.
- [22] Shawn Im and Yixuan Li. Understanding the learning dynamics of alignment with human feedback. arXiv preprint arXiv:2403.18742, 2024.
- [23] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
- [24] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
- [25] Ziniu Li, Tian Xu, and Yang Yu. Policy optimization in rlhf: The impact of out-of-preference data. arXiv preprint arXiv:2312.10584v2, 2024.
- [26] OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett-Shapiro, Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Juston Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Gontijo-Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, Łukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Jan Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Łukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reiichiro Nakano, Rajeev Nayak, Arvind Neelakantan, Richard Ngo, Hyeonwoo Noh, Long Ouyang, Cullen O’Keefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokorny, Michelle Pokrass, Vitchyr H. Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Proehl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin Sokolowsky, Yang Song, Natalie Staudacher, Felipe Petroski Such, Natalie Summers, Ilya Sutskever, Jie Tang, Nikolas Tezak, Madeleine B. Thompson, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Preston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vijayvergiya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang, Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojciech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. Gpt-4 technical report, 2024.
- [27] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744, 2022.
- [28] Arka Pal, Deep Karkhanis, Samuel Dooley, Manley Roberts, Siddartha Naidu, and Colin White. Smaug: Fixing failure modes of preference optimisation with dpo-positive. arXiv preprint arXiv:2402.13228, 2024.
- [29] Jason Palmer. Relative convexity. UC San Diego Technical Report, 2003.
- [30] Ryan Park, Rafael Rafailov, Stefano Ermon, and Chelsea Finn. Disentangling length from quality in direct preference optimization. arXiv preprint arXiv:2403.19159, 2024.
- [31] Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weighted regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv:1910.00177, 2019.
- [32] Jan Peters and Stefan Schaal. Reinforcement learning by reward-weighted regression for operational space control. In Proceedings of the 24th international conference on Machine learning, pages 745–750, 2007.
- [33] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024.
- [34] Rajkumar Ramamurthy, Prithviraj Ammanabrolu, Kianté Brantley, Jack Hessel, Rafet Sifa, Christian Bauckhage, Hannaneh Hajishirzi, and Yejin Choi. Is reinforcement learning (not) for natural language processing: Benchmarks, baselines, and building blocks for natural language policy optimization. arXiv preprint arXiv:2210.01241, 2022.
- [35] B.D. Rao, K. Engan, S. F. Cotter, J. Palmer, and K. Kreutz-Delgado. Subset selection in noise based on diversity measure minimization. IEEE Trans. Signal Processing, 51(3):760–770, March 2003.
- [36] Paul Rubenstein, Olivier Bousquet, Josip Djolonga, Carlos Riquelme, and Ilya O Tolstikhin. Practical and consistent estimation of f-divergences. Advances in Neural Information Processing Systems, 32, 2019.
- [37] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
- [38] Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize from human feedback, 2020. URL https://arxiv. org/abs, 2009.
- [39] Fahim Tajwar, Anikait Singh, Archit Sharma, Rafael Rafailov, Jeff Schneider, Tengyang Xie, Stefano Ermon, Chelsea Finn, and Aviral Kumar. Preference fine-tuning of llms should leverage suboptimal, on-policy data. arXiv preprint arXiv:2404.14367, 2024.
- [40] Yunhao Tang, Zhaohan Daniel Guo, Zeyu Zheng, Daniele Calandriello, Rémi Munos, Mark Rowland, Pierre Harvey Richemond, Michal Valko, Bernardo Ávila Pires, and Bilal Piot. Generalized preference optimization: A unified approach to offline alignment. arXiv preprint arXiv:2402.05749, 2024.
- [41] Chaoqi Wang, Yibo Jiang, Chenghao Yang, Han Liu, and Yuxin Chen. Beyond reverse KL: Generalizing direct preference optimization with diverse divergence constraints. International Conference on Learning Representations, 2024.
- [42] David Wipf and Srikantan Nagarajan. Iterative reweighted and methods for finding sparse solutions. Journal of Selected Topics in Signal Processing (Special Issue on Compressive Sensing), 4(2), 2010.
- [43] David Wipf and Haichao Zhang. Revisiting Bayesian blind deconvolution. Journal of Machine Learning Research (JMLR), 2014.
- [44] Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian-Yun Nie, and Ji-Rong Wen. A survey of large language models. arXiv preprint arXiv:2303.18223, 2023.
- [45] Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J Liu. SLiC-HF: Sequence likelihood calibration with human feedback. arXiv preprint arXiv:2305.10425, 2023.
- [46] Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593, 2019.
Appendix A Additional Experimental Details and Results
This section describes experiment details/settings and additional results.
A.1 Details of the Tests with Synthetic Data
-
•
For the tests of interpolation, preservation and constraints, we train the models with Adam optimizer [23] and clip the gradients via a max norm of 10. And we run the experiments of the tests on a single A10 GPU. Unless otherwise mentioned, we use batch size of 1.
-
•
For the interpolation tests, we use batch size of 20 and choose , , and . We use learning rate of for DPO, IPO and -DPO and for TYPO; we train DPO, IPO and TYPO for 1,000 epochs and -DPO for 3,000 epochs as it converges slower.
-
•
For the preservation test, we choose
(18) And for the reference model we select , and . We randomly sample examples for good and bad prompts respectively. The model parameters are and we set the values of and as vectors of and .
-
•
In the constraint test, we use the same setting and data as the interpolation test. We use for both RLHF and DPO and train them for 100 epochs for all the values of .
A.2 Additional Results with Synthetic Data
We conduct additional experiments for the interpolation test by varying from very small to very large values as shown in Figure 5 and Figure 6.
![Refer to caption](x4.png)
![Refer to caption](x5.png)
A.3 Details of Experiments on Anthropic HH Dataset
We train the SFT model with 2 epochs and 1 epoch for all the other models with a learning rate of and batch size of 40. We set for DPO, for IPO and for TYPO. We evaluate the win rate on the single turn dialogues in the test set with GPT-4 using modified version used in the DPO paper to cover harmlessness examples as shown in Figure 7. All the experiments are conducted in a 8A100 40G GPU instance.
For the training of TYPO, we first sample responses from the reference model, i.e. the SFT model, for the unsupervised term. We apply vLLM [24] to randomly sample responses from the Anthropic HH dataset by setting temperature=1, top_k=60, top_p=0.8, max_tokens=256 and repetition_penalty=1.1. During the training, we use one sampled response for each prompt in the unsupervised term.
![Refer to caption](x6.png)
Appendix B Extended Related Work
There has been a flurry of interesting recent work on DPO-related topics, with numerous papers appearing on arXiv not long before the NeurIPS deadline. In this section we call attention to several notable examples that propose modifications of the original DPO paradigm, or else provide relevant analysis of its properties. We believe these efforts to be complementary to our contribution, as well as the existing DPO-like extensions by others discussed in the main body of paper.
Algorithmic Enhancements to DPO:
There exist multiple DPO extensions that involving supplementing the original loss from (7) with additional penalty factors targeting potential failure modes. For example, based on the observation that DPO may exhibit a decrease in accuracy when applied to preference data with small edit distances between responses, the Smaug framework [28] augments the DPO loss with an additional factor designed to maintain high log-likelihoods in such cases. Meanwhile, sensitivity to response lengths are investigated in [30], where as a counter-measure, the DPO loss is supplemented with a penalty on length differences between winning and losing responses. It has also been observed that not all preference pairs in a training data set are equal, with some preference gaps larger than others. As a mitigation strategy for this discrepancy, the ODPO approach [2] introduces a preference offset term during model training. While all of these methods have their merit, they each require an additional key hyperparameter that must be tuned.
Somewhat differently, the ORPO algorithm [21] proposes an alternative to DPO that combines an odds ratio-based penalty with a conventional negative log-likelihood SFT (i.e., supervised fine-tuning) loss. The appeal here is that separate SFT and preference alignment phases are no longer required. Another deviation from DPO is proposed in [19], whereby the reference policy itself is no longer fixed, but iteratively updated during training.
Analysis of DPO:
Topics addressed by recent work include analysis of DPO learning dynamics [22], the impact of out-of-preference data on estimation errors [25], and the disproportionate rates with which the DPO loss gradients favor reducing the probability of dispreferred responses relative to increasing the probability of desired responses [16]. Broader consideration of preference optimization spanning various DPO-based and RLHF-based approaches is presented in [39]
Appendix C DPO Loss Induces Noise Adaptive Regularization
Using several straightforward algebraic manipulations, the DPO loss from (7) can be modified as
(19) | |||||
excluding constants independent of . This expression represents an expectation over a regularization factor in the form , where corresponding to is fixed, and corresponding to is the variable of interest to be optimized. We will now examine several notable properties of that serve to elucidate underappreciated DPO regularization characteristics. For this purpose, we first introduce the following definition from [29]:
Definition 3
Let be a strictly increasing differentiable function on the interval . Then the differentiable function is concave relative to on iff
(20) |
where and denote the respective derivatives.
Intuitively, this definition indicates that if is concave relative to , it has greater curvature at any evaluation point once normalizing (via an affine transformation of or ) such that and . Equipped with this definition, we then point out the following observations linking DPO with prior work on robust estimation in the presence of noise:
-
•
is a concave non-decreasing function of , which represents a well-known characteristic of sparsity-favoring penalty factors commonly used in robust estimation [12, 13, 15, 35].777Most prior work involves parameters that can be negative, which can be accommodated by simply replacing with . Such penalties introduce a steep gradient around zero, but then flatten away from zero to avoid incurring significant additional loss (as would occur, for example, with a common quadratic loss).
- •
-
•
Prior work [10, 42] has investigated general optimization problems of the form
(21) sometimes generalized to over a concave, non-decreasing function of , where is some constraint set.888In some applications the constraint set may be replaced by an additional regularization factor, and there is often an equivalency between the two. Moreover, reflects a noise parameter or an analogous measure of uncertainty, with relative concavity dictated by as above. In these contexts, it has been argued that adjusting the curvature of the regularization factor based on noise levels can provide additional robustness to bad local minima and high noise regimes [10, 14, 43]. The basic intuition here is that when noise is high, a more convex shape is preferable, while when the noise is low, a more concave alternative may be appropriate.
-
•
Regarding DPO, it is natural to treat as an analogous noise factor, given that whenever this ratio is large, it implies that our reference policy is poor. Hence, once we introduce a constraint on (as will always occur in practice; see Section 3.3), solving
(22) can be viewed as a special case of (21), involving a robust regularization factor with noise-adaptive curvature.
![Refer to caption](x7.png)
Appendix D DPO from a Naive Gaussian Estimation Perspective
Any preference probability given by the BT model in (2) can be equivalently re-expressed as
(23) |
where is a conditional probability of given (i.e., the BT-optimal policy introduced in Section 3) and is a monotonically increasing function. While we may optionally choose to exactly reproduce the BT model, it is of course reasonable to consider other monotonically increasing choices to explore the additional generality of (23) (and indeed we will exploit one such alternative choice below).
Given a trainable policy we can always minimize the negative log-likelihood averaged over preference samples to approximate ; however, this procedure would be completely independent of any regularization effects of a reference policy . We now examine how to introduce the reference policy by relying only on a simple Gaussian model with trainable variances, rather than any association with RLHF or implicit reward modeling. The end result is an independent re-derivation of DPO using basic Gaussian assumptions.
For convenience, we first define the functions and as
(24) |
Now suppose we assume the naive joint distribution given by
(25) |
where denotes a 2D, zero-mean Gaussian with covariance , and is a variance parameter that depends on the tuple . Since each is unknown, we can group them together with and estimate all unknowns jointly. In the context of labeled human preference data drawn from , this involves minimizing
(26) |
where is a identity matrix and is any constraint set on as introduced in Section 3.3. The intuition here is that, although is unknown, sharing this parameter across both and and estimating jointly will induce a reference policy-dependent regularization effect.
And indeed, this simple Gaussian model exactly reproduces DPO. More concretely, the stated equivalence follows from the fact that, for an arbitrary vector we have that
(27) |
And therefore, we have
(28) |
excluding irrelevant constants. Returning to (26), if we first optimize over for each tuple, we obtain the loss factor
(29) |
From here, by choosing we can modify (29) as
(30) | |||||
ignoring the irrelevant constant which is independent of . Hence we have recovered the DPO loss for each tuple and once the requisite expectation is reintroduced, we exactly recover the full DPO loss from (7).
Appendix E Technical Proofs
E.1 Proof of Theorem 1
Definition 4
We define labeled human preference data as some , as introduced via (1), satisfying the following additional properties:
-
1.
The prompts drawn from are split between two disjoint support partitions and , i.e., with probability one, with .
-
2.
For each prompt within , the preference distribution filling out maintains support over a single (prompt-dependent) response pair .
-
3.
Pair-wise preferences are dictated by a ground-truth BT model satisfying for all .
Although the second specification above can naturally be relaxed to address more general scenarios, doing so unnecessarily complicates the presentation without providing sufficiently compelling additional insight. Additionally, for convenience below we adopt to indicate an arbitrary distance measure.
Theorem 1
(Restated formal version) Assume preference data that satisfies Definition 4. Furthermore, assume a reference policy such that for and for , where is a BT-optimal policy. It follows that for any selection of , if
(31) |
then
(32) |
where .
The proof proceeds as follows. With some abuse/imprecision of notation, we first define
(33) |
Next, per the assumptions of the theorem statement and Definition 4, we have that the QPO loss decouples as
(34) | |||
Now consider a single prompt drawn from . In order to reduce , it must be the case that , which then implies that . To achieve this, must be chosen such that
(35) |
However, to simultaneously maintain for some prompt drawn from , it must also be true, for the same fixed tuple, that
(36) |
But this is a contradiction, as the respective arguments that minimize (35) and (36) will be identical. Hence if (35) is true then for .
E.2 Proof of Proposition 1
DPO lower limit:
Given our assumption that , it follows that an optimal finite reward exists. Moreover, given that and are drawn from finite sample spaces, there will exist finite maximum and minimum optimal rewards, i.e., for some . Furthermore,
(37) |
Additionally, given that the data are generated by (1), we also know that the same optimal reward satisfies
(38) |
which is independent of . However, without constraints on , there also exists a bijection between policy and reward such that
(39) |
Hence the DPO reparameterization produces the policy given by (5) with . From this point we then observe that
(40) |
noting that for any we have as . Hence we have fulfilled the requirements of the lower limit.
DPO upper limit:
The upper limit follows trivially from the fact that for any bounded reward
(41) |
E.3 Proof of Proposition 2
Establishing the upper and lower limiting values for IPO follows a similar pattern to the proof of Proposition 2. However, because the IPO reward is bounded between zero and one by definition, we ultimately do not require any constraint on as we did for DPO.
E.4 Proof of Theorem 2
We first define
(42) |
Now suppose that for a given tuple we observe
(43) |
for some optimal and fixed , where is a finite value dependent on through the definition of . Therefore, we have that
(44) |
Obviously this ratio will depend on for any fixed . To satisfy the SIC though, in the limit the optimized policy must be independent of and converge to . However, the only way for to be independent of is if . But if so, only the WIC is achievable, not the SIC.
E.5 Proof of Theorem 3
Our strategy here is to construct a simplified situation whereby we can pinpoint emergent differences between RLHF and DPO losses in the presence of policy constraints. To this end, we assume the following:
-
•
For all , there exists two unique responses and with equal probability of 1/2 under ;
-
•
Preference data are sampled according to (1);
-
•
The loss trade-off parameter satisfies ; and
-
•
for all and .
RLHF loss processing:
When evaluated with optimal reward model , we have that
(45) | |||||
where
(46) |
This stems directly from the analysis in [31, 32]. However, because we are assuming and is constant for any given , it follows that
(47) |
where the denominator is independent of . Since the BT-optimal solution satisfies
(48) |
we may conclude that , and therefore
(49) |
under the stated conditions.
DPO loss processing:
When and is constant, we have that
(50) | |||||
Next, given the additional data generation assumptions, it follows that , and so the DPO loss can be further modified as
(51) | |||||
where is an irrelevant constant. Note that in progressing from the first to second equality, we can ignore cases where where sampled responses satisfy , since these contribute only another irrelevant constant to the loss. Along with our stated response data assumptions, this allows us to remove expectation over without loss of generality.
Final step:
From (49) and (51) we observe that the only difference between the RLHF and DPO losses under the given conditions is whether a forward or backward KL is used. And of course without any constraints, the minimizing solutions are equivalent as expected, consistent with the analysis from [33], i.e.,
(52) |
Critically though, this KL equivalence transparently need not still hold once constraints are introduced, as the forward KL will favor mode covering while the backward KL will push mode following [7].
E.6 Proof of Propositions 3 and 4
These results both follow directly from the original design of . Regarding Proposition 3, given that for all , then for the unsupervised term we have
(53) |
And for the supervised term we have
(54) |
Hence overall, for any , will be optimal for any , as this selection independently optimizes the constituent terms. Moreover, this optimality is independent of optimization over , which retains the flexibility to achieve solutions with . From this Proposition 3 immediately follows.
Additionally, Proposition 4 follows from the same basic line of reasoning. For completeness, we note that when , only the supervised term will be minimized (which recovers the BT-optimal policy as above), while when , the unsupervised term will dominate the optimization (which transparently produces ).
Appendix F Other Derivations
F.1 Derivation of (12)
Note that
(55) | |||||
where and . The policy so-defined is necessarily BT-optimal by construction. From here then we have
(58) | |||||
which is the definition of .
F.2 Additional -DPO Analysis
-PDO represents a novel generalization of DPO, but there remain certain aspects worth considering.
Minima that ignore the reference policy:
Consider general -DPO losses as described in Section 2.4, which as special cases of QPO are expressible in the form
(59) | |||
In addition to the requirements on to form an -divergence, to produce a valid -DPO loss per Theorem 1 from [41] it must be that is invertible with . Therefore the domain of will be and as because of convexity. But if this is the case, upon inspection of (59) we observe that when , then for any fixed the input argument to the logistic function will converge to infinity, pushing the output to one and subsequently minimizing the corresponding negative-log factor. And so the global optimum can be achieved independent of the value of .
F.3 Derivation of (16)
(60) | |||||
where is a constant independent of . Additionally, the third-to-last equality stems from the definition of how tuples are sampled. In particular, for a given pair , by definition a proportion of the time , while a proportion of the time . Hence
(61) |
when the latter is averaged over the preference distribution.
Appendix G Limitations
As more of an analysis-driven contribution, our experiments on real-world data are limited to Figure 4. Moreover, there are promising possibilities raised by pairing our contribution with prior work in new ways that we have not yet been explored. One example is the potential use of REINFORCE in conjunction with modifications to the proposed loss.
Appendix H Broader Impacts
Aligning the output of LLMs with human preferences has obvious, well-documented benefits. However, there nonetheless remains the risk that tools designed to improve LLM responses could be repurposed for nefarious aims. For example, preference data labels could potentially be modified to train models, using preference losses such as ours, that intentionally produce toxic content.