New Desiderata for Direct Preference Optimization

Consider now the reward-dependent RLHF loss $\ell_{\tiny\mbox{RLHF}}$ from (4) defined w.r.t. and arbitrary reward function $r(y,x)$. DPO [33] is based on the observation that, provided $\pi_{\theta}$ is sufficiently flexible such that we may treat it as an arbitrary function for optimization purposes,³³3This is a key assumption with non-trivial consequences; Section 3.3 will explore this issue in further detail. the minimum of $\ell_{\tiny\mbox{RLHF}}\left(\pi_{\theta},\pi_{\tiny\mbox{ref}},r,\lambda\right)$ w.r.t. $\pi_{\theta}$ can be directly computed as

$$\pi_{r}(y|x)~{}:=~{}\arg\min_{\pi_{\theta}}\ell_{\tiny\mbox{RLHF}}\left(\pi_{\theta},\pi_{\tiny\mbox{ref}},r,\lambda\right)=\frac{1}{Z(x)}\pi_{\tiny\mbox{ref}}(y|x)\exp\left[\frac{1}{\lambda}r(y,x)\right],$$

(5)

where $Z(x):=\sum_{y}\pi_{\tiny\mbox{ref}}(y|x)\exp\left[\frac{1}{\lambda}r(y,x)\right]$ is the partition function ensuring that $\pi_{r}(y|x)$ forms a proper distribution [31, 32]. From here, assuming $\pi_{\tiny\mbox{ref}}(y|x)>0$, we can rearrange (5) to equivalently establish that

$$r(y,x)=\lambda\log\frac{\pi_{r}(y|x)}{\pi_{\tiny\mbox{ref}}(y|x)}+\lambda\log Z(x).$$

(6)

Because thus far $r$ has remained unspecified, it naturally follows that these policy/reward relationships hold even for the ground-truth reward $r^{*}$ and the associated optimal policy $\pi^{**}(y|x):=\arg\min_{\pi_{\theta}}\ell_{\tiny\mbox{RLHF}}\left(\pi_{\theta},\pi_{\tiny\mbox{ref}},r^{*},\lambda\right)$. Hence instead of approximating $r^{*}(y,x)$ with $r_{\phi}(y,x)$ as in (2), we may equivalently approximate $\pi^{**}(y|x)$ with some $\pi_{\theta}(y|x)$ leading to the DPO loss    $\ell_{\tiny\mbox{DPO}}(\pi_{\theta},\pi_{\tiny\mbox{ref}},\lambda)~{}:=$

$$\ell_{\tiny\mbox{BT}}\left(\lambda\log\frac{\pi_{\theta}(y|x)}{\pi_{\tiny\mbox{ref}}(y|x)}\right)=\mathbb{E}_{\{y_{w},y_{l},x\}\sim{\mathcal{D}}_{\tiny\mbox{tr}}}\left[-\log\sigma\left(\lambda\log\frac{\pi_{\theta}(y_{w}|x)}{\pi_{\tiny\mbox{ref}}(y_{w}|x)}-\lambda\log\frac{\pi_{\theta}(y_{l}|x)}{\pi_{\tiny\mbox{ref}}(y_{l}|x)}\right)\right],$$

(7)

noting that the partition function $Z(x)$ conveniently cancels out and can be excluded from further consideration. It is now possible to directly optimize (7) over $\pi_{\theta}$ using SGD without the need for any challenging RLHF procedure. The basic intuition here is that the parameterized policy $\pi_{\theta}$ induces an implicit reward $\lambda\log\left[\pi_{\theta}(y|x)\pi_{\tiny\mbox{ref}}^{-1}(y|x)\right]$ that is being optimized via the original BT preference model. Moreover this equivalence is exact assuming data distributed as in (1).

Similar to DPO, the identity preference optimization (IPO) formulation [3] avoids both a 2-step learning process and cumbersome, potentially unstable RL training. To accomplish this, IPO is predicated on minimizing the original RLHF loss from (4) but with an alternative reward function. Specifically, the motivating IPO objective is to minimize $\ell_{\tiny\mbox{RLHF}}\left(\pi_{\theta},\pi_{\tiny\mbox{ref}},r_{\tiny\mbox{IPO}},\lambda\right)$, where

$$r_{\tiny\mbox{IPO}}(y,x):=\mathbb{E}_{y^{\prime}\sim\pi_{\tiny\mbox{ref}}(y|x)}\big{[}p^{*}(y\succ y^{\prime}|x,y,y^{\prime})\big{]},$$

(8)

over $\pi_{\theta}$.⁴⁴4Note that in principle the distribution used to draw samples $y^{\prime}$ in defining $r_{\tiny\mbox{IPO}}$ need not be set to $\pi_{\tiny\mbox{ref}}$; however, in practice $\pi_{\tiny\mbox{ref}}$ is a typical choice, which we adopt throughout for simplicity. Because of the special structure of this particular reward function, it turns out that it is possible to minimize $\ell_{\tiny\mbox{RLHF}}\left(\pi_{\theta},\pi_{\tiny\mbox{ref}},r_{\tiny\mbox{IPO}},\lambda\right)$ over $\pi_{\theta}$ without RL. In brief, this is accomplished by first noting that for any pair of responses $y_{1}\neq y_{2}$ the existence of an optimal IPO policy, denoted $\pi_{\tiny\mbox{IPO}}$, evaluated at these responses can be computed as a function of the reward $r_{\tiny\mbox{IPO}}$ using (5). Combining $y_{1}$ and $y_{2}$ dependent terms, after a few algebraic manipulations this then leads to the equivalence relation

$$\log\left[\frac{\pi_{\tiny\mbox{IPO}}(y_{1}|x)\pi_{\tiny\mbox{ref}}(y_{2}|x)}{\pi_{\tiny\mbox{IPO}}(y_{2}|x)\pi_{\tiny\mbox{ref}}(y_{1}|x)}\right]=\frac{1}{\lambda}\big{[}r_{\tiny\mbox{IPO}}(y_{1},x)-r_{\tiny\mbox{IPO}}(y_{2},x)\big{]}.$$

(9)

However, unlike DPO where an analogous expression is inverted to create an implicit reward for integration within the BT model, IPO instead attempts to approximate this equivalence relation by replacing the unknown $\pi_{\tiny\mbox{IPO}}(y|x)$ with some $\pi_{\theta}(y|x)$. Although technically $r_{\tiny\mbox{IPO}}$ is also unknown, given samples $\{y_{w},y_{l},x\}\sim{\mathcal{D}}_{\tiny\mbox{tr}}$, it is nicely shown in [3] that    $\ell_{\tiny\mbox{IPO}}(\pi_{\theta},\pi_{\tiny\mbox{ref}},\lambda)~{}~{}:=$

	$\displaystyle\mathbb{E}_{\{y_{1},y_{2}\}\sim\pi_{\tiny\mbox{ref}}(y|x),x\sim{\mathcal{D}}}\left[\left(\log\left[\frac{\pi_{\theta}(y_{1}|x)\pi_{\tiny\mbox{ref}}(y_{2}|x)}{\pi_{\theta}(y_{2}|x)\pi_{\tiny\mbox{ref}}(y_{1}|x)}\right]-\frac{1}{\lambda}\big{[}r_{\tiny\mbox{IPO}}(y_{1},x)-r_{\tiny\mbox{IPO}}(y_{2},x)\big{]}\right)^{2}\right]$
	$\displaystyle=~{}~{}~{}\mathbb{E}_{\{y_{w},y_{l},x\}\sim{\mathcal{D}}_{\tiny\mbox{tr}}}\left[\left(\log\left[\frac{\pi_{\theta}(y_{w}|x)\pi_{\tiny\mbox{ref}}(y_{l}|x)}{\pi_{\theta}(y_{l}|x)\pi_{\tiny\mbox{ref}}(y_{w}|x)}\right]-\frac{1}{2\lambda}\right)^{2}\right]$		(10)

provided ${\mathcal{D}}_{tr}$ follows from (1). Note that this closed-form consistency is a direct consequence of how $r_{\tiny\mbox{IPO}}$ is defined in (8) and will not generally hold for other choices of the reward function. Regardless, it is straightforward to minimize $\ell_{\tiny\mbox{IPO}}(\pi_{\theta},\pi_{\tiny\mbox{ref}},\lambda)$ in its present form via SGD as with DPO.

From the expressions above, it is clear that both DPO and IPO reduce to functions of $\log\left[\frac{\pi_{\theta}(y_{w}|x)\pi_{\tiny\mbox{ref}}(y_{l}|x)}{\pi_{\theta}(y_{l}|x)\pi_{\tiny\mbox{ref}}(y_{w}|x)}\right]$ and a tunable hyperparameter $\lambda$. As such, it is natural to consider extensions to broader choices in the form

$$\ell_{\tiny\mbox{QPO}}(\pi_{\theta},\pi_{\tiny\mbox{ref}},\psi,\mu,\lambda):=\mathbb{E}_{\{y_{w},y_{l},x\}\sim{\mathcal{D}}_{\tiny\mbox{tr}}}~{}\psi\left(\mu\left[\frac{\pi_{\theta}(y_{w}|x)}{\pi_{\tiny\mbox{ref}}(y_{w}|x)}\right]-\mu\left[\frac{\pi_{\theta}(y_{l}|x)}{\pi_{\tiny\mbox{ref}}(y_{l}|x)}\right],\lambda\right),$$

(11)

where $\mu:\mathbb{R}^{+}\rightarrow\mathbb{R}$ is a monotonically increasing function (which generalizes the logarithm), and the function $\psi:\mathbb{R}\times\mathbb{R}^{+}\rightarrow\mathbb{R}$ influences the overall loss shape. We stipulate that $\psi$ is a differentiable quasi-convex function [20]; hence the chosen loss notation $\ell_{\tiny\mbox{QPO}}$ for quasi-convex preference optimization. By definition of quasi-convexity, $\psi$ monotonically increases to the right or left away from the minimum.

These specifications cover DPO and IPO as representative special cases, and include essentially all reasonable choices for a loss within this family, e.g., it is nonsensical to include multi-modal losses. The generalized preference optimization (GPO) [40] and f-DPO [41] frameworks are also special cases of QPO as defined herein. With GPO, $\mu$ is a logarithm and $\psi$ is chosen as an arbitrary convex function (such as used by SLiC [45]). Meanwhile f-DPO involves $\psi(\cdot,\lambda)=-\log\sigma[\lambda(\cdot)]$ analogous to DPO but with $\mu=f^{\prime}$, where $f^{\prime}$ denotes the derivative of an $f$-divergence [36]; given that $f$ must be convex, its derivative will necessarily be monotonically increasing. In this way, the RLHF objective from (4) is still optimized via $f$-DPO, but with an $f$-divergence replacing the KL term.

While overall quite general, we will nonetheless later demonstrate that any loss in the form of (11) will unavoidably be saddled with certain limitations. See also Appendix B for additional context w.r.t. very recent and/or concurrent DPO enhancements that lie outside the scope of our present work.

Consider the following plausible scenario, variations of which are likely to occur (at least in varying degrees) with real-world data. Suppose the support of prompts generated by ${\mathcal{D}}_{x}$ partitions as $d_{x}^{good}\cup d_{x}^{bad}$, with $d_{x}^{good}\cap d_{x}^{bad}=\emptyset$. Furthermore, assume we have access to a reference policy $\pi_{\tiny\mbox{ref}}$ such that $\pi_{\tiny\mbox{ref}}=\pi^{*}$ for $x\in d_{x}^{good}$ and $\mbox{dist}[\pi_{\tiny\mbox{ref}},~{}\pi^{*}]\gg 0$ for $x\in d_{x}^{bad}$, where $\mbox{dist}[\cdot,\cdot]$ is an arbitrary distance measure. In other words, when evaluated w.r.t. a policy $\pi^{*}$ that proportionally reflects human preferences, $\pi_{\tiny\mbox{ref}}$ performs ideally on a subset of prompts but not on others.

This dichotomy provides a useful lens for examining certain loss function properties. In particular, we would like any policy that minimizes a candidate loss to preserve $\pi_{\tiny\mbox{ref}}$ for prompts $x\in d_{x}^{good}$, while pushing away from $\pi_{\tiny\mbox{ref}}$ towards $\pi^{*}$ for prompts $x\in d_{x}^{bad}$. However, because of uniform regularization effects intrinsic to the QPO loss, it is not actually possible to achieve even this modest objective.

The proof and formal version are provided in Appendix E.1, while Figure 1(left) below provides an illustration. The somewhat unexpected implication here is that if we minimize any possible QPO loss in the form of (11) and improve the policy quality in areas where $\pi_{\tiny\mbox{ref}}$ performs poorly w.r.t. $\pi^{*}$, then it must also be the case that performance becomes worse in areas where $\pi_{\tiny\mbox{ref}}$ was originally optimal. This phenomena represents an unavoidable trade-off when we restrict ourselves to using a QPO loss, of which DPO and IPO (as well as GPO and $f$-DPO) are special cases inheriting the same limitation. The core issue here is that QPO losses unselectively apply the same regularization, starting from the same initialization point, to both good and bad cases relative to $\pi^{*}$.

As the underlying goal shared by all approaches is to balance proximity to a reference policy $\pi_{\tiny\mbox{ref}}$ with respect for the human preference model $p^{*}$, a non-negative trade-off parameter $\lambda\in[a,b]$ that allows for interpolating between these competing objectives is inevitable, where $a\in\mathbb{R}$ and $b\in\mathbb{R}$ are lower and upper bounds respectively.⁵⁵5Depending on the method, if $a=0$ or $b=\infty$ we may replace the $\lambda$ range with an open set. In this section we examine more closely the nature of loss function minimizers as $\lambda$ is varied, zooming in on their behavior in the limit as $\lambda\rightarrow a$ and $\lambda\rightarrow b$. To this end, we first introduce the following definitions :

In summary, the only difference between these interpolation criteria is their limiting behavior w.r.t. the lower bounding $\lambda$; for the SIC we approach the BT-optimal policy, while for the WIC we approach a degenerate policy with all probability mass restricted to the mode of the BT-optimal policy. We remark that both the SIC and WIC cannot be simultaneously satisfied unless $\pi^{*}$ itself is a degenerate delta function. We now explore how these distinctions are reflected in the behavior of DPO and IPO loss minimizers, with Figure 1(middle) illustrating the basic concepts.

In terms of practical applicability of this result, there exists one important caveat: the empirical distribution of a finite set of labeled preference data need not actually satisfy the conditions of Proposition 1. For example, suppose for each prompt $x\in{\mathcal{D}}_{x}$ we collect only two responses $\{y_{1},y_{2}\}$ along with a single preference label $z$, which together produce the tuple $\{y_{w},y_{l},x\}$. In this scenario, which reflects certain publicly-available human preference datasets [4, 18], the empirical distribution of preferences will be $p^{*}(y_{w}\succ y_{l}|x)=1\notin(0,1)$ for all $x\in{\mathcal{D}}_{x}$. Notably, Proposition 1 will not hold, and in particular, it can be easily shown that minimizers of any valid $f$-DPO loss will be completely independent of $\pi_{\tiny\mbox{ref}}$ for all $\lambda\in(0,\infty)$; in other words, no interpolation occurs at all; see Appendix F.2 for the derivation. A similar observation specific to DPO (but not $f$-DPO) can be found in [1]. The fact that DPO-based solutions may still reflect $\pi_{\tiny\mbox{ref}}$ in practice, and more-so as $\lambda$ increases, relates to implicit constraints and subtle regularization effects as discussed further in Section 3.3 and Appendix C.

Comparing Proposition 2 with Proposition 1, we observe that IPO maintains its ability to interpolate under broader conditions than DPO, particularly in the empirical sampling regime involving binary probability values. That being said, neither DPO nor IPO satisfy the SIC, which motivates consideration of alternative losses that do, at least if our priority is to actually achieve the SIC (which of course may depend on the application scenario). For this purpose, it turns out that selections beyond the family of QPO objectives (which includes DPO, $f$-DPO, and IPO) are necessary per the following:

Section 4 will consider objectives outside of the QPO family which circumvent this limitation.

Originally in [33], and later supported by follow-up analysis [3], it has been shown that minimizing the DPO loss $\ell_{\tiny\mbox{DPO}}(\pi_{\theta},\pi_{\tiny\mbox{ref}},\lambda)$ is effectively the same as minimizing the RLHF loss $\ell_{\tiny\mbox{RLHF}}\left(\pi_{\theta},\pi_{\tiny\mbox{ref}},r^{*},\lambda\right)$ with optimal reward model $r^{*}$. But there is a pivotal assumption underlying this association which previous analysis has not rigorously accounted for. Specifically, the key equalities that facilitate the DPO and IPO reparameterizations, namely (6) and (9) (and the analogous for $f$-DPO), are all predicated on the solution of an uncononstrained optimization problem over an arbitrary policy $\pi_{\theta}$.

However, when actually training models in real-world settings, constraints will always exist, whether implicitly or explicitly. Such constraints stem from any number of factors including the model architecture/capacity limitations, early stopping, weight decay, drop-out regularization, machine precision, and so on. Hence in reality we are never exactly minimizing some preference loss $\ell\left(\pi_{\theta},\pi_{\tiny\mbox{ref}},\lambda\right)$ over any possible $\pi_{\theta}$ (as assumed by DPO, IPO, and $f$-DPO derivations). Instead, we must consider properties of the constrained problem $\min_{\pi_{\theta}\in{\mathcal{S}}_{\pi}}\ell\left(\pi_{\theta},\pi_{\tiny\mbox{ref}},\lambda\right)$, where ${\mathcal{S}}_{\pi}$ is a constraint set. For example, if we restrict training to a single epoch with a fixed learning rate, then ${\mathcal{S}}_{\pi}$ can be viewed as the set of all points reachable within a limited number of SGD updates.