Preference optimization with multi-sample comparisons

Chaoqi Wang, Zhuokai Zhao, Chen Zhu, Karthik Abinav Sankararaman, Michal Valko, Xuefei Cao, Zhaorun Chen, Madian Khabsa, Yuxin Chen, Hao Ma, Sinong Wang

2025 · in (ICLR 2025)

Abstract

Traditional post-training approaches like RLHF rely on single-sample comparisons, which may not capture group-level characteristics like diversity and bias. We introduce Multi-sample Direct Preference Optimization (mDPO) and Multi-sample Identity Preference Optimization (mIPO), extending post-training to incorporate multi-sample comparisons focusing on group-wise attributes. Multi-sample comparisons are more effective than single-sample comparisons and offer a more robust optimization framework, particularly with label noise.

PDF · International Conference on Learning Representations · arXiv preprint · bibtex · DOI