Beyond Pairs: Your Language Model is Secretly Optimizing a Preference Graph