Training a Small Model from Scratch: 3. Differences Between Traditional RLHF and DPO

I actually skipped this step. If my goal is to build a model for the security domain, I personally do not think this step is particularly necessary. Regarding alignment with human preferences: after the preceding training steps, the model already has basic conversational capabilities.

At this point, the model still does not know what constitutes a good answer or a bad one. We want it to align more closely with human preferences and reduce the likelihood of producing unsatisfactory answers.

Simply put, this teaches the model to discuss security at your company in a way that matches your own preferred style; it only knows that responding this way is better.

Primary Goal: Achieve Alignment with Human Preferences

• Make AI behavior conform to human expectations:
  • Produce more helpful answers
  • Reduce harmful, incorrect, or misleading content (for example, when we ask security-related questions, the model may be expected to refuse to answer)
  • Follow instructions and understand subtle human intent
  • Reflect human values and ethical principles

Therefore, I have omitted this process from my training steps for now.

Differences Between Traditional RLHF and DPO

Traditional RLHF (Reinforcement Learning from Human Feedback) and DPO (Direct Preference Optimization) are both methods for optimizing language models to align with human preferences, but they differ significantly in how they are implemented:

Traditional RLHF

1. Three-stage process:
   • Stage 1: Supervised Fine-Tuning (SFT) — fine-tune the base model on high-quality data
   • Stage 2: Train a Reward Model (RM) — train it using human preference feedback
   • Stage 3: Reinforcement Learning — use the PPO (Proximal Policy Optimization) algorithm to optimize the policy based on the reward model
2. High complexity:
   • Requires training a separate reward model
   • The reinforcement learning stage is computationally complex and unstable
   • Requires careful tuning of the KL-divergence penalty to prevent the model from deviating from its initial distribution
3. Computational resources: Requires substantial computational resources, particularly during the PPO stage

DPO

1. Simplified process:
   • Stage 1: Supervised Fine-Tuning (SFT) — the same as RLHF
   • Stage 2: Direct optimization — uses human preference data directly without requiring a separate reward model
2. Mathematical theory:
   • Reformulates the RLHF problem as a direct optimization problem
   • Demonstrates that a reward model can be learned implicitly by optimizing a specific loss function
3. Advantages:
   • Simplifies the training process by eliminating the need for an explicit reward model
   • Offers greater computational efficiency and faster training
   • Is easier to implement and more stable
   • Reduces the risk of deviating from the initial distribution during optimization

Comparison of Data Requirements

• RLHF: Requires human preference data to train the reward model, followed by extensive environment interaction to perform PPO
• DPO: Requires only human preference data, typically in the form of “chosen-rejected” pairs

Performance Comparison

Research shows that, in many application scenarios, DPO can achieve performance comparable to or even better than traditional RLHF while significantly reducing training complexity and computational resource requirements.

This simplification enables more researchers and small teams to implement alignment techniques while making the model optimization process more transparent and interpretable.