8 Model Fine-Tuning: Key Hyperparameters

1. Learning Rate

The learning rate determines how much the model’s parameters are adjusted with each update. It is usually between (0, 1).

It has no impact on VRAM usage.

Impact on Learning Speed

It tells the model how quickly to “learn” during training, balancing the speed of progress with stability.

Recommended Values

For common LoRA fine-tuning, the recommended values are 4e-5 and 5e-5. (These are generally effective values derived from previous experiments and experience.)

For full-parameter fine-tuning, 5e-6 is recommended.

What happens if the learning rate is too high?

Each adjustment will be large, making it easy to override what the model previously learned and forcing it to relearn from scratch. It is fast but unstable.

What happens if the learning rate is too low?

Learning will be very slow with each update, but the model is less likely to go off course. It requires a long time to learn gradually; it is slow but stable.

Practical Training Considerations:

Do not worry about slow training. Stability is more important. Do not use a high learning rate for small datasets.

2. Training Epochs

Core Concept

A training epoch means that the model has completed one full pass through the entire training dataset.

Key Points

What happens if there are too few epochs?

Underfitting occurs because the model has not learned enough. Put simply, it has not learned anything properly.

What happens if there are too many epochs?

Overfitting occurs. The model only memorizes the contents of the training dataset and generalizes poorly, like learning by rote.

Recommended setting: Three epochs are generally sufficient.

Adjust this later based on the LOSS value. It is generally best to keep LOSS between 0.5 and 1.5.

If loss approaches 0, the model has essentially overfit.

Practical Training Considerations:

The smaller the dataset, the more epochs are required.

3. Batch Size

Batch size refers to the number of samples used each time the model parameters are updated during training.

It is the number of samples learned in a single update.

Two Key Parameters

Effective batch size = per-device batch size × gradient accumulation steps

Impact on VRAM

The larger the batch size, the more VRAM it consumes.

Gradient accumulation can achieve the effect of a large batch size with limited VRAM.

Comparison of Batch Size Advantages and Disadvantages

What happens with a large batch size?

Training is more stable, convergence toward the global optimum is easier, and computational efficiency is higher.

What happens with a small batch size?

It saves VRAM and provides stronger generalization.

Practical Recommendations

For small models or small datasets, start with batch_size=1 or 2.

Recommendation: Pair a large batch_size with a high learning rate.

4. LoRA Rank

What is LoRA rank?

This is a mathematical concept, so it will not be explained for now.

In simple terms, the model summarizes different approaches to solving problems.

What happens if the LoRA rank is high?

The model will have multiple ways to solve problems and will perform better when encountering complex questions.

It consumes more VRAM.

What happens if the LoRA rank is low?

Learning is stable, but the model’s approach is fixed and its generalization is poor. In other words, it may be unable to solve complex questions.

How Do You Choose an Appropriate LoRA Rank?

1. Recommended Values Based on Experience

Dataset size + task complexity

• Small datasets (<1000 samples): Use a lower rank of 4–8 to avoid overfitting.

• Medium datasets (1000–10000 samples): Use a medium rank of 8–16.

• For simple tasks, you can test values based directly on the dataset size.

• For complex tasks, increase the rank somewhat. For a small dataset (<1000 samples) combined with a complex task, a rank of 10 is recommended.

Actual VRAM usage:

For a 7B model, increasing the rank from 8 to 64 increases VRAM usage by 2 GB.

5. Validation Set Size

Even without a validation set, the effectiveness of model training is not affected.

Adding one simply makes it easier to observe and evaluate the validation loss curve and understand the training results.

Recommended proportions based on the number of samples:

• Small datasets (<1000 samples): Set it to 0.1–0.2.
• Large datasets (<10000 samples): Set it to 0.1.

For simple tasks, the proportion can be smaller. Otherwise, increase it somewhat.