Training a Small Model from Scratch: 4. LoRA Fine-Tuning

1. LoRA (Low-Rank Adaptation)

LoRA is an efficient parameter-efficient fine-tuning (PEFT) method. It fine-tunes pretrained models through low-rank decomposition.

Compared with full fine-tuning, LoRA only needs to update a small number of parameters. The core idea behind LoRA is to introduce low-rank decomposition into the model’s weight matrices, updating only the low-rank components while keeping the original pretrained weights unchanged.

This was actually one of the first areas I studied. It is also one of the most widely used approaches in real-world applications, because most of us will not train an entire model ourselves. In most cases, we fine-tune an existing model instead. Training a model yourself is extremely expensive, the results can be poor, and processing the dataset is also very troublesome. Although the model I trained was very small, its answers were not particularly good when neither the number of training epochs nor the dataset was optimized. Still, my main purpose was indeed to learn.

Why use fine-tuning? (Advantages):

1. Adding domain knowledge: For example, if a model lacks knowledge in medicine or cybersecurity, relevant domain knowledge can be added to the original model to improve its performance.
2. Preventing catastrophic forgetting: This is a technical term. Put simply, it means preserving the model’s original fundamental capabilities while it learns knowledge from a new domain. LoRA can address this problem very well. (This is explained below in the section on technical principles.)
3. Drastically reducing the parameter count: Compared with full fine-tuning, LoRA can reduce the number of trainable parameters by more than 97%.
4. Reducing VRAM requirements: Only a small number of parameters need to be stored and updated, greatly reducing GPU memory requirements.

2. Technical Principles of LoRA

Mathematical Principle: Low-Rank Adaptation

The core idea behind LoRA is to introduce low-rank decomposition into the model’s weight matrices, updating only the low-rank components while keeping the original pretrained weights unchanged. In my personal understanding, it is more like a mathematical trick from linear algebra.

Here is a simple explanation, although you still need to understand the most basic linear algebra. If you know nothing about it, you may still find this difficult to understand, but this is already the easiest explanation based on simple calculations.

Original Matrix (Original Model Parameters) W

Suppose we have an original 5×5 weight matrix W whose elements are integers. This W contains the original model parameters.

Suppose the Fine-Tuned Matrix Is W’

After fine-tuning, some of these parameters change slightly. We can expand this as:

W’ = W+ A⋅B

This becomes a simple calculation involving the original matrix W plus the matrix AB, so we only need to calculate AB.

A*B can then be expanded further. Viewed this way, the number of parameters is immediately reduced.

In this way, we only need to train matrices A and B instead of the entire matrix W.

In practice, when the results are good, the proportion of trainable parameters can be reduced to around 0.01%-3%.

Low-Rank Decomposition:

Basic knowledge: Any matrix can be decomposed into the product of three matrices through singular value decomposition (SVD).

We then only need to retain the largest singular values, allowing us to represent W approximately with fewer parameters.

The retained singular-value information can also be used to calculate how much information is preserved in the reconstructed matrix—for example, 99%.

The application of singular value decomposition (SVD) in LoRA is very clever. Through SVD, we can decompose a large weight-update matrix into smaller components, greatly reducing the number of parameters that need to be stored and computed. This mathematical technique not only preserves model performance but also significantly improves training efficiency.

3. Code

The tooling is now relatively mature, and various popular frameworks can be used for fine-tuning.

Fine-tuning can even be performed without writing any code.

Recommended frameworks: https://github.com/hiyouga/LLaMA-Factory

https://github.com/hiyouga/LLaMA-Factory

Some AI Studio platforms also support fine-tuning directly.

Fine-tuning also depends on the base model: we need to consider the capabilities of the base model itself.

4. My Training Results

To be honest, my base model was indeed not very good. To save money, I trained it for too few epochs, and its answers were mediocre to begin with. After fine-tuning, it gained some domain-specific capabilities, but it still was not good enough.

When I train again in the future, I will not use my own model. Instead, I will directly fine-tune one of the latest base models with a small parameter count, such as llama3.2-7b or qwen-2.5-0.5b. I think models like these are quite good.

Therefore, I will demonstrate medical fine-tuning with qwen/Qwen2.5-0.5B.

modelscope download --model Qwen/Qwen2.5-0.5B-Instruct --local_dir ./dir

I fine-tuned it directly on the CPU, which shows that the resource consumption is indeed low.

The original response: the model has no medical knowledge in this case.

The result after fine-tuning:

It clearly has some medical knowledge now, and its answer sounds more like a doctor’s.

5. A Separate Discussion: Do We Really Need Fine-Tuning?

There is another fairly important question that we need to discuss separately:

Do we really need fine-tuning? Under what circumstances is fine-tuning necessary? Should we implement it using RAG instead?

I will write another detailed article comparing model fine-tuning with RAG.