Training a Small Model from Scratch: 2. Supervised Fine-Tuning (SFT)

Introduction

Training a small model from scratch is an important part of my personal study of large language models (LLMs).

By carrying out a minimal reproduction of a small-scale model, I can not only gain an in-depth understanding of the basic model training workflow, but also systematically learn its core principles and actual operating mechanisms. This hands-on approach allows me to experience each stage of model training firsthand while mastering the relevant technical details and implementation methods.

In my current work, there are indeed certain tasks that require small models like this. I am also using the learning process to understand how they work, striving to implement minimal models for certain subtasks while consuming as few resources as possible.

ALL in AI

The code has been uploaded to https://github.com/godzeo/miniGPT

SFT: Supervised Fine-Tuning

Why is this step necessary?

After pretraining, an LLM has acquired a large amount of knowledge, but at this point it is merely a cold, mechanical next-word prediction machine and is not yet able to converse with people.

We want it to stop blindly continuing text and instead understand that it is participating in a complete conversation.

This is why we need instruction fine-tuning: it teaches a student who already has foundational knowledge how to solve problems, rather than endlessly memorizing passages, idioms, and conclusions. The student needs to learn how to answer questions based on the prompt and understand the question-and-answer format.

Training command:

torchrun --nproc_per_node=4 scripts/train_sft.py \
    --model_path outputs/pretrain/final \
    --tokenizer_path mistralai_tokenizer \
    --output_dir outputs/sft \
    --data_file /root/autodl-tmp/data/sft_mini_512.jsonl \
    --batch_size 16 \
    --gradient_accumulation_steps 8 \
    --num_epochs 1

Dataset

https://www.modelscope.cn/datasets/deepctrl/deepctrl-sft-data

The dataset was mainly created by cleaning the Jiangshu Large Model SFT Dataset, retaining only shorter content. After extensive reduction, the resulting streamlined dataset contained 1.21 million samples for training.

{"conversations": 
[{"role": "user", "content": "写一篇关于人工智能发展历史的文章。"},
 {"role": "assistant", "content": "好的，我将写一篇关于人工智能发展历史的文章。人工智能源于20世纪50年代的探索，经历了多次热潮和降温期。早期主要以符号推理为主，而后引入了机器学习和深度学习等技术，目前已广泛应用于图像识别、自然语言处理、游戏、机器人等领域。"}, 
 {"role": "user", "content": "提取一下人工智能在游戏领域的应用。"}, 
 {"role": "assistant", "content": "在游戏领域，人工智能主要应用于游戏智能代理、游戏设计辅助等。智能代理可以使玩家在游戏中享受更好的游戏体验，同时也可以提高游戏难度，增加游戏的挑战性。游戏设计辅助则主要应用于游戏开发中，帮助游戏开发者设计游戏难度、关卡等。"}, 
 {"role": "user", "content": "我想要这篇文章的摘要。"}, 
 {"role": "assistant", "content": "这篇文章主要介绍了人工智能的发展历史以及其在游戏领域的应用。人工智能的发展历程经历了多次的起落，随着技术的发展不断完善和创新，已经广泛应用于图像识别、自然语言处理、游戏、机器人等众多领域。在游戏领域，人工智能主要应用于游戏智能代理、游戏设计辅助等。智能代理可以使玩家在游戏中享受更好的游戏体验，同时也可以提高游戏难度，增加游戏的挑战性。游戏设计辅助则主要应用于游戏开发中，帮助游戏开发者设计游戏难度、关卡等。"}]}

However, the results were mediocre. I later considered trying the black-box distilled data from https://www.modelscope.cn/datasets/AI-ModelScope/R1-Distill-SFT.

Test results:

python test_model.py –model_path ../outputs/sft/final –tokenizer_path ../tokenizer

Testing the conversational performance

The results are actually not good. I only ran one pretraining epoch and 11 SFT epochs, both of which are far too few. The current model is like a child.

This article is relatively dry and feels as though it is mostly about searching for datasets. The next article will cover the more important fine-tuning stage.