0x00 Background

Why research this? ChatGPT has attracted widespread attention both in China and abroad, and many universities, research institutions, and companies are planning to launch similar models. However, ChatGPT is not open source and is extremely difficult to reproduce. Even now, no organization or company has been able to fully replicate GPT-3’s capabilities. Recently, OpenAI released the GPT-4 model, which supports multimodal text and image input. Its capabilities are significantly more advanced than those of ChatGPT, seemingly heralding the arrival of a fourth industrial revolution led by artificial general intelligence.

Whether in China or abroad, the gap with OpenAI is widening. Everyone is striving to catch up, and competition in this technological revolution is intense. Many large companies have currently adopted closed-source research and development strategies. Very few details about ChatGPT and GPT-4 are available, unlike the detailed information provided in previous papers. OpenAI’s era of commercialization has arrived. Of course, some organizations and individuals have also conducted their own explorations on open-source platforms. This article summarizes those efforts and will continue to track and update developments on open-source platforms.

0x01

An Affordable ChatGPT Implementation

Below are several popular large models that I have recently explored.

ChatGLM

https://github.com/THUDM/ChatGLM-6B

ChatGLM is a conversational model from the open-source GLM series released by Zhipu AI, a company established to commercialize technological achievements from Tsinghua University. The model supports both Chinese and English, and its 6.2-billion-parameter version has been open-sourced. It not only inherits the advantages of previous GLM models but also optimizes the model architecture, lowering the barrier to deployment and application and enabling large-model inference on consumer-grade graphics cards. For specific technical details, refer to its GitHub page.

From a technical perspective, ChatGLM implements ChatGPT’s reinforcement learning-based human alignment strategy, making its generated output more consistent with human values. Its current capabilities include self-awareness, outline writing, copywriting, email writing assistance, information extraction, role-playing, review comparison, travel recommendations, and more. In addition, a 130-billion-parameter model is currently undergoing internal testing, making it one of the largest conversational models by parameter count on open-source platforms.

LLaMA

The LLaMA model is a large language model based on natural language processing technology. Its name is an abbreviation of “Language Model for the Martian.” Released by Meta, Facebook’s artificial intelligence research organization, it is a pretrained model consisting of more than 70 billion parameters.

https://github.com/facebookresearch/llama

The LLaMA model performs well on tasks such as natural language generation, dialogue generation, text summarization, mathematical theorem proving, and protein structure prediction. Compared with other similar language models, LLaMA offers greater versatility and applicability. It supports more than 20 languages, including English, Spanish, Arabic, Russian, and German. It also supports different alphabets and symbol systems, such as the Latin, Greek, and Cyrillic alphabets.

For natural language generation, LLaMA can automatically produce different types of text, including paragraphs, stories, poems, and dialogue. For text summarization, it can automatically generate a brief summary of a given article. For mathematical theorem proving, it can generate the corresponding proof steps and explanations. For protein structure prediction, it can predict a protein’s three-dimensional structure from its sequence information.

The release of LLaMA has had a major impact on the field of artificial intelligence, providing strong support for the development of natural language processing technology. It has also served as an important reference for the development and application of other similar large language models.

It is worth noting that the LLaMA model was involved in a leak in 2021, which resulted in the model’s data becoming public. This incident had an enormous impact on the field of artificial intelligence and accelerated the open-source development of ChatGPT-like models.

Alpaca

Because LLaMA’s current license is relatively restrictive, it can only be used for research and does not permit commercial use.

Alpaca, whose full name is Stanford Alpaca, is a natural language processing model released by Stanford University. It is a model obtained by fine-tuning LLaMA. Alpaca’s basic idea is to have OpenAI’s text-davinci-003 model generate a large number of instruction samples using the self-instruct approach, and then use those samples to fine-tune the LLaMA model.

Excellent results were achieved using a dataset of only 52K samples. https://github.com/tatsu-lab/stanford_alpaca/blob/main/alpaca_data.json

Alpaca’s training process is very interesting. During training, the researchers intensively trained the text-davinci-003 model, having it generate a series of instruction samples—including dialogue, text, and code—based on its own judgment and instructions. These instruction samples were then used to fine-tune the LLaMA model, improving its performance on specific tasks. This self-training approach allows Alpaca to be fine-tuned without human intervention, greatly reducing training costs and time.

Alpaca’s model files have not yet been open-sourced, but its training data, code for generating the training data, and hyperparameters have been made publicly available on GitHub. Because of its low cost and readily available data, the Alpaca project has received widespread attention and praise in the field of natural language processing.

stanford-alpaca

https://github.com/tatsu-lab/stanford_alpaca

alpaca-lora

Stanford University released another important project: alpaca-lora.

This project reproduced Alpaca’s results using LoRA and adopted a lower-cost approach. With just one RTX 4090 graphics card and five hours of training, it produced a model comparable to Alpaca. Moreover, the model can run on a Raspberry Pi. The project uses Hugging Face’s PEFT to achieve inexpensive and efficient fine-tuning. PEFT is a library that supports LoRA and can fine-tune various Transformer-based language models with LoRA, enabling inexpensive and effective model fine-tuning on ordinary hardware. The project’s GitHub address is: https://github.com/tloen/alpaca-lora。

Although Alpaca and alpaca-lora achieved substantial improvements, both use English as the seed task and lack support for Chinese. To solve this problem, three independent developers from institutions including Central China Normal University open-sourced a Chinese language model named Luotuo. The model builds on previous work such as alpaca-lora and can be trained and deployed on a single graphics card. The project has released two models, luotuo-lora-7b-0.1 and luotuo-lora-7b-0.3, with another model also planned. The project’s GitHub address is: https://github.com/LC1332/Chinese-alpaca-lora。

Vicuna and Chinese-Vicuna

Researchers from Stanford, in collaboration with CMU, UC Berkeley, and others, released a brand-new 13-billion-parameter model called Vicuna, commonly known as the “little alpaca” or “llama.” The model is produced by fine-tuning LLaMA on user-shared conversations collected from ShareGPT. During testing, GPT-4 was used as the judge, and the results showed that Vicuna-13B matched the capabilities of ChatGPT and Bard in more than 90% of cases. It achieved 90% of ChatGPT’s performance for only $300. Recently, UC Berkeley’s LMSys organization also released a 7-billion-parameter Vicuna model. It is compact, efficient, and powerful, can run on an M1/M2-based Mac with just two commands, and supports GPU acceleration. The project’s open-source GitHub address is: https://github.com/lm-sys/FastChat/

There is also a Chinese version of Vicuna named Chinese-Vicuna, which has also been open-sourced. Its GitHub address is: https://github.com/Facico/Chinese-Vicuna

Fine-Tuning Approaches

Problems with model fine-tuning

Here is a comparison of the results from several fine-tuning approaches:

Likewise:

Abstract

Prompt tuning can reduce storage and training-time memory consumption for downstream tasks by tuning only continuous prompts on a frozen language model. However, previous work has shown that prompt tuning does not perform well on NLU tasks with normally sized pretrained models. This article also finds that existing prompt methods cannot handle difficult sequence-labeling problems, demonstrating a lack of generality. With proper optimization, prompt tuning can be effectively applied across different model scales and NLU tasks. The proposed P-Tuning V2 is not a new method, but an optimized and improved version of Prefix-tuning. Introduction

Fine-Tuning

In fine-tuning, we copy some or all of the weights from a pretrained model into a new task, and then use data from the new task to fine-tune those weights so that they adapt to it. Fine-tuning usually requires training additional layers for the new task. These are new layers outside the pretrained model, and their primary purpose is to adjust the pretrained model’s output appropriately so that it is suitable for the new task.

Existing fine-tuning approaches can achieve good results, but they also have major problems.

Especially when working with large amounts of data, fine-tuning requires a great deal of data to achieve good results.

Many different types of data are needed to make all kinds of adjustments.

LoRA

简单的说就是：
在模型外面在潜入一个层
用小模型去影响大模型

LORA (Layer-wise Relevance Analysis) is a model interpretation method used to explain the predictions of deep neural networks. Its basic idea is to insert an interpretation layer between each layer of the model and explain the model’s prediction process by calculating the importance of each input feature to each interpretation layer.

More specifically, LORA first embeds a small interpretation model between every layer of the original model. This interpretation model is usually a simplified linear model or a shallow neural network. The importance of each feature to the model’s prediction is then evaluated by calculating the weight of each input feature in the interpretation model. In this way, the prediction process of the original model can be explained by analyzing the output of each interpretation layer.

Using LORA, we can determine how much each feature contributes to the model’s predictions and thus better understand the model’s decision-making process. This is very helpful for assessing the model’s interpretability and trustworthiness. LORA can also be used to debug and improve models. By analyzing the output of each interpretation layer, we can identify potential problems in the model and take corresponding measures to improve it.

Problems with Model Fine-Tuning

Especially when working with large amounts of data, fine-tuning requires a great deal of data to achieve good results.

A major problem with model fine-tuning is that a large amount of data is required to achieve good results. In particular, when dealing with large-scale datasets, fine-tuning a model requires various types of data to adapt it to different tasks. This makes fine-tuning difficult and time-consuming.

Another problem is overfitting during fine-tuning. When a model is fine-tuned, it may overfit, meaning that it performs well on the training data but poorly on the test data. This may occur because the amount of fine-tuning data is limited, causing the model to adapt too closely to that data and preventing it from generalizing to new data.

In addition, managing trade-offs during fine-tuning is also challenging. Fine-tuning requires appropriate adjustments for the new task while retaining the pretrained model’s knowledge. This requires finding a balance between preserving the model’s ability to generalize and its ability to adapt.

In summary, model fine-tuning faces issues involving substantial data requirements, overfitting, and trade-offs. Solving these problems requires appropriate data augmentation techniques, regularization methods, and model architecture designs, as well as careful adjustment of hyperparameters during fine-tuning.