Fine tune LLM LoRA adapters: data, training and a first test on one GPU

To fine tune LLM LoRA adapters on a single GPU, write examples of the exact task you want, train a small adapter with Hugging Face PEFT and the TRL SFTTrainer, and compare its answers with the base model on examples it never saw. Before any of that, confirm that fine-tuning is the right fix, because many problems that look like model limits are prompt or retrieval problems.

This guide uses TRL 1.15.0 and PEFT 0.21.0, the versions the Hugging Face docs showed when it was written. The model in the examples is Qwen/Qwen3-0.6B, the one TRL uses in its own quick start.

Fine-tuning, RAG or a better prompt: which problem do you have?

Fine-tuning changes how a model behaves. It is a poor way to give a model facts, because facts change and a trained model cannot cite where it learned them. Use this decision table before you collect any data. It is a rule of thumb, not a benchmark result.

If two rows apply, combine them. A fine-tuned model can still sit behind a retrieval step.

What LoRA changes in a model

LoRA, from the 2021 paper LoRA: Low-Rank Adaptation of Large Language Models (opens in a new tab), freezes the pretrained weights and trains a pair of small matrices next to selected layers. The PEFT LoRA reference (opens in a new tab) describes the pair as update matrices that represent the weight change through low-rank decomposition. At inference, the original and adapted weights are combined.

Three details matter in practice:

• Rank. The r setting sets the size of the update matrices. A higher rank means more trainable parameters and more capacity. The PEFT default is r=8.
• Scaling. The update is scaled by lora_alpha/r, or by lora_alpha/math.sqrt(r) with use_rslora=True. The default lora_alpha is 8.
• Starting point. By default PEFT sets the B matrix to zero, so an untrained adapter changes nothing.

The trainable share is tiny. The PEFT quicktour (opens in a new tab) wraps the q_proj layers of meta-llama/Llama-3.2-1B at rank 8 and reports 524,288 trainable parameters out of 1,236,338,688. The saved adapter holds only the trained weights: the quicktour's opt-350m example has an adapter_model.safetensors file of 6.3MB.

Preparing a training set

The TRL SFT Trainer docs (opens in a new tab) list two data types the trainer accepts: language modeling data (a text field or a messages list) and prompt-completion data. For a single task, conversational prompt-completion is the cleanest choice. By default the trainer computes loss only on the completion, so the model learns to answer and does not spend capacity learning to repeat prompts.

Write one JSON object per line. The ticket-triage task below is illustrative.

{"prompt": [{"role": "user", "content": "Triage: The CSV export returns a 500 error since this morning."}], "completion": [{"role": "assistant", "content": "{\"category\": \"bug\", \"priority\": \"high\"}"}]}
{"prompt": [{"role": "user", "content": "Triage: Could you add dark mode to the dashboard?"}], "completion": [{"role": "assistant", "content": "{\"category\": \"feature_request\", \"priority\": \"low\"}"}]}

When the data is conversational, the trainer applies the model's chat template for you. Split the data into three files before training: train.jsonl, valid.jsonl for the loss curve, and test.jsonl that training never touches. A few checks pay off before the first run:

1. Every completion is an answer you would accept in production.
2. The label set is balanced enough that the model cannot score well by always guessing one class.
3. No test prompt appears, even reworded, in the training file.
4. Examples fit within max_length. TRL defaults to 1024 tokens and truncates longer sequences.

How to fine tune LLM LoRA adapters with PEFT and TRL

Install the three libraries. Each install command comes from that library's own docs.

pip install trl
pip install peft
pip install datasets

Then pass a LoraConfig to the trainer through peft_config. TRL wraps the model, trains only the adapter and saves it.

# train.py
import torch
from datasets import load_dataset
from peft import LoraConfig
from trl import SFTConfig, SFTTrainer

data = load_dataset(
    "json",
    data_files={"train": "train.jsonl", "validation": "valid.jsonl"},
)

peft_config = LoraConfig(
    r=16,
    lora_alpha=32,
    lora_dropout=0.05,
    target_modules="all-linear",
    task_type="CAUSAL_LM",
)

args = SFTConfig(
    output_dir="qwen3-tickets-lora",
    learning_rate=1e-4,
    num_train_epochs=3,
    eval_strategy="epoch",
    model_init_kwargs={"dtype": torch.bfloat16},
)

trainer = SFTTrainer(
    model="Qwen/Qwen3-0.6B",
    args=args,
    train_dataset=data["train"],
    eval_dataset=data["validation"],
    peft_config=peft_config,
)
trainer.train()
trainer.save_model()

The rank and alpha values are a starting point to tune, not a recommendation from the docs. A few other settings are worth knowing:

• target_modules="all-linear" applies LoRA to every linear layer except the output layer. The PEFT guide notes this matches the QLoRA approach.
• learning_rate=1e-4 follows TRL's tip for adapters. The SFTConfig default is 2e-5, which suits full fine-tuning.
• model_init_kwargs sets the load precision. Without a dtype, TRL loads the model in float32.
• bf16 defaults to True when fp16 is not set. On a GPU without bfloat16 support, set fp16=True and load in torch.float16 instead.
• gradient_checkpointing defaults to True in SFTConfig, which saves memory at some cost in speed.

For QLoRA, SFTTrainer also takes a quantization_config, which TRL documents as the way to combine quantized loading with peft_config.

Unsloth as the faster option

Unsloth (opens in a new tab) is an open-source library for training and running models that works with TRL's SFTTrainer. Its README claims fine-tuning that is 2x faster with 70% less VRAM and no accuracy loss. TRL's integration page quotes up to 80% less VRAM. Both figures are the project's own.

The change to the script is small. You load the model through FastLanguageModel and add LoRA with its get_peft_model, then hand the result to SFTTrainer without a peft_config. This version follows TRL's Unsloth integration page, which loads a 7B Mistral model in 4-bit.

from trl import SFTConfig, SFTTrainer
from unsloth import FastLanguageModel

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="unsloth/mistral-7b",
    max_seq_length=2048,
    dtype="auto",
    load_in_4bit=True,
)

model = FastLanguageModel.get_peft_model(
    model,
    r=16,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
                    "gate_proj", "up_proj", "down_proj"],
    lora_alpha=16,
    lora_dropout=0,
    bias="none",
    use_gradient_checkpointing=True,
    random_state=3407,
)

trainer = SFTTrainer(
    model=model,
    args=SFTConfig(output_dir="./output", max_length=2048),
    train_dataset=dataset,
)
trainer.train()

TRL's page installs it with pip install unsloth. The README's current Linux route uses a uv environment and uv pip install unsloth --torch-backend=auto. Unsloth also ships a web UI, Unsloth Studio, started with unsloth studio. The core package is Apache 2.0, and optional parts such as Studio are AGPL-3.0.

Testing the adapter against the base model

Training loss only says the model fits the training data. The test that matters is whether the adapter beats the base model on test.jsonl. PEFT makes this cheap: disable_adapter() is a context manager that runs the same loaded model as the base model, so you only load one copy.

# compare.py
import json

from peft import AutoPeftModelForCausalLM
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3-0.6B")
model = AutoPeftModelForCausalLM.from_pretrained("qwen3-tickets-lora")
model.eval()


def reply(messages):
    inputs = tokenizer.apply_chat_template(
        messages, tokenize=True, add_generation_prompt=True, return_tensors="pt"
    ).to(model.device)
    outputs = model.generate(**inputs, max_new_tokens=128, do_sample=False)
    new_tokens = outputs[0][inputs["input_ids"].shape[1]:]
    return tokenizer.decode(new_tokens, skip_special_tokens=True)


with open("test.jsonl") as f:
    for line in f:
        row = json.loads(line)
        tuned = reply(row["prompt"])
        with model.disable_adapter():
            base = reply(row["prompt"])
        expected = row["completion"][0]["content"]
        print(json.dumps({"expected": expected, "base": base, "tuned": tuned}))

Score the output with a rule that fits the task. For the triage example, parse the JSON and count exact matches on category and priority. Also run a handful of general prompts outside the task. A narrow adapter that breaks ordinary answers can still be the right result, but you should know before you ship it.

Merging, exporting and running it locally

An adapter needs its base model at load time. To get one standalone model, merge the adapter into the base weights. The PEFT docs stress that merge_and_unload() returns a new model rather than working in place, so assign the result.

# merge.py
from peft import AutoPeftModelForCausalLM
from transformers import AutoTokenizer

model = AutoPeftModelForCausalLM.from_pretrained("qwen3-tickets-lora")
merged = model.merge_and_unload()
merged.save_pretrained("qwen3-tickets-merged")
AutoTokenizer.from_pretrained("Qwen/Qwen3-0.6B").save_pretrained("qwen3-tickets-merged")

To run the merged model in Ollama, point a Modelfile at the directory. Ollama's Modelfile docs say the directory must hold Safetensors weights for a supported architecture, so check that list for your base model.

FROM ./qwen3-tickets-merged

ollama create qwen3-tickets -f ./Modelfile
ollama run qwen3-tickets

If you trained with Unsloth, it can write GGUF files directly through llama.cpp, for example model.save_pretrained_gguf("directory", tokenizer, quantization_method="q4_k_m"). A Modelfile can then use FROM with the path to the .gguf file it writes. For other runners such as LM Studio, check which formats they import before you choose an export path.