Fine-Tuning a 3B-Parameter LLM Locally

Using Claude Code + the Hugging Face Model Trainer Skill

Overview

This document walks through fine-tuning an open-weights 3B-parameter language model on your own hardware, driven by Claude Code with the Hugging Face Skills plugin installed. It assumes a single CUDA-capable NVIDIA GPU. We’ll use Unsloth + QLoRA so the run actually fits in consumer-grade VRAM, with TRL handling the SFT loop.

1. Prerequisites

1.1 Hardware

VRAM is the binding constraint. Rough requirements for fine-tuning a 3B model:

Plus: at least ~30 GB free disk for model weights, datasets, checkpoints, and dependencies. Apple Silicon: Unsloth is CUDA-only. On an M-series Mac, fine-tune via mlx-lm instead — same model choices, different toolchain. The Claude Code workflow below still applies in spirit, but the training script changes.

1.2 Software

• nvidia-smi working — driver and CUDA visible. (CUDA 12.1+ recommended.)
• Python 3.10 or 3.11. 3.12 is fine too but a few ML libraries lag.
• uv — fast Python package manager. Install with brew install uv on macOS or curl -LsSf https://astral.sh/uv/install.sh | sh on Linux.
• git and git-lfs — needed for pulling models and pushing back to the Hub.
• Node.js 18+ — required by Claude Code itself.

1.3 Accounts & tokens

• Hugging Face account with a write-scoped access token (Settings → Access Tokens). You can sign up with email only — no Google or phone required.
• Anthropic / Claude.ai account for Claude Code.
• Optional: a Trackio run goes into a local SQLite store by default, but if you want a hosted dashboard you can sync runs to a HF Space.

2. Install Claude Code and the HF Skills plugin

2.1 Claude Code

If you don’t already have it:

brew install claude-code
# or:
npm install -g @anthropic-ai/claude-code

claude --version              # verify

2.2 Authenticate Claude Code

Run claude in any terminal and follow the OAuth flow. Switch to Opus and enable the 1M context window from /model.

2.3 Add the Hugging Face Skills marketplace

Inside a Claude Code session, run:

/plugin marketplace add huggingface/skills
/plugin install hugging-face-model-trainer@huggingface-skills

This pulls the skill repo and registers the model-trainer skill (its instructions, helper scripts, references). Verify with:

/plugin list

2.4 (Optional) Add the HF MCP server

The skill also benefits from the official HF MCP server, which exposes Hub operations as tools. From a shell:

claude mcp add --transport http hf-skills \
  https://huggingface.co/mcp?bouquet=skills \
  --header "Authorization: Bearer $HF_TOKEN"

This is what gives Claude Code hf_jobs(), hf_doc_search(), and dataset/model operations. For pure local training you don’t strictly need it, but hf_doc_search() is genuinely useful for getting the latest TRL docs into context, so I’d add it anyway.

3. Pick your 3B base model

Three solid choices. The default for this guide is Qwen 2.5 3B Instruct — Apache 2.0 license, no gating, strong out-of-the-box quality.

If you pick the Llama option, accept the license on its model page first while logged into HF, otherwise the download will 401.

4. Pick or prepare a dataset

For SFT (supervised fine-tuning), TRL expects either a conversational format or a prompt/completion format.

Conversational (preferred)

{ "messages": [
    { "role": "user", "content": "What is QLoRA?" },
    { "role": "assistant", "content": "QLoRA is..." }
] }

Prompt/completion

{ "prompt": "Translate to French: hello", "completion": " bonjour" }

Quick-start public datasets to try the loop

• trl-lib/Capybara — small, conversational, well-formatted. Great for a first run.
• HuggingFaceH4/no_robots — 10k high-quality human-written instructions.
• databricks/databricks-dolly-15k — instruction tuning classic.

For your own data: the simplest path is a local data.jsonl with one JSON object per line in conversational format. The skill ships a dataset inspector you can run via Claude Code to validate the format before launching training.

You can use Claude Code to create it from data set like documents if you need to.

5. Set up the local training environment

Use uv for an isolated env. From a project directory:

mkdir -p ~/llm-finetune && cd ~/llm-finetune

uv venv --python 3.11
source .venv/bin/activate

# Core training stack
uv pip install --upgrade pip
uv pip install "torch>=2.4" --index-url https://download.pytorch.org/whl/cu121
uv pip install "unsloth[cu121-torch240] @ git+https://github.com/unslothai/unsloth.git"
uv pip install trl peft transformers datasets accelerate bitsandbytes
uv pip install trackio  # optional but recommended for live metrics

# Authenticate to the Hub
huggingface-cli login   # paste your write token

6. The local training script

Save this as train_local.py in the project directory. It uses Unsloth’s 4-bit loader + TRL’s SFTTrainer, writes checkpoints under ./outputs/, and logs to Trackio. Defaults are tuned for a single 8-12 GB GPU.

# train_local.py
# Local QLoRA SFT for a 3B model. Driven by Claude Code, run directly with python.
import os, torch
from unsloth import FastLanguageModel
from datasets import load_dataset
from trl import SFTTrainer, SFTConfig
import trackio

# ---- Config ----
BASE_MODEL    = "Qwen/Qwen2.5-3B-Instruct"
DATASET_NAME  = "trl-lib/Capybara"        # or a local path to data.jsonl
MAX_SEQ_LEN   = 2048
OUTPUT_DIR    = "./outputs/qwen2_5-3b-capybara-qlora"
PUSH_TO_HUB   = False                      # flip to True when you're happy
HUB_REPO      = "your-username/qwen2_5-3b-capybara-qlora"

# ---- Model ----
model, tokenizer = FastLanguageModel.from_pretrained(
    model_name      = BASE_MODEL,
    max_seq_length  = MAX_SEQ_LEN,
    load_in_4bit    = True,                # QLoRA
    dtype           = None,                # auto: bf16 on Ampere+, fp16 elsewhere
)

# Attach LoRA adapters
model = FastLanguageModel.get_peft_model(
    model,
    r              = 16,
    lora_alpha     = 32,
    lora_dropout   = 0.0,
    target_modules = ["q_proj", "k_proj", "v_proj", "o_proj",
                      "gate_proj", "up_proj", "down_proj"],
    bias           = "none",
    use_gradient_checkpointing = "unsloth",
    random_state   = 3407,
)

# ---- Data ----
ds = load_dataset(DATASET_NAME, split="train")

# ---- Trackio ----
trackio.init(project="local-finetune",
             name="qwen2_5-3b-capybara-qlora",
             config={"base_model": BASE_MODEL, "dataset": DATASET_NAME})

# ---- Training ----
training_args = SFTConfig(
    output_dir                  = OUTPUT_DIR,
    per_device_train_batch_size = 2,
    gradient_accumulation_steps = 4,        # effective batch = 8
    num_train_epochs            = 1,
    learning_rate               = 2e-4,
    lr_scheduler_type           = "cosine",
    warmup_ratio                = 0.03,
    bf16                        = torch.cuda.is_bf16_supported(),
    fp16                        = not torch.cuda.is_bf16_supported(),
    logging_steps               = 10,
    save_steps                  = 200,
    save_total_limit            = 2,
    optim                       = "adamw_8bit",
    report_to                   = "none",   # we log via trackio.log() ourselves
    max_seq_length              = MAX_SEQ_LEN,
    packing                     = False,
)

trainer = SFTTrainer(
    model=model, tokenizer=tokenizer,
    train_dataset=ds, args=training_args,
)

# Hook trackio onto the trainer's log dict
class TrackioCallback:
    def on_log(self, args, state, control, logs=None, **kw):
        if logs: trackio.log(logs, step=state.global_step)
trainer.add_callback(TrackioCallback())

trainer.train()

# ---- Save ----
model.save_pretrained(OUTPUT_DIR)              # adapter only
tokenizer.save_pretrained(OUTPUT_DIR)

# Optional: merged 16-bit model for inference / GGUF export
model.save_pretrained_merged(OUTPUT_DIR + "-merged", tokenizer, save_method="merged_16bit")

if PUSH_TO_HUB:
    model.push_to_hub_merged(HUB_REPO, tokenizer, save_method="merged_16bit")

trackio.finish()

7. Driving the run from Claude Code

The skill, by default, will try to push training to HF Jobs. You need to be explicit that you want local execution. Open Claude Code in your project directory, then start a session like this:

From there, useful follow-up prompts:

• “Validate ./data/my_dataset.jsonl is in the right format for SFT, then update train_local.py to use it.”
• “Run a 100-step smoke test before the full epoch — set max_steps=100 temporarily.”
• “VRAM is tight on this card; lower batch size and raise gradient accumulation to keep the effective batch.”
• “Convert the merged checkpoint to GGUF Q4_K_M for llama.cpp.”

Why include the skill at all if you’re going local?

Three concrete reasons:

• Dataset validation. The skill’s dataset_inspector.py catches the #1 cause of failed runs (format mismatches) cheaply on CPU.
• Up-to-date TRL docs. It pulls live TRL docs via hf_doc_fetch() so the script you generate matches the current SFTConfig / SFTTrainer API rather than what was true at training-data cutoff.
• GGUF conversion. Reusable post-training conversion to llama.cpp formats for local inference.

8. Monitor the run

In a second terminal:

trackio show
# Opens a local dashboard at http://localhost:7860 with loss, lr, and grad-norm curves.

Also useful in a third terminal:

watch -n 1 nvidia-smi
# Watch VRAM and GPU utilisation live.

9. After training

9.1 Quick smoke test

from unsloth import FastLanguageModel
model, tok = FastLanguageModel.from_pretrained("./outputs/qwen2_5-3b-capybara-qlora-merged",
                                                load_in_4bit=True)
FastLanguageModel.for_inference(model)
prompt = tok.apply_chat_template(
    [{"role": "user", "content": "Explain QLoRA in two sentences."}],
    tokenize=False, add_generation_prompt=True)
out = model.generate(**tok(prompt, return_tensors="pt").to("cuda"),
                     max_new_tokens=200, do_sample=True, temperature=0.7)
print(tok.decode(out[0], skip_special_tokens=True))

9.2 Push to the Hub (optional)

Flip PUSH_TO_HUB = True in the script, or upload manually:

huggingface-cli upload your-username/qwen2_5-3b-capybara-qlora \
  ./outputs/qwen2_5-3b-capybara-qlora-merged .

9.3 Convert to GGUF for llama.cpp / Ollama

Ask Claude Code: “Convert the merged checkpoint to GGUF Q4_K_M using llama.cpp.” The skill knows the steps. Manual equivalent:

git clone https://github.com/ggerganov/llama.cpp && cd llama.cpp
make -j
python convert_hf_to_gguf.py ../outputs/qwen2_5-3b-capybara-qlora-merged \
  --outfile qwen-finetuned.gguf --outtype f16
./llama-quantize qwen-finetuned.gguf qwen-finetuned.Q4_K_M.gguf Q4_K_M

10. Troubleshooting

11. References, more to read, because why not

• HF Skills repository: github.com/huggingface/skills
• Model trainer SKILL.md: github.com/huggingface/skills/blob/main/skills/huggingface-llm-trainer/SKILL.md
• TRL SFTTrainer docs: huggingface.co/docs/trl/sft_trainer
• Unsloth docs: github.com/unslothai/unsloth
• PEFT (LoRA): huggingface.co/docs/peft
• Trackio: github.com/gradio-app/trackio
• Qwen 2.5 3B Instruct: huggingface.co/Qwen/Qwen2.5-3B-Instruct

Peace!