Fine-Tuning Hyperparameters & Calibration Guide
Parameter-efficient fine-tuning (PEFT) on consumer hardware requires rigorous hyperparameter calibration. This guide details the mathematical foundations and operational rules enforced by FineTuneMyAI to ensure training convergence and eliminate Out-Of-Memory (OOM) failures.
Guarantees consistent gradient magnitude regardless of rank dimension updates.
Reserved memory cushion absorbing variable sequence lengths and PyTorch buffer spikes.
Maintained via 16 accumulation steps with micro-batch size 1 on consumer cards.
1. LoRA Rank (r) and Scaling Alpha (α)
In Low-Rank Adaptation (LoRA), weight updates ΔW are decomposed into two low-rank matricesW_0 + ΔW = W_0 + B × A, where B ∈ R^(d × r) andA ∈ R^(r × k) with rankr ≪ min(d, k).
Recommended Rank Settings:
- Rank 8 (α = 16): Ideal for stylistic shifts, tone alignment, and classification tasks with minimal VRAM overhead (<50MB adapter).
- Rank 16 (α = 32): Default production sweet spot balancing factual domain adaptation and compute efficiency.
- Rank 32 (α = 64): Deep reasoning, syntax instruction following, and mathematical workflows requiring extensive parameter capacity.
2. Memory Headroom & OOM Prevention
The most common reason local fine-tuning fails is CUDA or MPS memory fragmentation. FineTuneMyAI incorporates an automated Memory Safety Calibrator that profiles the system before training commences:
If the projected sequence memory exceeds 82% of total GPU memory, the engine automatically recommends reducing max sequence length (e.g. from 2048 to 1024) or switching to 4-bit QLoRA.
3. Batch Size & Gradient Accumulation
Small micro-batch sizes introduce stochastic noise into parameter updates. To stabilize convergence without demanding datacenter VRAM, FineTuneMyAI enforces gradient accumulation:
| Hardware Tier | Micro Batch | Accumulation Steps | Effective Batch | Gradient Checkpointing |
|---|---|---|---|---|
| 8GB GPU / M1 Mac | 1 | 16 | 16 | Enabled (Mandatory) |
| 12GB - 16GB GPU | 2 | 8 | 16 | Enabled |
| 24GB+ (RTX 4090 / A10) | 4 | 4 | 16 | Optional |
4. Target Modules & Optimizer Policies
Targeting only attention query and value projections (q_proj, v_proj) yields inferior results compared to adapting all linear layers in the transformer block:
Default Target Modules (Llama, Mistral, Qwen, Gemma)
For optimizer stability, FineTuneMyAI employs Paged AdamW 8-bit (NVIDIA) orAdamW FP32 with weight decay 0.01 (Apple Silicon MPS), paired with a cosine learning rate schedule and 3% linear warmup.