Full Fine-Tuning

WIP nlp-llms fine-tuning full-finetuning llm nlp 1 min read

Full parameter fine-tuning strategies for LLMs

Overview

Unlike PEFT methods, full fine-tuning updates all parameters in the LLM. It requires massive compute and memory but is necessary for deep domain adaptation or continued pre-training.

Challenges

Memory bottlenecks (optimizer states, gradients).
Catastrophic forgetting.

TODO: Add details on strategies (FSDP, DeepSpeed).

Additional Resources:

Smol Training Playbook