⚡️ PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models
Significantly improved finetuned perf by simply changing the initialization of LoRA's AB matrix from Gaussian/zero to principal components.
On GSM8K, Mistral-7B fine-tuned with PiSSA achieves an accuracy of 72.86%, outperforming LoRA’s 67.7% by 5.16%.
▪Github: https://github.com/GraphPKU/PiSSA
▪Paper: https://arxiv.org/abs/2404.02948
@opendatascience
>>Click here to continue<<