DoRA

DoRA, or Weight-Decomposed Low-Rank Adaptation, is designed for fine-tuning large language models by decomposing pre-trained weights into magnitude and direction components. This approach employs LoRA for efficient directional updates, enhancing learning capacity and training stability while minimizing trainable parameters. It is particularly useful for tasks involving commonsense reasoning, visual instruction tuning, and image/video-text understanding.