OwLore: Outlier-weighed Layerwise Sampled Low-Rank Projection for Memory-Efficient LLM Fine-tuning
Paper
•
2405.18380
•
Published
•
1
Liu
Shiweiliuiiiiiii
AI & ML interests
LLM, reasoning, ML efficiency
Organizations
None yet
authored
16
papers
11 months ago
FFN-SkipLLM: A Hidden Gem for Autoregressive Decoding with Adaptive Feed Forward Skipping
Paper
•
2404.03865
•
Published
Found in the Middle: How Language Models Use Long Contexts Better via Plug-and-Play Positional Encoding
Paper
•
2403.04797
•
Published
•
1
The Unreasonable Effectiveness of Random Pruning: Return of the Most Naive Baseline for Sparse Training
Paper
•
2202.02643
•
Published
•
1
Sparse Training via Boosting Pruning Plasticity with Neuroregeneration
Paper
•
2106.10404
•
Published
•
1
The Emergence of Essential Sparsity in Large Pre-trained Models: The Weights that Matter
Paper
•
2306.03805
•
Published
•
1
Dynamic Sparse No Training: Training-Free Fine-tuning for Sparse LLMs
Paper
•
2310.08915
•
Published
AdaMerging: Adaptive Model Merging for Multi-Task Learning
Paper
•
2310.02575
•
Published
•
1
Sparse MoE as the New Dropout: Scaling Dense and Self-Slimmable Transformers
Paper
•
2303.01610
•
Published
Q-GaLore: Quantized GaLore with INT4 Projection and Layer-Adaptive Low-Rank Gradients
Paper
•
2407.08296
•
Published
•
33
From GaLore to WeLore: How Low-Rank Weights Non-uniformly Emerge from Low-Rank Gradients
Paper
•
2407.11239
•
Published
•
8
O1-Pruner: Length-Harmonizing Fine-Tuning for O1-Like Reasoning Pruning
Paper
•
2501.12570
•
Published
•
28
The Curse of Depth in Large Language Models
Paper
•
2502.05795
•
Published
•
40
Mask-Enhanced Autoregressive Prediction: Pay Less Attention to Learn More
Paper
•
2502.07490
•
Published
•
10
Stable-SPAM: How to Train in 4-Bit More Stably than 16-Bit Adam
Paper
•
2502.17055
•
Published
•
20
SoS1: O1 and R1-Like Reasoning LLMs are Sum-of-Square Solvers
Paper
•
2502.20545
•
Published
•
22