Language & LLMs
Optimized prompt tuning can match fine-tuning across model sizes and language tasks
In a 2022 paper in the Annual Meeting of the Association for Computational Linguistics, researchers from Tsinghua University, the Beijing Academy of Artificial Intelligence, and the Shanghai Qi Zhi Institute describe a method called P-Tuning v2 that can match the performance of fine-tuning on natural language understanding tasks while tuning only 0.1% to 3% of a model's parameters, the adjustable numeric weights inside the model. Prompt tuning keeps a language model frozen and trains only a small set of continuous prompt inputs. Earlier methods struggled with normal-sized pretrained models and with harder sequence-labeling tasks. The researchers found that properly optimized prompt tuning can close that gap.
Why did earlier prompt tuning fall short?
The paper reports that prior prompt tuning did not perform well for normal-sized pretrained models. The authors also found that existing methods could not handle hard sequence-labeling tasks, which they say pointed to a lack of universality.
What did the study find?
The researchers found that properly optimized prompt tuning can be effective across a wide range of model scales and natural language understanding tasks. It matched the performance of fine-tuning while tuning only 0.1% to 3% of parameters. Their method, P-Tuning v2, is an implementation of deep prompt tuning optimized and adapted for natural language understanding.
Why does tuning fewer parameters matter?
Because the language model stays frozen, prompt tuning substantially reduces per-task storage and memory usage during training. The authors believe the method's universality and simplicity make it an alternative to fine-tuning and a strong baseline for future research.
Limits and what's next
This is one empirical study, and the abstract reports no independent replication. The experiments concern natural language understanding tasks, including sequence labeling, but the abstract does not name the datasets or the exact range of model sizes. The comparison is with fine-tuning in the settings the authors tested. The abstract does not report results outside natural language understanding. The authors state that they believe P-Tuning v2 can serve as an alternative to fine-tuning and as a baseline for future research, which is an aim, and the abstract does not report a separate measured result for that role.
Source: Xiao Liu et al., “P-Tuning: Prompt Tuning Can Be Comparable to Fine-tuning Across Scales and Tasks,” Annual Meeting of the Association for Computational Linguistics (ACL), 2022.