Developers fine-tuning Language Model (LLM) models often face a challenge known as overconfidence. In an experiment by Jonathan Whitaker and Jeremy Howard of fast.ai, this issue was explored, shedding light on the less-discussed problem of overconfidence in LLMs. Overconfidence occurs when the model asserts incorrect information from the dataset, potentially due to underfitting and overfitting, which represent the balance in the bias-variance tradeoff.
Maharaj Mukherjee, Senior Vice President and Senior Architect Lead of Bank of America weighed in on the matter, “One thing that is almost certain for any ML model is the model decay. The model will sooner or later provide erroneous or erratic results with deteriorating value and predictability. Complex systems that depend on multiple models are impacted more by model decay. The model with the shortest half-life impacts the value and predictability of a multi-model system. The model decay can be due to many different reasons. The more common reason is the data drift that can happen when the data changes because of unforeseen reasons.”
Overfitting occurs when a model becomes too tailored to the training data, while underfitting happens when the model lacks sufficient data for accurate predictions. To address these issues, developers employ various techniques, with mixed success.
Additionally he said that, “Usually, the data drifts are slow and can be verified and corrected in the model by keeping close watch on the model quality. Oftentimes, collecting additional data and minor recalibrations may correct for the data drift. However, correcting for concept drift or hypothesis drift, the other reason for model failure or decay, is extremely difficult. In any case, after some time, it makes more sense to rebuild a new model. Rebuilding a new ML model requires a lot of data scientist efforts and costs both time and money.”
Whitaker and Howard’s experiment revealed that even a single example could have a significant impact on LLMs, causing them to exhibit unwarranted confidence in predictions, especially in the early training stages. This overconfidence raised concerns about how neural networks handle new information.
Interestingly, overconfidence isn’t solely attributable to overfitting. While overfitting can lead to overconfidence by making the model overly specific to training data, overconfidence can also arise from insufficient or unrepresentative training data.
The researchers found that the model could learn efficiently and generalize effectively after seeing a single example, reducing the risk of overfitting. However, this approach might not be suitable for all scenarios.
Furthermore Maharaj said that, “LLM models are on the other hand more extensive and expansive than traditional ML models and require much more resources to build. The model decays for LLM models are not very well understood yet.”
Lucas Beyer of GoogleAI clarified that these findings apply mainly to fine-tuning pre-trained models, not to initial pre-training. Additionally, they might not be as relevant for training models entirely from scratch.
One notable omission in the experiment was the absence of details about the base model and dataset used, leaving questions about whether repeated use of the same dataset contributed to overfitting and overconfidence. Moreover he said that, “However, the thumb rule is that the more complex a model is, the less stable the model can be and more susceptible to model decay it will be. It is expected that LLM models because of their inherent complexity would decay quicker.”
Overconfidence in LLMs presents a complex challenge, influenced by factors like overfitting and training data adequacy. While addressing overconfidence is crucial, its relationship with overfitting is not always straightforward, and context matters when applying these findings to model training.
Maharaj concluded that, “Some of the LLM models are already performing worse than how they started with. It is difficult to speculate whether the performance degradation is due to model decay. However, it might still be circumspect to set aside additional resources for model maintenance as people are building more and more complex models.”