‘I’m not sure’—AI finally learns three words that could make its biggest mistakes far less dangerous
Credit: Image generated by the editorial team using AI for illustrative purposes.
A new approach has been proposed to address the problem of “overconfidence”—one of the most critical risks of artificial intelligence (AI) in areas such as autonomous driving and medical diagnosis, where AI shows high confidence in incorrect predictions. A KAIST research team has developed a training method that enables AI to recognize situations involving unfamiliar or unseen knowledge, laying the foundation for reducing overconfidence and improving reliability.
Pinpointing the roots of overconfidence
The team, led by Distinguished Professor Se-Bum Paik from the Department of Brain and Cognitive Sciences, has identified that random initialization—widely used in deep learning (an AI technique that learns from data using artificial neural networks)—may be a fundamental cause of overconfidence in AI. To address this, they propose a “warm-up” strategy in which the neural network is briefly trained using random noise (meaningless arbitrary input data) before learning from real data. The findings are published in the journal Nature Machine Intelligence.
The research team found that AI overconfidence already appears at the initialization stage, which can propagate and cause significant errors during subsequent training. In fact, when random data were input into a randomly initialized neural network, the model exhibited high confidence despite not having learned anything. This characteristic can lead to hallucinations in generative AI, where false information is produced in a plausible manner.
The research team found clues for solving this issue in the biological brain. The human brain forms neural circuits through “spontaneous neural activity”—brain signals generated without external input—even before birth.
Warm-up training with random noise enables confidence calibration in neural networks. Credit: Nature Machine Intelligence (2026). DOI: 10.1038/s42256-026-01215-x
Borrowing ideas from brain development
Applying this concept to artificial neural networks, the researchers introduced a “warm-up phase” in which the network undergoes brief pre-training with random noise inputs before actual learning. This corresponds to a process in which AI adjusts its own uncertainty before starting data learning. After the warm-up process, the AI model’s initial confidence is aligned to a low level close to chance, significantly reducing the overconfidence bias observed in conventional initialization.
In other words, before learning from real data, the model first learns the state of “I don’t know anything yet.” As a result, the model’s accuracy (how often predictions are correct) and confidence (how strongly the model believes its predictions) naturally become aligned.
A notable difference was also observed in responses to unseen data. While conventional models tend to give incorrect answers with high confidence even for data they have not encountered during training, models with warm-up training showed a clear improvement in their ability to lower confidence and recognize that they “do not know.”
Implications for safer, more reliable AI
This also led to strong performance in out-of-distribution detection, which refers to identifying data that differ from the training distribution.
This study suggests the possibility that AI can go beyond simply producing correct answers and develop the ability to distinguish “what it knows” from “what it does not know”—that is, meta-cognition, the ability to recognize its own cognitive state.
Professor Paik stated, “This study demonstrates that by incorporating key principles of brain development, AI can recognize its own knowledge state in a way that is more similar to humans,” adding, “This is important because it helps AI understand when it is uncertain or might be mistaken, not just improve how often it gives the right answer.”
This technology is expected to be applied not only to fields requiring high reliability, such as autonomous driving, medical AI, and generative AI, but also to the initialization methods of nearly all deep learning models, making it a key technology for improving overall AI reliability.
Publication details
Jeonghwan Cheon et al, Brain-inspired warm-up training with random noise for uncertainty calibration, Nature Machine Intelligence (2026). DOI: 10.1038/s42256-026-01215-x
Key concepts
Trustworthy machine learningMachine learning methodologies
Provided by
The Korea Advanced Institute of Science and Technology (KAIST)
Who’s behind this story?
Lisa Lock
Full profile →
Robert Egan
Full profile →
Citation:
‘I’m not sure’—AI finally learns three words that could make its biggest mistakes far less dangerous (2026, May 10)
retrieved 10 May 2026
from https://techxplore.com/news/2026-05-im-ai-words-biggest-dangerous.html
This document is subject to copyright. Apart from any fair dealing for the purpose of private study or research, no
part may be reproduced without the written permission. The content is provided for information purposes only.
