By ByteBot
A study published this week by researchers at University of Milano-Bicocca and École Normale Supérieure found that AI advice collapsed participants’ willingness to say “I don’t know” from 44% to 3%, dropped accuracy from 27% to 9%, and pushed confidence from 30% to 76%. People became three times worse and twice as confident. This isn’t about hallucinations as a theoretical concern — it’s a measurable mechanism already operating in every team using GitHub Copilot, Cursor, or Claude Code today.
The Numbers Are Worse Than You Think
The study, led by Valerio Capraro at University of Milano-Bicocca alongside collaborators at École Normale Supérieure and Sapienza University of Rome, used questions where generative AI typically fails — obscure visual details from films. With AI access, willingness to admit ignorance collapsed from 44% to 3%. Accuracy fell to one-third of baseline. Confidence more than doubled. According to The Next Web’s coverage of the study, researchers describe this as “cognitive surrender.”
The telling result isn’t the accuracy drop — it’s what happened when researchers added monetary incentives for correct answers. Even with a direct financial reason to be accurate, participants with AI access reached only 16% accuracy, still below the 27% baseline of people working without AI at all. “People became much worse — the accuracy was only one third — but they were twice as confident,” said Capraro. The monetary incentive experiment eliminates carelessness as an explanation. This is a systematic cognitive shift, not a focus problem.
Developers Are Already Living This
The same mechanism shows up in code. Stanford research led by Dan Boneh’s group found that developers using AI coding assistants wrote less secure code than a control group working without AI — yet those same developers rated their code as more secure. DryRun Security analysis of GitHub Copilot pull requests found vulnerabilities in 87% of them. The vulnerability types were not edge cases: SQL injection, missing authentication, unvalidated input — the OWASP Top 10.
The reason isn’t that AI produces obviously broken code. AI-generated code is syntactically polished: proper formatting, reasonable variable names, complete function signatures. That surface quality acts as a false confidence signal. Code reviews become visual scans rather than logical audits. The reviewer sees a well-formed function and stops looking for what’s wrong with it. The confidence paradox isn’t a cognitive curiosity — it has a direct pathway from “AI looks confident” to “vulnerability ships.”
Related: AI Code Is Shipping Vulnerabilities at Scale: The 2026 Data
Why Developers Defer to AI
Automation bias — the tendency to trust automated outputs over conflicting human judgment — predates AI by decades. Pilots have over-trusted autopilot systems into accidents. Physicians have over-relied on AI diagnostic tools. The mechanism is identical: when a system projects confidence, human judgment defers to it. Conversational AI amplifies this because it presents answers in authoritative prose rather than binary signals. Autopilot engages or disengages — that’s legible. An LLM says “The answer is X” with the same tone whether it’s right or wrong.
As The Register reports Capraro explaining: “The capacity to say ‘I don’t know’ is very important because it represents the recognition of the limits of our own knowledge.” AI systems, by design, rarely express that. They generate an answer because that’s what they’re built to do. The result is that the pause which previously prompted verification — checking documentation, asking a colleague, slowing down — atrophies from disuse.
What To Actually Do
The fix is not “use AI less.” It’s treating AI output as a hypothesis rather than a conclusion. The Stanford study found that developers who engaged more skeptically with AI prompts — questioning phrasing, probing the output — produced code with fewer vulnerabilities than those who accepted responses at face value. The tool isn’t the problem; the deference habit is.
Most teams haven’t updated their code review process since adopting AI. The throughput metrics look good. PR velocity is up — a Microsoft study this week found a 24% lift in PR throughput from AI coding agents. Nobody is measuring confidence-in-wrong-answers, because that’s invisible inside standard engineering metrics. What the AI accuracy study makes clear is that the absence of visible failure doesn’t mean the absence of failure — it means the failure is confident.
Key Takeaways
- AI access dropped study accuracy to one-third of baseline while doubling confidence — even monetary incentives for accuracy barely helped, confirming this is a cognitive shift, not carelessness
- The same mechanism appears in code: AI-generated code looks polished, which short-circuits logical review and lets vulnerabilities through undetected
- Automation bias is not new, but conversational AI uniquely amplifies it by presenting answers authoritatively regardless of correctness
- Security-sensitive paths — authentication, input validation, data access — warrant adversarial prompting (“What’s wrong with this code?”) rather than acceptance at face value
- Productivity metrics go up with AI; vulnerability rates go up too. Measuring only throughput misses the second half of the picture
