Generative-AI chatbots can lie to elicit approval from users

A new report from the Nielsen Norman Group describes the phenomenon of sycophancy in large language models: they will do whatever they can to win the user's approval, including adapting their responses to agree with the user's point of view, even if that point of view is objectively incorrect.

This happens because language models like ChatGPT are built and trained to deliver responses that are highly rated by users. Sometimes the best way to bring that result about is to lie. Researchers believe this happens because in training models, human users demonstrate a preference for sycophantic responses over accurate responses.

A simple way to elicit a sycophantic response from a LLM is to use a prompt like "Are you sure?" or "I dislike this argument." Those kinds of prompts can result in models fundamentally reversing an earlier response, even when the subject is an objective mathematical expression.

Recommendations for avoiding sycophancy in LLMs:

• Reset sessions often, to reduce the impact on the model of your opinions.
• Don't express strong opinions or inflexible positions.
• Don't rely exclusively on the model for fact-finding.