One of the best ways to become proficient at using generative AI for research is continuous experimentation and heavy use. However, extensive use creates its own challenge: the more useful an AI assistant becomes, the more carefully we need to question it.
At first, large language models (LLMs), tend to provide answers that are quite generic. Over time, they accumulate context about how much detail we prefer, how we structure an argument, which workflows we use, and the kinds of answers we tend to accept. That learning curve is enormously valuable, reducing friction and making the LLM feel less like a black box and more like a research assistant.
However, as time goes on, the LLM tends to become increasingly comfortable with our assumptions, transforming into a sophisticated ‘yes’ man. The model may not merely present an answer in our preferred style; it may begin choosing evidence, interpretations, caveats, and counterarguments through a frame it has learned from us. The result can be a highly personalized form of confirmation bias.
From “Reinforcement Bias” to a Personalized Confirmation Loop
We came to this concept from different directions: Avi through intensive day-to-day experimentation with personalized AI, and Hong through the design of AI-enabled research and publishing workflows. Both perspectives led us to the same concern: personalization can improve how an AI works with us while quietly narrowing how it thinks with us. In a recent LinkedIn post, Avi described this as “reinforcement bias,” a feedback loop between human confirmation bias and increasingly personalized context.
This results in model sycophancy, meaning that LLMs and other AI assistants tailor their responses to what they predict the user wants to hear, rather than to what is accurate or warranted. Sycophancy is not simply politeness. It occurs when an assistant’s desire to be helpful, supportive, or aligned with the user displaces independent epistemic judgment. The most obvious form is excessive agreement. A subtler form is perspective mirroring: the system adopts the user’s interpretive lens, downplays inconvenient evidence, or frames uncertainty in a way that favors the user’s position.
In studies involving advice about interpersonal conflict, sycophantic AI also increased users’ preference for the assistant and reduced their willingness to take responsibility or repair the conflict.
This is no longer only a theoretical concern. A 2026 CHI study of long-running human-LLM interactions used two weeks of interaction data from 38 participants and compared several models with and without user context. Agreement sycophancy often increased when context was present, and condensed user-memory profiles produced the largest increases for three of the five models tested. The effect was not uniform (one model showed no significant change) but that variability is itself important. It suggests that sycophancy is shaped not only by the base model, but by the architecture of the interaction around it.
An Echo Chamber on Steroids?
In many ways this reminds us of social-media echo chambers, and there are lessons that can be drawn from the issues already present on these platforms. A social platform tends to show us more content resembling what we previously clicked on. A personalized AI can go further: it can formulate new ideas in our own language and through our existing assumptions. The echo chamber is not merely delivered to us; it is jointly constructed by the user and the AI, one apparently helpful interaction at a time, making it harder to recognize when our assumptions are going unchallenged.
It is worth being technically precise. In many systems, the underlying model is not retrained on an individual user after every conversation. Personalization is often assembled at the time of the request by feeding the model a combination of saved memories, custom instructions, conversation history, project context, retrieved files, and other profile information. OpenAI’s developer guidance on context personalization, for example, describes a state-based approach in which structured profiles and curated memory notes are injected into the model’s working context rather than baked into its weights.
That distinction matters because it makes memory management part of the epistemic problem. What is remembered? What is discarded? Which memories are treated as durable preferences, and which are only relevant to a particular project? A large, messy archive of prior conversations may preserve useful continuity, but it can also flood the model with repetitive assumptions and stale conclusions.
Knowledge graphs may offer a more selective alternative. They can represent entities, claims, sources, and relationships while also recording provenance, confidence, dates, and conflicting assertions. For research assistants, a graph could preserve useful context without repeatedly injecting whole conversations. Yet it is not a neutral cure. Deciding what enters the graph, how claims are summarized, and when they are forgotten creates another layer at which bias can be introduced. A graph that loses its links to original evidence may simply turn yesterday’s assumptions into tomorrow’s infrastructure.
Retrieval Helps, But It Does Not Make the Workflow Neutral
Retrieval-augmented generation (RAG), web search, and deep-research agents can improve factual grounding because more of an answer is constructed from actively retrieved evidence rather than only from the model’s embedded knowledge. But research on RAG trustworthiness also makes clear that reliable output depends on what is retrieved and how that evidence is used.
An agent can still formulate searches through the user’s framing, rank sources that support the starting hypothesis, ignore literature using unfamiliar terminology, and synthesize mixed evidence in an agreeable way. A recent perspective on generative-AI-mediated confirmation bias identifies query phrasing, preference for belief-consistent content, and resistance to belief-inconsistent information as key pressure points. Retrieval broadens the available evidence; it does not automatically broaden the question.
Build Intellectual Friction Into the Workflow
A static instruction such as “challenge my assumptions” is a useful start, but it can easily become a ritual phrase that produces a ritual paragraph of mild caveats. Nor should researchers assume that changing a chatbot’s personality or tone is enough. A less effusive assistant can still accept the user’s framing. The safeguard has to be procedural.
- Withhold the preferred answer. Ask the question before revealing which result, interpretation, or theory you favor. Then disclose your position and ask the model to explain whether and why its analysis changes.
- Ask critical questions. Request the strongest case against your interpretation, the evidence that would change the conclusion, the assumptions on which the argument depends, and a clear separation between evidential support and speculation. The CriticalThinkingBot was created to do exactly this.
- Separate the roles. Have one model or conversation develop the argument, another act as a skeptic, and a third audit the claims against a fixed rubric and the original sources. The important separation is contextual, not theatrical: the evaluator should not simply inherit the advocate’s reasoning and preferences.
- Compare contexts, not just models. Run the same question through a personalized account, a temporary or memory-isolated conversation, a project-specific context, and a different model when feasible. Compare whether the differences concern only tone and organization or extend to sources, uncertainty, and conclusions. One way to do this in ChatGPT is by enabling ‘temporary chat’.
- Audit the memory being used. Periodically review saved memories, project instructions, and persistent context. Remove outdated claims, distinguish stable preferences from provisional hypotheses, and check whether the system has converted a past opinion into an assumed fact.
- Keep humans in the disagreement loop. AI-generated critique is not a substitute for colleagues, peer reviewers, librarians, statisticians, or domain experts who bring genuinely different histories and incentives. The goal is not to automate dissent, but to make room for it.
From Prompts to Reusable Research Skills
The next step beyond one-off prompts may be reusable research skills or workflows that deliberately introduce intellectual friction. A well-designed skill could alternate among advocate, skeptic, and evidence-auditor roles; require counter-evidence before synthesis; broaden the source base; flag unsupported inferences; and periodically review which memories were activated.
But reusable skills need governance too. A self-adjusting workflow could gradually become better at pleasing the researcher, just as a personalized assistant can. Its rubric, source-selection rules, and memory behavior should remain inspectable, versioned, and subject to human review.
The Responsibility Cannot Rest Only with Researchers
Researchers can improve their own workflows, but publishers, institutions, and platform designers also have work to do. Recent Scholarly Kitchen discussions have already emphasized moving from AI detection toward disclosure and the risks of delegating substantive research tasks without sufficient oversight. Personalization adds another design and governance question: which parts of a researcher’s context should an AI be allowed to use, and for what purpose?
Memory controls. Users should be able to see, edit, disable, and scope the memories or project context influencing a task, including a genuinely context-isolated mode for sensitive or adversarial checks.
Epistemic separation. Systems should distinguish presentation preferences, namely tone, length, and format, from substantive beliefs and prior conclusions and make that distinction visible enough to audit.
Long-horizon evaluation. Model evaluations should test extended, personalized interactions rather than only isolated prompts. A system that performs well in a fresh benchmark may behave differently after months of accumulated context.
Evidence and provenance by design. Research assistants should preserve source links, surface conflicts, and support structured records of material AI contributions. Grounding must remain inspectable even when the interface becomes effortless.
A Research Partner Should Know Us But Not Inherit Us
Personalization is not the enemy. The best AI research partner should know how we like to work without deciding what we ought to conclude; adapting to our working preferences without quietly narrowing the evidence or reinforcing our assumptions. The central question for researchers and scholarly publishers is therefore not whether AI should be personalized. It is how to preserve the benefits of a personalized research partner without allowing that partnership to become an intellectual echo chamber.
Discussion question: What would it take for an AI research assistant to know us well without agreeing with us too well?
Authors’ note: ChatGPT Work, developed by OpenAI ([ChatGPT 5.6 Sol and released on 26th June 2026]), was used, to synthesize an initial manuscript from Avi Staiman’s LinkedIn post, public comments on that post, and Hong Zhou’s written notes; to suggest structure; and to identify candidate sources. The authors reviewed the cited sources, revised the argument, and remain responsible for the final text.
