Information Cocoon in the Era of Generative AI

Introduction

I feel more strongly than ever that I am living inside a massive information cocoon. The sense of separation between reality and the virtual world has become increasingly clear and tangible in both my life and my work.

I say this with good reason. Before AI demonstrated its explosive emergent capabilities, most intelligent components on the internet were designed primarily to optimize the modeling of human behavior. Broadly speaking, this includes the algorithms and architectures used in recommendation systems, advertising, and search engines. The defining feature of these systems is that they analyze historical human interaction data to extract patterns, then use those patterns to make predictions and influence subsequent interactions. The concept of the “information cocoon” emerged in this context of algorithmic pervasiveness. The American scholar Cass Sunstein first introduced the term in his book Republic.com. He argued that personalized services online guide people toward information that is more likely to please and satisfy them, gradually forming closed and homogeneous groups that resist opposing views.

Sunstein published this argument in the early 2000s. At that time, AI technology and adoption were far from large-scale. Over the past two decades, humanity has developed transformative information technologies, and recommendation, advertising, and search algorithms have permeated every corner of daily life. Now, about twenty-five years later, looking back at Sunstein’s argument, I find that the “information cocoon” effect brought by generative AI far exceeds anything we have experienced before. The generative AI version of the information cocoon is quieter, more seemingly reasonable, and far more profound.

Unconscious influence

Believe it or not, over the past year, much of my fragmented knowledge has come from Xiaohongshu (Little Red Book). For those unfamiliar with it, Xiaohongshu is something of a miracle within the Chinese internet ecosystem. As it has become increasingly difficult to find a search engine in Chinese comparable to Google, Xiaohongshu has taken on the role of providing timely information to internet users. In its early growth years, it offered high-quality content that aligned well with fast-consumption internet culture. As its scale expanded, more and more participants from different sectors joined content creation. I followed many leading voices in technology and AI within the Chinese-speaking community. Their posts allowed me to access frontline information anytime. For those who remember RSS in the early 2000s, you’ll understand: I was effectively using Xiaohongshu as RSS - it’s just different in a way my feed is the group of topics that I have interest in.

However, things began to change after generative AI became widespread. Starting in the second half of 2025, I gradually noticed that much of the content I should have been interested in displayed a kind of “uniform boredom.” The posts that talk about technology are full of AI generated marks. Very similar to watching a AI produced video that plots something against physics law, the posts written in AI do not have anything concretely insightful but full of meaningless repetition of the same thing. This phenomenon intensified after the rise of GEO (generative engine optimization). Because of the potential commercial opportunities GEO offers, more and more people began using “knowledge injection” techniques to bias large models toward mentioning their products more frequently when answering questions.

This is actually very different from the traditional information system like recommender: in generative AI systems, information recipients have no participation in the biased generation process. They do not know what is inside the black box of the large model. All they see is the output in the chat window. In recommendation scenarios, users can express explicit feedback such as “like,” “dislike,” “click,” or “purchase.” In large-model applications, however, users are subject to silent influence. This influence may evolve as models are updated and fine-tuned, but the user does not participate in that process. Even though many applications include “thumbs up” or “thumbs down” buttons, such feedback has almost no meaningful impact on the model’s overall behavior. From beginning to end, humans remain passive. The feedback loop is not personalized; it depends on silent updates at the model and application layers.

Apparent reasonableness

Unlike the real world, everything in AI-constructed virtual space appears reasonable. People marvel at AI solving world-class math problems, generating Hollywood-style short films, or producing complete software applications in minutes. This near-perfect sense of reasonableness is one of the most addictive aspects of generative AI.

But when everything appears reasonable, that itself becomes unreasonable.

Human society is filled with irrationality. Not every question deserves a rational answer. Sometimes the question itself contains logical errors. Extending those errors with a polished answer is itself irrational. Yet in the AI-shaped world, this rarely happens—we can always expect an answer, and often a satisfying one. Whether that answer truly creates value or erodes our own judgment, reasoning, and intellectual capacity becomes secondary to the pleasure of receiving a “reasonable” response.

In a recent test, most large language models exhibited a paradoxical behavior. The test was simple: someone asked, “I’m going to wash my car. Should I walk there or drive there?” Astonishingly, the model provided a “reasonable” but logically flawed answer¹:

“Walk!”

Distance? ≤ 300 meters → Walk 300 meters – 1 kilometer → Depends on the weather 1 kilometer → Drive

“Conclusion: walk. Consider it five minutes of light exercise—healthier than starting a cold engine.”

This highlights a common criticism among AI scientists and practitioners today: large models may not always possess true reasoning or deductive capabilities. They encode vast amounts of seen, read, and heard information into dense representational space, then retrieve the most statistically similar pattern when prompted. They resemble a student who memorized all the answers for passing the test but does not truly understand the underlying logic.

Profound long-term impact

As I mentioned at the beginning, I am a heavy AI user. I interact daily with tools such as OpenAI’s ChatGPT and GitHub Copilot for learning, work, and advice. Since the release of ChatGPT, I have been immersed in large models for over three years. Frankly, I do not know whether my knowledge base has genuinely expanded because of AI or whether I am simply spinning in place.

When I try to recall certain knowledge points that may have gradually embedded themselves in my brain, I sometimes cannot distinguish whether they came from AI or from reliable sources such as books, lectures, or human experts.

For example, since becoming a parent, I have spent considerable time learning about pediatric illnesses. Most of this knowledge came from doctors during consultations. I trust the doctors in Singapore, especially pediatricians at KK Women’s and Children’s Hospital. After multiple visits for similar conditions, I gradually developed my own judgment about common symptoms and how to handle them.

Of course, I am not a professional doctor. When encountering symptoms that seem familiar yet uncertain, I seek a second opinion. This is where AI influences me. I have often asked ChatGPT about symptoms I could not confidently assess. Because I cannot always return to a hospital to verify every answer, AI has gradually inserted many pieces of knowledge into my mental framework that are not clearly labeled as “trusted.”

Later, when dealing with my child’s illness, this created a subtle illusion. That is, I could not clearly distinguish whether certain knowledge came from a doctor or from ChatGPT. When uncertain, I choose the cautious route. I visit a clinic again, confirm with a doctor, and mentally stamp that information as “verified.”

However, many aspects of life and work cannot be strictly verified. With AI available, human inertia tends to favor easily accessible information. The profound consequence is that what appears to be a constantly reinforced knowledge system may actually be filled with AI-generated inaccuracies or distortions. More troubling still, once this system forms, we may no longer know when to halt decisions based on such unverifiable knowledge.

Wrap-up

After finishing this blog post, I opened Xiaohongshu. At the very top of my feed, unsurprisingly, was yet another discussion about how “vibe coding” is reshaping the future of humanity. I glanced back at the technical problems my colleagues and I have been struggling with over the past two weeks. Not a single one of those problems could be easily solved by vibe coding.

References

1. Cass R. Sunstein (2001). Republic.com. Princeton University Press. https://press.princeton.edu/books/paperback/9780691070254/republiccom
2. Aggarwal, Pranjal et al. (2023). GEO: Generative Engine Optimization. https://arxiv.org/abs/2311.09735
3. Jin, Mingyu et al. (2024). Large Language Models Are Better Reasoners with Self-Verification. https://www.ijcai.org/proceedings/2024/0444.pdf
4. Fleming, Scott M. et al. (2024). Large language models reflect human-like patterns of metacognition. Nature Communications 15, 10814. https://www.nature.com/articles/s41467-024-54837-0
5. World Health Organization (2024). Landscape analysis of large multimodal models in health. https://www.who.int/publications/i/item/9789240097749
6. OpenAI (2023). GPT-4 Technical Report. https://openai.com/index/gpt-4-research

Citation

Plain citation as

Zhang, Le. Information Cocoon in the Era of Generative AI. Thinkloud. https://yueguoguo.github.io/2026/02/14/keep-intelligence/, 2026

or Bibliography-like citation

@article{yueguoguo2026informationcocoon,
   title   = "Information Cocoon in the Era of Generative AI",
   author  = "Zhang, Le",
   journal = "yueguoguo.github.io",
   year    = "2026",
   month   = "Feb",
   url     = "https://yueguoguo.github.io/2026/02/14/keep-intelligence/"
}