Evidence-Based Analysis of AI Chatbots in Oncology Patient Education: Implications for Trust, Perceived Realness, and Misinformation Management

Connecting Human-Seeming Chat Partners and the “Dead Internet Theory”

The “Dead Internet Theory” suggests that a meaningful fraction of what appears to be organic online engagement (i.e., genuine interactions between real human users on digital platforms) is actually automated, orchestrated by advanced bots [21]. While much of this notion remains speculative, it resonates with concerns that the internet’s once-vibrant human exchange could be drowned out, or even replaced, by mechanized systems. If indeed large swaths of social media users, commentators, and “supporters” in cancer forums are bots, the concept of a robust, empathetic online community becomes precarious. The presence of AI-driven cancer chatbots that convincingly pass as genuine survivors or dedicated volunteers effectively reinforces this skepticism [22].

Should it turn out that many helpful “voices” in online oncology groups are non-human, the psychological consequences for genuine participants may be profound. Patients may feel alienated if they realize that the words of comfort or personal experiences come from a language model, rather than someone who truly walked that journey. Others might lose confidence in digital channels for health advice altogether, creating a ripple effect in communities where reputable telemedicine or genuine patient-led advocacy has been a lifeline [23]. In this sense, the interplay between the “dead internet” hypothesis and human-seeming chat partners raises broader questions about how best to preserve authenticity and ensure that technology enhances, rather than compromises, the human dimension of cancer care.

Another significant concern is the risk of training recursion, which occurs when LLMs increasingly rely on data generated by other LLMs rather than human-produced content. This feedback loop has been shown to degrade the performance and accuracy of LLMs over time, as the quality and diversity of the training data diminish [24].

Current Evidence of Chatbot Integration in Oncology Education

Despite these concerns, AI chatbots can confer certain benefits, particularly in educational contexts [25–28]. When powered by meticulously curated data sets and regularly monitored by expert clinicians, chatbots are capable of answering straightforward questions about cancer biology, standard therapies, and the logistics of appointments or follow-up visits. Studies exploring their utility suggest that patients value the immediacy of responses, the ability to pose questions multiple times without feeling guilty, and the non-judgmental tone. In these investigations, participants reported improved recall of basic information about side effects and were more likely to adhere to recommended screening intervals [29–33].

However, the distinction between a beneficial chatbot and one that might cause confusion or harm is not always evident. Many systems lack robust transparency mechanisms to declare, prominently and repeatedly, that they are computer programs rather than living counselors or clinicians. Often, disclaimers exist but are buried in terms-of-service documents or at the bottom of a chat window, insufficient to prevent users from concluding that they are conversing with a knowledgeable individual. Early-phase evaluations of “human-approximate” chatbots in oncology have highlighted this divergence between user perception and the system’s true nature [34, 35]. Some chatbots have included well-intentioned emotional content in their answers, for example, responding “I’m sorry to hear that” or “I know how frightening it can be.” While such empathetic phrases can be comforting, they can also give patients a misleading sense of personal understanding and clinical acumen on the part of the chatbot.

Chatbot guardrails are widely utilized to constrain the scope of chatbot responses, ensuring they remain within predefined boundaries. These mechanisms have proven effective in restricting outputs on sensitive topics, such as political opinions or other controversial subjects, in commercial chatbot applications [36]. Applying similar guardrails to oncology chatbots could help ensure that their responses are aligned with evidence-based guidelines and remain focused on providing accurate, relevant healthcare information.

Researchers evaluating pilot chatbot deployments in cancer education have found a surprisingly high level of user engagement, suggesting that the line between curiosity-driven usage and genuine reliance on AI for decisions may be quite thin [33]. In these scenarios, the underlying algorithms were generally not advanced enough to mislead participants about being human. Yet as LLMs grow more sophisticated, the risk of deception rises, particularly if the chatbot adopts a persona—complete with a name, a fictional biography, and a semblance of shared experiences. This phenomenon disrupts established heuristics about trusting digital sources: historically, users who encountered simplistic automated systems recognized them as mere scripts, whereas present-day technologies are advanced enough to trigger “presence illusions” [37].

Ethical and Practical Complications

At the heart of the issue lies an ethical paradox: the more closely an AI system mimics a caring, experienced oncology professional or a sympathetic survivor, the more it can manipulate user emotion, wittingly or unwittingly. This deception might arise even if the chatbot’s developer has no malicious intent. The authenticity crisis intensifies in public forums, where these bots could unilaterally reply to posts with questionable or dangerously incomplete information. Under the “dead internet” perspective, a portion of those replies might even come from other AI agents reinforcing or echoing the initial statement, an echo chamber of automated participants, all “talking” to each other and presenting a unified but potentially erroneous message [38].

Among the key hurdles is the challenge of accuracy. Though chatbots can be pre-trained on reputable cancer information, new treatment guidelines or newly published clinical trial data appear at a swift pace, meaning the system must be updated. If periodic re-training processes fail to incorporate the latest oncology standards, or if the chatbots inadvertently integrate unverified content, misinformation will gradually accumulate [39, 40]. The impetus to operate chatbots in real-time on social media platforms further heightens this risk. Continuous, large-scale user interactions might shift the chatbot’s language model weights (in certain adaptive paradigms), making it prone to drifting away from evidence-based knowledge [41]. For example, an oncology chatbot initially programmed with accurate data on chemotherapy protocols may become outdated if it fails to integrate emerging evidence on novel immunotherapies or targeted treatments, potentially leading patients to make uninformed decisions based on obsolete recommendations. Similarly, if a chatbot repeatedly encounters and internalizes user-generated content that misrepresents clinical trial eligibility criteria, such as falsely suggesting that all stage IV cancer patients qualify for experimental therapies, it could unintentionally propagate these inaccuracies, undermining both patient decision-making and clinician guidance.

Beyond accuracy, privacy concerns loom large. Patients discussing personal health matters with a chatbot risk exposing detailed clinical histories, psychological states, or preferences about therapy. The chatbot, if improperly configured, might store or inadvertently reveal these disclosures to third parties, undermining patient confidentiality [42]. More subtly, a generative model that was originally trained on real-world patient records might replicate fragments of those records if prompted in certain ways. This overlap between the chatbot’s structured “memory” and user interactions raises troubling scenarios in which personal information emerges out of context, or in which anonymized data become re-identified through repeated conversation [43].

The CARE-AI Approach and Emerging Responses

Addressing these concerns calls for initiatives like the proposed CARE-AI (Collaborative Assessment for Responsible and Ethical AI Implementation) framework, which seeks to align AI technologies with rigorous ethical standards and practical safeguards [44]. CARE-AI emphasizes risk assessment for misinformation, data privacy, fairness across diverse patient populations, and transparent declarations of an AI system’s non-human nature. Such a model is adaptable to oncology education, where ensuring the veracity and safety of content is paramount. For instance, a clinical center could mandate that any chatbot integrated into patient education portals undergo third-party evaluations, focusing on the chatbot’s risk of generating harmful advice and its provisions for disclaimers about its automated nature.

A structured approach might begin with setting explicit boundaries (i.e., guardrails) regarding the chatbot’s scope: clarifying that it provides general educational pointers but not individual treatment prescriptions, and that no “emotional support” from the chatbot can replace real counseling from mental health professionals or oncology social workers [45]. As illustrated in Fig. 1, this involves balancing opportunities such as patient empowerment and accessibility with challenges like misinformation and ethical concerns, while integrating safeguards such as evidence-based design and transparency measures. After formal risk assessment, institutions could roll out pilot versions of the chatbot in controlled environments, collecting logs of user interactions and analyzing any recurring errors. The logs, stripped of identifiable data, would then be reviewed by certified oncologists and data ethics personnel, who would flag inaccuracies or manipulative language patterns. Only once the system meets stringent performance metrics, and has robust disclaimers integrated throughout its interface, would it be deployed more broadly [46].

Recent advancements have focused on developing robust safety guardrails for LLMs [47]. For instance, Meta’s Llama Guard system evaluates the user input, the LLM-generated output, and a predefined set of safety conditions to determine whether a response is “safe” or “unsafe.” However, a significant challenge lies in defining a comprehensive and effective list of safety conditions or examples. Moreover, research has demonstrated that these guardrails can sometimes be circumvented through sophisticated prompting techniques, highlighting the need for continuous refinement and improvement in safety mechanisms [48].

Implications for Practice and the Role of Educators

While industry-led chatbot development marches forward, educators, professional societies, and policy-makers have critical roles to play in preventing an erosion of trust. From a pedagogical standpoint, one promising strategy involves equipping oncology trainees with skills to evaluate AI-based communication [49]. Such training would teach future clinicians how to identify chatbot inaccuracies, counsel patients about safe use of AI resources, and remain vigilant for manipulative or misleading chat partners. Another strategy might be to involve patient advocacy groups in chatbot oversight committees, ensuring that the voices of survivors and caregivers inform the design and maintenance of these systems.

Public outreach efforts could also educate patients on how to distinguish reliable digital tools from questionable ones. For example, public-awareness campaigns might highlight simple methods for verifying the chatbot’s affiliation (such as checking hospital or academic credentials), looking for standard disclaimers, and cross-referencing any critical recommendations with official cancer guidelines. This proactive approach might preserve the benefits of advanced AI, particularly the ability to disseminate health advice instantaneously, while curbing the illusions fostered by chatbots seeking to emulate supportive individuals.

Envisioning the Future: A Cautious Yet Constructive Path

It would be a mistake to assume that all AI chatbots in oncology education are inherently harmful or that the “dead internet” scenario is uniformly bleak. On the contrary, well-designed AI systems could enhance knowledge dissemination, direct users toward credible sources, and alleviate some of the burden on oncology practitioners [50]. Yet the fundamental issues of misrepresentation and misguided trust, along with the broader backdrop of potential widespread bot proliferation, must be addressed head-on. Healthcare institutions, patient communities, and technology firms alike have a shared responsibility to enforce guardrails that minimize harm while preserving opportunities for constructive innovation.

A practical framework for addressing these challenges lies in the integration of academic rigor, transparent verification, and regular data updates. Technology providers could improve trust and reliability by making parts of their algorithmic architecture available for independent auditing and ensuring that training datasets are aligned with evidence-based cancer protocols. Such measures may help narrow the gap between the “human-like” responses generated by AI and the clinically grounded reality required for patient care. However, the prospect that a significant proportion of online cancer-related discussions may be artificially generated highlights the need for ongoing vigilance. This is particularly critical in psychosocial support, where the perceived authenticity of interactions plays a vital role in fostering trust, instilling hope, and providing meaningful emotional support.

Achieving a balanced way forward will rely on collaboration between all key parties. Oncologists and educators should critically review chatbot content to ensure it aligns with the latest research and professional guidelines. Regulatory bodies could introduce clear labeling requirements, ensuring AI tools transparently indicate their non-human status and provide clarity on how their reliability is judged. Tech platforms hosting these tools must adopt robust systems to swiftly address and correct any false or harmful information. In this way, AI’s role in oncology education can evolve into providing transparent, reliable, and ethically designed support, enhancing both learning and patient care without overstepping its limitations.

Ethical Approval

Formal ethical approval was not required according to applicable legislation and institutional guidance.

Conflict of Interest

Vagelis Hristidis is the founder of SmartBot360, a company specializing in the development of chatbots for healthcare and other industries. SmartBot360 is neither reviewed nor referenced in this article. Aaron Lawson McLean declares no conflicts of interest related to the research, authorship, or publication of this article.