Article body

introduction

Rare diseases are defined as conditions that can lead to chronic disabilities, including physical, cognitive, and sensory limitations, which may cause persistent or intermittent pain (1). There are approximately 7,000 recognized rare diseases, spanning various categories such as genetic disorders, certain cancers, and infectious diseases, including conditions like cystic fibrosis and Crohn’s disease (2-3). A disease is classified as rare when it affects fewer than one individual per 2,000 people, a threshold that international organizations consider as an indication of low prevalence (2). Collectively, rare diseases affect about 7% of the global population, representing a significant public health challenge worldwide (4).

People with rare diseases often experience a reduced quality of life, as their unique challenges are frequently misunderstood, leading to a series of unsuccessful medical appointments that can further complicate their situation (5). This issue is particularly significant because diagnosing rare diseases can be a lengthy process, with the risk of “diagnostic overshadowing”, where symptoms are mistakenly attributed to more common conditions rather than the rare disease itself (3). For example, studies show that 40% of patients with rare diseases are misdiagnosed at least once, resulting in delays in receiving appropriate care (3-6). The symptoms of a rare disease often overlap with those of more common conditions, which complicates the diagnostic process and increases the likelihood of misdiagnosis (3,7-8). Therefore, a diagnostic tool specifically designed to help identify rare diseases could greatly support healthcare professionals, enabling more accurate diagnoses and reducing the risk of misdiagnosis, ultimately improving the quality of care for patients with these conditions (9).

Artificial intelligence (AI), defined as “the ability of a machine to mimic or surpass intelligent human behavior and activities” (10), offers promising potential for improving symptom screening, which could support the diagnosis of various conditions, including rare diseases. AI tools are already widely used across the health sector, including in areas such as home care, pharmaceutical research, and public health (11). These tools provide a range of services, from daily assistance and medical monitoring to virtual consultations for symptom tracking and mental health consultation (12-13). Some AI applications are available as smartphone apps that engage users in human-like conversations; known as “chatbots”, these are automated dialogue systems that interact with users through natural language, either via text or voice, to answer questions on specific health-related topics (12-14). Chatbots can be programmed to perform various functions, with types and capabilities varying based on their design. The first health-focused chatbot, launched in 2017 by the San Francisco-based startup Woebot Health, was designed to improve mental well-being by helping to reduce symptoms of depression and anxiety (12-15). Since then, additional chatbots such as Babylon, Ada, Your.MD, and K Health have entered the market, primarily as symptom checkers that offer potential medical diagnoses based on symptoms shared by users (14). These chatbots invite users — whether patients or caregivers — to describe their symptoms, which the AI system then analyzes to suggest possible conditions, providing valuable insights that can guide further medical evaluation (12-16).

The ability of AI chatbots to provide differential diagnoses could significantly enhance the diagnosis of rare diseases by minimizing diagnostic overshadowing and expediting the diagnostic process. By integrating symptom analysis and offering tailored preliminary insights, these chatbots can facilitate timely access to appropriate management and treatment, ultimately reducing the social and professional impacts of rare conditions on patients’ daily lives (3). The scientific literature indicates that medical chatbots are a promising tool for addressing the unique challenges faced by individuals with rare diseases, particularly in contexts where specialist care may be limited (3-16). However, the deployment of such AI-driven tools raises ethical concerns that must be carefully considered before their widespread adoption. Issues such as data privacy, informed consent, and the accuracy of AI-generated diagnoses are particularly pertinent in the realm of rare diseases, where misdiagnosis can have profound consequences for patients’ health and well-being (17). Therefore, the objective of this study is to explore the ethical challenges associated with the use of AI chatbots in rare disease diagnosis through a scoping review, providing an overview of the ethical landscape surrounding these technologies.

Methodology

Study design

We conducted a scoping review to examine the ethical issues associated with the use of AI chatbots for diagnosing rare diseases. Following the methodological framework established by Arksey and O’Malley (18), this review is particularly relevant for this topic as it allows for the identification of research gaps and the synthesis of ethical considerations that have been explored to date, providing a comprehensive overview of the current landscape in this emerging area.

Eligibility criteria

For screening literature, we included studies that address chatbots and rare diseases; English or French language articles (regardless of country of origin); studies published between January 1, 2010, and February 10, 2024; peer-reviewed or published work in journals (e.g., reviews, commentaries, empirical research); and grey literature, if available such as national and international policy reports. We excluded clinical studies such as cohort studies, controlled trials, animal and in vitro studies, thesis dissertations and abstracts, and public opinion articles from non-experts.

Information sources and search

Information sources included the following electronic databases: Medline, PubMed, PsycNet, PsycInfo, Scopus, Web of science and Cinhal. We further searched grey literature and other academic papers through individual searching, including Google searches and lists of references directly in selected papers. A librarian from the Université Laval was also consulted to guide research strategies. Our search was based on the following key concepts: “artificial intelligence”, “rare disease”, “chatbot”, “conversational agent”, “ethical”, “legal”, and social”, and used this search strategy (“artificial intelligence”) AND (“conversational agent” OR chatbot OR “dialogue system”) AND (“ethic*” OR social OR legal) AND (“rare disease*” OR “orphan disease*”).

Study selection

As shown in the PRISMA diagram (Figure 1), we used Covidence Software to screen sources by removing duplicates, then by reading titles and abstracts. AB and MOL independently screened titles, abstracts and full text documents for studies meeting eligibility criteria. Both AB and MOL extracted data from the included full text articles and then compared results to reach a consensus. In the event of disagreement, a third judge (HH) would intervene to analyze the study and decide on its exclusion or inclusion according to the criteria adopted.

Figure 1

PRISMA diagram and selection process

PRISMA diagram and selection process

-> See the list of figures

Data mapping and collection

Both reviewers extracted data from the literature and recorded relevant information, including title, year of publication, type of source, type of conversational agent, methodology, results, limitations, elements relating to trust, data security, humanization, commercialization, diagnostic accuracy and validity. The initial sample included 376 studies before screening. Subsequently, 103 duplicates were removed by Covidence Software, narrowing the sample down to 273 articles. Eligible studies were then analyzed based on their titles and abstracts which excluded 228 articles. The remaining 45 articles underwent a full text review resulting in a final sample of 6 articles that were analyzed for the purpose of the current review. Themes from data extraction were categorized based on a codebook developed in an inductive manner by both authors AB and MOL.

The inclusion of only 6 articles reflects the novelty and specificity of the topic, which focuses on the use of AI-based conversational agents/chatbots for rare disease diagnosis, an emerging and highly specialized domain. Research on rare diseases already faces diverse challenges: these conditions affect small patient populations, leading to limited clinical data, fragmented reporting, and fewer large-scale studies (19). When combined with the recent development of conversational AI in healthcare, the intersection of these two fields further amplifies these difficulties. This convergence explains the scarcity of eligible studies while underlining the importance of synthesizing the limited research currently available on the topic.

Table 1

Synthesis of studies included in this scoping review

Synthesis of studies included in this scoping review

*This article does not focus on a specific type chatbot but rather discusses the essential characteristics that medical chatbots must possess to be trusted by both users and developers.

-> See the list of tables

Results

Both reviewers (AB and MOL) independently organized the themes and then discussed them for consolidation, resulting in a final categorization that grouped the results under the following four themes: 1) trust in AI chatbot; 2) (de)humanization of AI; 3) data security and commercialization; and 4) psychometric considerations (see Table 1). In this paper, we use the terms “AI systems” and “chatbots” interchangeably, recognizing them as confounded terms within the context of rare disease diagnosis.

Trust in AI Chatbot

The reviewed literature indicates that a significant concern when using AI in the context of rare disease diagnosis is the level of trust that patients and caregivers have in chatbots that assist with diagnosing these conditions. Laumer et al. (6) distinguish between trust in the AI technology itself and trust in the provider behind the AI, with a pivotal role placed on the latter. For instance, an individual is more likely to use AI if they trust the institution responsible for its development and deployment. Conversely, without trust in the provider, users are unlikely to engage with the chatbot, regardless of its technical capabilities (6). Both caregivers and patients often express reluctance to use AI, primarily due to a lack of trust (20). For patients, trust is based on several factors, including the AI’s technical performance, expected accuracy and reliability of the chatbot, perceived ease or effort required to interact with the technology, and social influence of peers who may either endorse or discourage AI use. Additionally, trust seems to be influenced by the user’s overall satisfaction with the AI and their habitual comfort in using similar AI technologies (3). Building trust in AI can be complex, but it is essential for fostering acceptance, particularly in sensitive areas like healthcare. A critical approach to enhancing trust involves ensuring that the AI is developed by experienced designers who work closely with healthcare professionals (20). Rapp et al. (21) argue that trust can be bolstered by presenting chatbots as systems capable of mimicking certain human-like qualities, which can make interactions feel more natural and reassuring. Similarly, Lambert et al. (3) suggest that AI systems designed with human-like characteristics — such as appearing knowledgeable, maintaining a relaxed demeanor, and using a friendly tone — can significantly enhance users’ feelings of trust. Pereira and Diaz (22) further assert that the trust relationship between users and chatbots is dynamic and can grow over time, provided that the AI consistently meets users’ expectations and maintains a positive rapport.

AI (De)Humanization

The literature reviewed shows that humanization is a critical aspect to consider as it addresses the affective dimension of the interaction between chatbots and users. In this context, humanization involves incorporating human-like traits into AI systems, such as a calm and confident demeanor, which can significantly enhance user trust in AI (3). For a meaningful bond of trust to form between AI and patients, it is essential for the AI to replicate human qualities such as empathy, understanding, and the ability to recognize and respond to the person’s emotional state (22). Conversely, the absence of these human-like qualities can lead to reduced acceptance of chatbots. When AI lacks empathy and emotional sensitivity, users may be less motivated to engage with it and share confidential information, undermining the effectiveness of the diagnostic process (21). Furthermore, humanization extends to the personalization of treatment recommendations. As AI systems interact with patients, they gather insights that allow them to tailor interventions based on individual needs and responses. This personalized approach enhances the relevance and effectiveness of the treatment suggestions provided by the AI (4). By interacting in a more human-like manner, AI can also foster greater patient engagement, encouraging users to adhere to the treatment plans suggested and improving overall treatment outcomes (22).

Data security and commercialization

The use of AI in diagnosing rare diseases raises significant issues related to data security and commercialization. One ethical concern is the protection of personal data. Hallowell et al. (20) argue that many individuals perceive private companies managing their personal data as lacking transparency, which fosters mistrust and unease. Similarly, Rapp et al. (21) observed that people fear that AI systems might misuse their personal data, leading to hesitation in using medical chatbots due to potential security gaps. Conversely, some studies suggest that certain individuals may feel more comfortable confiding in medical chatbots than in human healthcare providers (22). This preference often stems from the belief that AI systems offer stronger safeguards and are less prone to human error or breaches of confidentiality. As a result, these individuals tend to disclose sensitive health details more openly to chatbots, reducing feelings of shame or embarrassment that might arise in face-to-face interactions. In this context, chatbots are perceived as capable of enhancing data security through advanced measures such as two-factor authentication (22), which ensures that only authorized users can access personal data. Such mechanisms reinforce confidentiality and strengthen patients’ sense of security, encouraging them to share private health information without fear of exposure.

In addition to data security, the literature highlights the ethical issue of data commercialization through the use and deployment of AI by private companies, which often play a significant role in introducing these technologies into healthcare (20). For instance, users express widespread apprehension about the potential misuse of personal data for financial gain, a perception that undermines trust in medical chatbots (21). Hurvitz et al. (4) note that AI can be used to rapidly identify patients with costly-to-treat diseases, facilitating quicker interventions and potentially reducing overall system costs by streamlining the diagnostic processes for rare diseases. To address trust deficits and minimize conflicts of interest, Hallowell et al. (20) recommend implementing strict regulations governing the ownership and operation of medical chatbots, including the enforcement of penalties for data misuse to ensure compliance.

Psychometric considerations (validity, reliability, sensitivity and specificity)

When considering the chatbot as an assessment tool, it is essential to reflect on psychometric considerations, which are crucial for developing confidence in the chatbot’s clinical outcome. Like many other assessment tools, the data derived from diagnostic analyses must undergo statistical testing to establish validity, reliability, sensitivity, and specificity. From the patient’s perspective, as previously mentioned, trust in how data will be used is paramount. Similarly, clinicians must also grapple with confidence in the value of chatbot assessments and their clinical outcomes.

Validity is particularly critical when using chatbots for diagnosing rare diseases, as it relates to their ability to deliver accurate diagnoses. Existing studies generally indicate that these tools perform well in this regard. For instance, Laumer et al. (6) note that chatbots provide superior diagnostic results compared to a simple “Google search,” which often fails to yield accurate information (3-6).

Regarding sensitivity and specificity in clinical judgment, some chatbots offer a percentage match between symptoms and diagnoses, presenting users with a range of diagnostic possibilities for a more comprehensive understanding. Pereira and Diaz (22) highlight the potential validity of medical chatbots in diagnosing rare diseases, attributing this to their capacity to learn and adapt through patient interactions. This iterative learning process enables chatbots to continuously refine their diagnostic algorithms based on accumulated experience, thereby improving accuracy over time. Furthermore, certain AI systems can access and analyze extensive personal information that may exceed the data typically available to healthcare professionals. This comprehensive analysis allows chatbots to suggest personalized habits or treatments that could improve patient health outcomes (22).

Despite these potential advantages, Hallowell et al. (20) argue that the scarcity of information about rare diseases complicates the generalization of symptoms, potentially undermining the validity of chatbot diagnoses. Lambert et al. (3) express similar concerns and advocate for continued research through controlled randomized trials. They further note that accurate diagnosis depends not only on the chatbot’s technical capabilities but also on users’ perceptions of its reliability. For chatbots to be recognized as valid and reliable diagnostic tools, it is crucial that they are developed and continuously monitored by qualified medical professionals. Regular evaluations and updates are essential to ensure that the chatbot maintains diagnostic validity and adapts to new medical insights (20). In the section titled “Threats to Validity,” Pereira and Diaz discuss threats to ensuring validity, internal validity, external validity, and conclusion validity (22). These threats encompass concerns related to the comprehensive coverage of research questions, inclusion of all relevant studies, classification categories, and potential biases in data collection.

Discussion

Our scoping review aimed to identify and map ethical issues related to the use of AI in rare disease diagnosis. Key benefits highlighted in the literature include improved diagnostic accuracy of rare diseases through the analysis of large and complex datasets (3), provision of supplementary information that broadens diagnostic possibilities (22), and reduction of diagnostic delays by rapidly identifying potential rare diseases (4). Further, AI systems can offer personalized treatment recommendations based on individual patient data, tailoring interventions to better meet patient needs (21). Beyond chatbots, predictive algorithms and machine learning models have demonstrated similar advantages, such as early detection of conditions like Fabry disease by leveraging electronic health records and genomic data (23). These capabilities underscore AI’s potential to transform rare disease care by accelerating diagnosis and optimizing treatment pathways.

Despite these benefits, several ethical challenges were raised, including trust in AI, AI (de) humanization, data security and commercialization, and psychometric considerations. Notably, trust emerged as a recurring theme across all identified issues.

Trust is a critical factor in successfully integrating AI into medical practice, influencing how healthcare professionals and patients interact with these tools (20). A primary ethical concern is the risk of errors or misdiagnoses, as AI systems — whether chatbots or predictive algorithms — remain fallible and may generate inaccurate results due to algorithmic limitations or biased training data (17). Such inaccuracies can undermine trust in AI technologies, limiting their adoption and positive impact on healthcare outcomes (17). This risk is particularly pressing in the context of rare diseases, where symptom complexity and data scarcity pose significant diagnostic challenges. Trust can be strengthened if AI systems are perceived as transparent and supported by mechanisms for error detection and correction. Regular assessments and updates by medical professionals can ensure that AI tools maintain their accuracy and reliability. This collaborative approach, where AI complements rather than replaces clinical decision-making, can promote confidence and mitigate misdiagnosis risks (6-20).

Data security is a critical concern in the use of AI for rare disease diagnosis, particularly regarding the storage and management of sensitive health information. Martineau and Godin (17) emphasize that current security measures are not infallible, with vulnerabilities that may go unnoticed and expose data to breaches. Risks such as cyberattacks and software flaws further threaten privacy, especially in rare disease contexts where data is highly personal (20-24). To address these concerns, anonymization and robust encryption are critical. Equally important is transparency in data handling: companies deploying AI must clearly communicate how data is collected, processed, and used, adhering to strict ethical standards. Such measures not only protect confidentiality but also enhance user trust and accountability in AI-driven healthcare (7-20).

Psychometric considerations — validity, reliability, sensitivity, and specificity — are central to evaluating AI diagnostic tools. While some advanced medical chatbots have demonstrated diagnostic accuracy approaching 90% in some studies (7), their reliability depends on the quality and representativeness of training datasets. Similarly, predictive algorithms excel in processing large-scale data but face challenges when datasets lack diversity, leading to bias and compromised validity (4-22). These limitations are particularly acute in rare diseases, where data scarcity (20) and fragmentation hinder algorithmic precision. Addressing these gaps requires collaborative data-sharing initiatives and inclusion of underrepresented populations to improve fairness and reduce disparities. Some AI models mitigate misdiagnosis risk by providing diagnostic suggestions with confidence scores, offering clinicians a measure of reliability (3-20).

Finally, the integration of AI into clinical practice raises questions about accountability. When AI-generated recommendations conflict with physician judgment, determining which should prevail remains contentious (17-25). While some argue that increasingly autonomous AI systems could bear moral responsibility for errors, others maintain that physicians should remain the ultimate decision-makers (17-26). A shared responsibility model, where AI and clinicians jointly contribute to diagnostic decisions offers a balanced approach, particularly in rare disease contexts where complexity demands human expertise (25).

The involvement of private companies in deploying AI chatbots in healthcare raises significant ethical concerns, particularly regarding conflicts of interest and data commercialization. When these tools are used to diagnose costly conditions, there is a risk that patients may be directed toward treatments that financially benefit these companies, rather than prioritize patient welfare. For example, while the rapid identification of patients with rare diseases through AI can reduce costs for both society and patients by minimizing the need for multiple appointments, it may also provide opportunities for companies to promote more expensive treatments that do not necessarily improve the quality of care. Lack of transparency in data use further underscores the need for stringent regulations and accountability mechanisms (17). In this context, Hallowell et al. (20) argue for enforcing regulations that prioritize patient interests, particularly when public funding supports technological development. Such measures are essential to safeguard ethical integrity and prevent the prioritization of profit over care quality (12).

Strengths and limitations

Scoping reviews are particularly valuable for mapping emerging fields, offering a structured overview of key themes and identifying gaps for future research. This review contributes to that goal by synthesizing ethical considerations surrounding the use of AI chatbots in rare disease diagnosis, a domain where evidence remains limited. Four central themes emerged: trust in AI chatbots, data security and commercialization, psychometric considerations, and the humanization of AI interactions. A notable strength of this review lies in its focus on chatbots, which represent a distinct category of patient-facing technologies designed to support symptom reporting, triage, and engagement. This approach enables a focused ethical analysis related to conversational systems, such as transparency, and trustworthiness during real-time exchanges (27), concerns that are particularly salient in rare disease contexts where symptom complexity and patient vulnerability heighten the stakes of inaccurate responses (19-28).

The geographic scope of studies retained for analyses, which include Germany, Mexico, Israel, and England, narrows the context in which these findings can be applied. Further, the small number of studies constrains thematic depth, potentially leaving emerging ethical concerns underrepresented as AI chatbot use for rare diseases continues to evolve (29). To mitigate this limitation, we incorporated insights from excluded articles and grey literature (7-12,17-30), which provided valuable context on issues such as data security, regulatory compliance, and governance frameworks, thereby broadening the discussion. Our analysis drew lessons from literature on AI applications beyond chatbots, including predictive algorithms for rare disease detection (20). These systems, which leverage structured data such as electronic health records and genomic profiles, have demonstrated potential for accelerating diagnosis of rare conditions (23-30). While they differ functionally from chatbots, they raise similar ethical challenges including algorithmic bias, diagnostic accuracy, and inequities in access to advanced technologies (30). Integrating these perspectives situates AI chatbot-specific concerns within the broader AI ethics landscape, reinforcing the need for governance strategies that address both conversational agents and data-driven modalities in rare disease context.

Conclusion

The use of AI chatbots for diagnosing rare diseases holds promise by offering the potential to improve diagnostic accuracy and provide timely assistance in an area marked by complexity and data scarcity. However, this promise is accompanied by several ethical and practical challenges. Key concerns include the trustworthiness of AI systems, data security, the potential for data commercialization, and the psychometric aspects of diagnostic accuracy, reliability, and validity. For AI chatbots to function effectively, they require access to high-quality, diverse datasets, which can be challenging in the context of rare diseases due to the inherent scarcity of relevant data. This limitation may impact the chatbot’s ability to generate reliable and accurate health assessments, raising doubts about their overall effectiveness in clinical settings. Given these challenges, the integration of AI along with clinical expertise remains crucial. Even with advanced AI chatbots, healthcare professionals must continue to play a central role in final diagnostic decision-making. Moving forward, research should focus on addressing the limitations related to data scarcity and accuracy, as well as exploring ways to enhance the collaboration between AI tools and healthcare providers to optimize the diagnostic process in the context of rare diseases.