Rijeka, December 2024 – At the Second International Conference on Teaching and Learning in Medical Education (MEDRI 2025 – FUTUREMED), organised by the Faculty of Medicine of the University of Rijeka, Damir Medved, Director of EDIH ADRIA, presented a comprehensive analysis of the technological, ethical and legal implications of the use of AI conversion agents in healthcare.
Revolution with Challenges
The lecture started with a fundamental question that defines the whole problem of the implementation of artificial intelligence in medicine: “Where is the boundary between the general information that an AI agent can adequately provide and domains that require professional expertise?” As an illustration of the importance of this issue, Medved pointed out that the wrong car configuration results in financial loss, while the wrong medical advice can have serious health consequences.
The experience of the EDIH ADRIA project, which is ranked within the top 5% between 225 European Digital Innovation Hubs per output, shows that 92.3% the services provided have an AI component in the services provided. These data highlight the incredible opportunity AI conversion agents provide to transform patient engagement, administrative efficiency and clinical support, but at the same time they also reveal significant challenges that require careful consideration.
The regulatory landscape being shaped
One of the central issues of the presented research relates to the regulatory classification of AI systems in health care. Under the new EU AI Act (Regulation 2024/1689), patient triage systems are explicitly classified as high-risk under Annex III, Section 5(d), regardless of any “administrative” framing. This classification entails a number of compliance requirements including risk management systems, data management documentation, automatic recording for audits, transparency mechanisms and human oversight provisions.
Medved has elaborated in detail the different levels of classification of AI functions. While pure appointments are classified as limited risk and are not medical devices, symptom collection is potentially high-risk and can be classified as medical software. Urgency assessment and recognition of emergencies fall unequivocally into the high risk category and require classification as medical devices of class IIa+ and IIb respectively under the Medical Devices Regulation (MDR 2017/745).
The key compliance deadlines that institutions must keep in mind are February 2025 for AI literacy training, August 2026 for Annex III systems and August 2027 for medical devices. These deadlines are not recommendations but binding requirements that require immediate action.
Responsibility in the Distributed System
A particularly interesting segment of the lecture was related to the complex distribution of responsibilities between different actors in the AI system implementation chain. Under the new Product Liability Directive 2024/2853, AI systems are now explicitly defined as “products”, assuming that non-harmonised systems are defective. The Directive also covers damages caused by the continuous learning of the system, which is particularly relevant for modern AI models.
Developers and providers of AI systems are subject to strict liability under this directive, while health institutions as users of the system remain liable under national tort laws. What is particularly important to emphasize is that healthcare professionals retain professional responsibility and cannot delegate clinical evaluation to AI systems. As Medved pointed out, human-in-the-loop monitoring does not eliminate the responsibility of service providers.
In addition, the application of GDPR Article 22 means that AI triage represents automated decision-making, while patients retain the right to human intervention. It is interesting to note that the originally proposed AI Liability Directive was withdrawn in October 2025, meaning that fault-based claims are now implemented through 27 different national regimes.
Security paradox: when accuracy is not sufficient
One of the most striking parts of the lecture related to clinical evidence on the performance of the AI symptom checker system. Analysis of 48 different systems showed that their accuracy in the primary diagnosis varies between only 19% and 38%, while the accuracy in triage reaches between 49% and 90%. Compared to general practitioners, the best AI systems achieve an accuracy of 70.5%The average number of doctors is 82.1% accuracy.
Of particular concern is the occurrence of sub-triage, where AI systems do not recognize serious conditions in one in seven to ten cases. Medved documented a number of actual failures of the AI system, including a Stanford 2025 survey that showed that 20% The response of the AI system contains unsafe recommendations, such as proposing the height of the bridge to people with depressed thoughts. ChatGPT has documentedly advised a person with a urinary tract infection to “drink urine”, leading to self-poisoning. Babylon Health showed gender bias and interpreted heart attack as a panic attack, while a study published in Annals of Internal Medicine found that as many as 88% answers can be false when chatbots are manipulated.
Ethical frameworks and medical guidelines
In 2021, the World Health Organization (WHO) published six fundamental principles for the implementation of AI in health, which include protecting autonomy, promoting safety, ensuring transparency, fostering accountability, ensuring inclusivity and promoting sustainability. The 2024 update added more than 40 recommendations specific to large language models, with specific warnings about the “bias of automation”.
In its 2025 declaration, the World Medical Association uses the term ‘complementary intelligence’ instead of ‘artificial intelligence’, stressing that AI supports, but does not replace, human judgment. The doctor must retain ultimate authority over all AI results. Similarly, in its statement of November 2024, the Standing Committee of European Doctors (CPME) clearly states that “AI has not been implemented to replace healthcare professionals or reduce patient autonomy”, and that doctors remain free to decide on the use of AI.
Applying classical bioethical principles to the AI triage, Medved explained that the principle of autonomy requires that patients be informed about the AI system and have the right to an assessment based solely on human judgment. The principle of benevolence requires rigorous validation of efficiency and safety before implementation. The principle of non-harm is particularly critical for identifying emergencies where the failure of a system can lead to immediate harm. Finally, the principle of justice requires validation through different populations and addressing the “digital divide”.
Patient trust as a fundamental prerequisite
The research presented in the lecture revealed a fascinating paradox in patients' attitudes towards AI in healthcare. As many as 60% patients feel discomfort with the use of AI in healthcare, while 66% considers it “very important” to know whether AI is involved in their care. Perhaps the most worrying fact is that 71.5% patients feel uncomfortable with an AI system that can not explain its logic, even if this system achieves 90% accuracy.
Demographic analysis shows that younger generations, especially between the ages of 18 and 34, show greater trust in AI (about 80%), while older generations over 55 show less than 60% acceptance. Interestingly, higher AI literacy does not increase trust, but actually reduces it, indicating that informed scepticism leads to cautious attitudes.
The key solution highlighted by Medved concerns the role of doctors as “facilitators of trust”. The confirmation and recommendation of a health professional significantly increases the acceptance of the AI system by patients. Visible human control acts as a mechanism that facilitates the appropriate adoption of technology.
A model of gradual autonomy as a practical solution
In response to the complex challenges posed by AI conversion agents, the lecture was presented by the innovative “Graduated Autonomy Framework”, which aligns the autonomy of the AI system with the level of clinical risk. This model recognises that the boundary between different levels of autonomy determines the regulatory classification and distribution of responsibilities.
For the lowest risk functions, such as providing information and answering frequently asked questions, the human-on-the-loop model is used, where human oversight is available but not mandatory for each transaction. The same model applies to basic appointments. For medium risk, such as a symptom questionnaire that collects data, ‘human-in-the-loop’ is used, where human review becomes an integral part of the process.
In high-risk functions such as emergency assessments requiring prioritisation, explicit human approvals are required before any action is taken. Finally, for critical functions such as emergency recognition, a human-in-command model is used, where the AI system can only alert staff who retain full control over decisions.
Practical demonstration: AI Assistant for Polyclinic
To illustrate the application of theoretical principles, Medved presented a demonstration of an AI assistant designed for a polyclinic. This system is designed according to clear principles of transparency, limitation, conservatism and auditing.
The assistant can understand patients' problems using natural language to identify needs, schedule appropriate services by matching needs with available appointments, provide general information on working hours, locations and preparation for examinations, and identify emergency signals and redirect communication while alerting staff.
What the assistant does not explicitly do is as important as what he does. The system never proposes medical diagnoses or suggests specific conditions. It does not make recommendations on treatment, does not propose medications or therapies. It does not conduct clinical prioritisation or rank the urgency of cases. Most importantly, it does not take autonomous action in emergencies, but only warns the staff.
This design allows the system to remain in the category of limited administrative risk by avoiding the interpretation of symptoms, while for the recognition of emergencies it applies human-in-command monitoring, which ensures regulatory compliance and clinical safety.
The Way Forward: Recommendations for implementation
Medved concluded the lecture with three key recommendations for health institutions considering the implementation of AI conversion agents. The first recommendation concerns the correct classification of functionality under the EU AI Act, whereby patient triage must be treated as a high-risk application and start the compliance process immediately.
The second recommendation emphasizes the implementation of robust human surveillance for all clinical functions, using a human-in-the-loop model where fully autonomous action must never occur. The third recommendation requires defining clear functional boundaries where scheduling is acceptable, but the assessment of urgency and emergencies always require immediate human review.
Key timelines for institutions to follow are February 2025 for AI literacy training, August 2026 for Annex III systems and August 2027 for medical devices. As Medved pointed out, responsible implementation is achievable through the application of a model of gradual autonomy and robust human oversight.
The lecture ended with a strong message: “AI conversion agents offer real value, but responsible implementation requires that we treat them as tools that complement – never replace – clinical practice.” This message resonates with EDIH ADRIA’s mission to support a digital transformation that is both innovative and responsible, sets European standards in the implementation of AI technologies and contributes to the creation of a health system that combines technological advancement with human care.
English 
Croatian 
Italian