Artificial intelligence (AI) is opening new frontiers in medicine, but a consultancy is warning that its use must be carefully managed to avoid patient risk.

Researchers at the Massachusetts Institute of Technology (MIT) have used AI to identify two compounds that could help tackle highly drug-resistant infections such as gonorrhoea and MRSA, with further work targeting Parkinson’s disease. The breakthroughs highlight AI’s potential to accelerate treatment discovery in previously intractable conditions.

However, a recent University of Oxford study found that large language models (LLMs) could provide “inaccurate and inconsistent information” when used by patients seeking medical advice. The research also identified communication gaps and difficulty reflecting the complexity of real-world patient interactions.

Justin Megawarne, Managing Partner at technology consultancy Megaslice, said the contrast was unsurprising. “The key to successful implementation is not solely the quality of the AI product, but also the infrastructure around them… Good AI applied in the wrong way can be a danger to humans.”

He stressed the need for human oversight: “The primary importance in a medical setting is safety, and on its own AI is nowhere near the safety levels required to operate without a clinician vetting its output. For now, the best way to think about AI models is that they are like very, very smart junior people.”

Megawarne added that AI should support, not replace, clinicians, and warned unresolved liability issues could further complicate adoption.

Health New Zealand recently implemented AI scribe tools in its hospitals to automatically transcribe patient consultations.