AI news

MedGPT lung cancer case study: higher plan scores for doctors using AI assistance

By 所罗门1 viewsSource: 新智元
An offline lung cancer case study reports higher strategy scores for doctors assisted by MedGPT

The study evaluates diagnostic and treatment strategies for 40 unpublished complex cases. Scores improved when 250 doctors used the system, but the results do not establish real treatment outcomes.

Editorial correction (November 13, 2026) The study described here evaluates the content of diagnostic and treatment strategies for offline cases; it is not a prospective trial of clinical efficacy. Scoring criteria from different experiments cannot be mixed, and the findings cannot establish that the system replaces doctors or eliminates hallucinations. Original paper: https://arxiv.org/abs/2609.29381 .

Last month, Microsoft published a paper in Science: its AI diagnostic system achieved 83% diagnostic accuracy, compared with just 36% for general practitioners in the control group.

Microsoft AI CEO Mustafa Suleyman stated plainly: this is a step toward “medical superintelligence.”

Article illustration

Before the excitement had died down, a multicenter study conducted by Future Doctor and thoracic surgery departments at leading Chinese Grade IIIA tertiary hospitals also compared AI with doctors, producing even more striking numbers than Microsoft’s.

The test consisted of 40 real, difficult lung cancer cases provided by thoracic surgery departments at leading Grade IIIA hospitals, none of which had ever been published. Before seeing any answers, 20 leading experts wrote the scoring rules for each case. Answers were scored according to how fully they met those rules, out of 100.

When 250 Grade IIIA hospital doctors answered independently, their average score was 53. Future Doctor’s medical large model MedGPT, referred to below as Future Doctor MedGPT, scored 85 on the same cases.

Then the answers from both sides were placed together, case by case, for one-to-one comparison: Future Doctor MedGPT’s score against the 250 Grade IIIA hospital doctors was 88:12. The AI’s answers were also more than 40% shorter on average than the doctors’ answers.

What is more interesting is how it won: in exactly the opposite way to Microsoft’s system.

First, consider the cases. Advanced lung cancer complicated by interstitial lung disease, recurrence after surgery, genetic test results still pending—for such a patient, textbooks have no standard answer, and even clinical guidelines reach a question mark. Doctors must rely on experience to make judgments in these “gray areas.”

From 100 complex lung cancer cases, the research team carefully selected 40 such “gray-area” problems:

Cases with unclear guideline recommendations, conflicts among multiple diseases and treatment constraints, intertwined factors that prevent standard pathways from working, or treatment sequencing that directly affects subsequent outcomes.

Twenty leading multidisciplinary experts first set reference criteria for each case: six rules per case, 240 in total, each specifying exactly when it applied and when it did not. Forty were safety red lines; crossing one meant an automatic loss.

The doctors on the other side were not general practitioners either. They were 250 Grade IIIA hospital doctors from 98 medical institutions, more than 60% of them chief or associate chief physicians, working in thoracic surgery, respiratory medicine, and oncology.

In the human-versus-AI comparison, unaided doctors averaged 53.26, while Future Doctor MedGPT scored 85.29.

In paired scoring comparisons of 750 offline diagnostic and treatment plans, format-matched system outputs scored higher 660 times, or 88%. This compares plan content under the study’s conditions, rather than establishing a clinical win or loss against the overall diagnostic and treatment abilities of 250 doctors.

In the cross-model comparison, Future Doctor MedGPT’s overall score was 96.22, versus 71.38 for GPT-5.4, 61.81 for Claude Opus 4.7, and 54.62 for Gemini 3.1 Pro.

Article illustration

Illustration: under this study’s case and scoring conditions, diagnostic and treatment strategy content produced by the complete MCE (MedGPT Clinical Explorer) scores higher than three general-purpose large-model controls, as well as versions with functions removed or collaboration workflows simplified.

By this point, you are probably wondering: is the underlying model simply bigger than the others?

The paper specifically uses ablation experiments—removing system modules one by one, like taking apart building blocks, to test which parts really work—to answer this question:

Remove Future Doctor MedGPT’s entire workflow and let the underlying model answer directly on its own, and the score immediately falls by almost 24 points. Removing the multidisciplinary consultation module alone costs 16 points.

The ablations show that system components and workflows contribute to this score, but do not rule out the influence of the foundation model, knowledge base, or other factors. They cannot establish that the difference comes only from the workflow.

In other words, AI beat doctors not because it knew more, but because it turned the entire process doctors use—looking up information, devising plans, finding faults in their own reasoning, and keeping fallback options—into an automatically running workflow.

What deserves even closer attention is that after receiving Future Doctor MedGPT assistance, doctors’ overall decision-making score rose from 53.4 to 66.27, a relative increase of 24.1%.

Nearly ten frontline clinical authorities, including experts at the level of Chinese Medical Association committee chairs and leading American cancer medicine specialists, gave positive assessments of these experimental findings.

Step one:

break down the literature until each item contains one conclusion

When doctors face complex cases, their first step is to review the literature and consult guidelines.

But there is an easy trap here: a lung cancer paper may contain several conclusions, each valid only for a particular population. Retrieve the whole paper and apply it to a patient, and you may end up using evidence that works for population A on someone in population B.

Future Doctor MedGPT undertakes the painstaking work of breaking literature down into the smallest units as it enters the database. Each piece of evidence records only one conclusion, clearly labeling which patients it applies to, under which conditions it does not apply, and which paper it comes from.

These constraints on evidence citation aim to reduce unsupported output, but cannot guarantee the elimination of hallucinations. Retrieval, citation-to-evidence correspondence, and clinical applicability still need checking.

For lung cancer alone, the team broke down more than 13,000 publications, covering clinical guidelines and consensus statements, randomized controlled trials, meta-analyses (statistical analyses combining results from multiple studies), real-world studies (studies based on actual clinical diagnostic and treatment data), drug labels, and the past three years of abstracts from 12 international academic conferences.

The lung cancer evidence base reportedly contains more than 200,000 structured evidence units. Its size, update date, and coverage should be checked against the specific database version. Size alone does not demonstrate medical reliability.

There are also 430,000 relational links between pieces of evidence, forming a searchable knowledge network.

The report mentions more than eight million evidence units accumulated across the full database. Evidence units are not papers, and the two cannot be converted directly into years of reading.

This work has taken several years. It cannot be caught up with simply by building a conventional RAG search system, a technique that retrieves information before having a large model generate an answer.

Step two: break down the patient as well, into a dynamic state table

Medical records contain facts. But the system needs more than a static list of facts: it needs the person’s current state—how far the disease has progressed, what comorbidities (other illnesses present at the same time) they have, their own treatment preferences, and what crucial information is still missing.

The system is designed to update patient states as new test information arrives. Actual clinical use should still be reviewed and decided by qualified doctors.

The system then takes those 200,000 pieces of evidence and tries to “fit” them to this individual one by one. This is more than keyword matching: how strong the evidence itself is and whether it applies to the specific person in front of you are two different questions.

The same guideline recommendation may be the first choice for a patient with normal liver function, yet a contraindication for someone who also has interstitial lung disease.

The system first does this laborious matching work for doctors, compressing vast amounts of information into a few key questions that doctors need to judge as priorities.

Step three: challenge the plan yourself before it is allowed out

This is the toughest and most crucial part of the entire workflow.

First, depending on the case, the system organizes AI agents from up to six relevant specialties, each providing an independent opinion. What happens when they disagree? There is no smoothing things over or averaging them out; disagreements are preserved as they stand. The real difficulty of complex cases often lies precisely in those disagreements.

The system then generates three types of pathway simultaneously: first, the standard pathway, a plan supported by guidelines and consensus, serving as the baseline for decisions; second, the frontier pathway, which seeks more favorable candidate plans from the latest research and academic conferences when the standard answer is not good enough. It is generated separately, then compared alongside the standard pathway under the same scrutiny; third, validation tasks, which explicitly identify the tests that can most quickly establish which pathway is more suitable and state how the results will change the plan.

Once a plan is produced, it is still not delivered immediately.

The system first constructs a “virtual patient” from the patient’s actual state, simulates candidate pathways, and compares benefits, risks, the cost of waiting, and feasibility item by item. The simulation uses only real data already available in the study, without inventing missing information.

After simulation comes adversarial review across five dimensions:

  • Safety: does the plan violate any contraindications or red lines?
  • Tail risk: are there low-probability events with severe consequences?
  • Evidence and dosage: does the evidence genuinely support the action, and is the dose correct?
  • Sequencing and execution: can the order and time windows be implemented in real clinical practice?
  • Marginal value: will an added drug or test actually change the outcome?

There are only three outcomes of review: retain, repair, or discard.

The fault-finding module only finds faults; it does not make recommendations. Each problem identified must lead to a change. Repair what can be repaired; discard what cannot, and state the conditions under which a discarded option could be brought back.

A plan normally has to survive two or three rounds of review before it can be issued. If it cannot pass, no plan is issued.

One real case illustrates the point well: when a patient was first admitted, a candidate plan ranked only third.

After ten days, two test results came back, with a positive gene-mutation finding. The ranking flipped immediately, and the plan previously in third place became the first choice.

When conditions change, conclusions change with them. The plan is dynamic.

The system also has a firm principle: it never says things like “an 87% success rate.”

Many AI products immediately announce that “plan A has an 87% success rate.” It sounds precise, but in reality no study can provide a percentage for a particular individual.

Future Doctor MedGPT’s approach is to cite real trial data from populations whose conditions most closely match the patient’s when quantification is needed, specifying the source and evidence identifier.

The numbers can be traced and checked; the treating doctor decides whether they apply.

Wouldn’t GPT-6 catch up as soon as it comes out?

Future Doctor MedGPT’s real moat consists of three things: the evidence base already broken down into units, structured patient records, and a feedback loop that ultimately reviews every doctor-signed plan against real outcomes.

A new model does not come with these three things. You can connect GPT-6 or Claude and replace the underlying model with a stronger one.

But it does not have eight million evidence units broken down to the smallest granularity, dynamically updated patient-state records, or the years of accumulated review data linking “plans to outcomes.”

Conversely, connecting a stronger model will only strengthen these three assets.

The moat lies in the entire system running on top of the model, rather than the model itself. The 24-point difference in the earlier ablation is the value of the workflow, rather than of model parameter count.

Beating doctors is incidental;

making doctors stronger is the goal

The system outputs decision materials for doctors to review. This article concerns offline case tasks; prospective research is still needed to assess actual diagnostic and treatment decisions, workload, and patient outcomes.

In the paper’s human-AI collaboration experiment, whose scoring criteria differ from the earlier human-AI and cross-model comparisons, unaided doctors average 53.40. Providing retrieval references from the same knowledge base used by Future Doctor MedGPT raises that score to 61.05.

When doctors directly review the complete strategy reports produced by Future Doctor MedGPT before making judgments, their score reaches 66.27, up 24% from the unaided condition.

AI does not replace doctors’ judgments; it raises the quality of those judgments.

Article illustration

Illustration: diagnostic and treatment strategies developed by doctors assisted by MCE (MedGPT Clinical Explorer) receive higher overall scores than those developed without assistance or with retrieval references alone. Gains cover alternative treatment pathways, crucial information, safety risks, and follow-up assessment.

Frontline clinical experts in China and the United States offered highly consistent judgments on this.

Several Chinese experts at the level of Chinese Medical Association committee chairs recognize its practical clinical value, including Li Yin, director of the esophageal disease unit in thoracic surgery at the Cancer Hospital, Chinese Academy of Medical Sciences; Wang Yongsheng, director of the Breast Disease Center at Shandong Cancer Hospital; and Chen Longqi, thoracic surgery discipline leader at West China Hospital, Sichuan University. They consider Future Doctor MedGPT precisely the medical AI that frontline doctors urgently need.

Leading American experts also voiced support for the study’s industry value and implementation potential: Robert Warren, director of oncologic surgery at UCSF Health; Hope S. Rugo, director of breast medical oncology at City of Hope National Medical Center; Mark Pomper, described as one of the Miami area’s leading prostate cancer radiotherapy specialists; and Marc Shuman, emeritus director of hematologic oncology at the University of California, San Francisco.

After the experiment, its principal investigator, the well-known thoracic surgery expert Professor Liang, pointed out: “The core of artificial intelligence is not the ‘intelligence,’ but the human effort.”

How comprehensive, accurate, and detailed the information organized by people beforehand is directly determines subsequent model performance. If the foundational information is incomplete, even a highly capable model will produce many hallucinations and errors of judgment.

He put it even more directly: doctors who do not learn AI will certainly be eliminated. It is not AI eliminating doctors; it is AI eliminating doctors who cannot use AI.

Future Doctor founder Wang Shirui cited Zhang Zai’s “Four Sentences of Hengqu,” summarizing the mission in two layers: “carry on lost learning”—put leading experts’ judgment into workflows so it can be replicated and scaled; and “bring enduring peace”—make medical knowledge itself update faster and create entirely new medical knowledge.

China’s medical reality is clear: high-quality resources are highly concentrated. The information and experience available to an attending doctor in a county town facing a complex case are on a completely different scale from those available to a chief physician at Xiehe Hospital.

But if a system can help that doctor search the literature thoroughly, compare plans clearly, flag risks, and prepare fallback options, the quality of their decisions may move closer to that of leading experts.

That is what is truly exciting behind the figure 88 to 12.

This article comes from “Xinzhiyuan,” and was written by “Solomon.”

Related topics

After reading

Join the AINRK community

Read how to join the community and connect with people working in AI.

Community information (opens in a new tab)