Scientific review articles are comprehensive summaries of the current state of research on a specific topic. They play a crucial role in the scientific community by providing an overview of existing studies, observations, and identifying gaps in knowledge. They are kind of the Reader’s Digest of academia. Although one benefit of these articles from the perspective of the reader is saving time (i.e. someone else has taken the time to survey the literature), there is a major time investment for the one(s) writing the review. The process of writing a scientific review article involves a comprehensive literature search to find relevant studies, critical evaluation of those studies, and synthesis of information to present a coherent narrative about the state of the art in that field. But couldn’t AI do some of that? That was the question asked recently by Kacena et al. , and here is what they found when they used ChatGPT to create drafts of review articles:
- Factual inaccuracies – Up to 70% of cited references generated by the AI-only approach were found to be incorrect or non-existent. Without human fact checking, AI writing risks misinforming readers.
- Risk of plagiarism – The AI-assisted first drafts showed higher plagiarism scores. The authors warn about potential copyright issues with AI.
- Inability to make connections – Humans still play an important role in synthesizing concepts and making meaningful connections between ideas, which AI currently struggles with.
- Information cut-off dates – The ChatGPT version used had a cut-off of Sept 2021. All newer data had to be manually provided, limiting its ability to review contemporary findings.
- Potential bias – The article mentions AI can produce biased output. Relying solely on AI carries risks of perpetuating biases.
- Lack of judgment on source quality – The authors note AI currently can’t judge the importance or reliability of different references on its own.
Good news: looks like we humans are still needed!
That is not to say that the AI approach did not have some potential advantages. The authors also found that AI significantly speeds up the process of composing literature reviews and initial drafts, with the AI-only method emerging as the quickest for generating drafts. Additionally, they observed that drafts written by AI tend to be more readable compared to those by humans (attributed to AI’s structured approach to paragraph organization). In fact, an AI-generated outline received better feedback from peer reviewers than its human counterpart. These benefits collectively point to AI’s potential to not only expedite research and writing tasks but also possibly elevate the quality and organization of scientific manuscripts.
So how could you use AI if you wanted to give it a roll? Here are some suggested steps:
- Conduct a comprehensive literature search and review on your topic manually: Do all the sourcing, evidence gathering, and citation managements the traditional way – don’t rely on AI for this important foundation, especially given the fact that training data won’t usually be up to date.
- Develop an outline for your review: Carefully structure sections and sub-sections for your article based on themes and importance to the field using your own human logic.
- Compose clear writing prompts for each section: Use detailed and specific prompts to guide the AI on the focus, citations to include, length, etc. as it drafts paragraph blocks dedicated to each topic. A typical prompt in this case might be 2-4 sentences in length.
- Run the AI-generated first full draft through plagiarism software (if you have it): Check for passages that may be too similar to sources and manually edit down as needed.
- Rigorously fact check statements and cites for accuracy: Don’t take AI verbatim – verify information, correct all factual errors, and add sources where lacking.
- Provide guidance queries on connecting concepts: Ask the AI chatbot explicit questions on relationships or gaps in understanding between topics that require human logic. This is where you where you put on your critical researcher cap. And remember: AI often wants to “please” you, and find answers that don’t exist to do so!
- Carefully proofread for typos, flow, repetition: Revise sections that feel disjointed or unclear and ensure polish.
If you’re saying to yourself now, Well, I might as well do all that without the AI! you wouldn’t be alone. It shows you’re, well, human. The technology frankly still has some maturing to do in terms of its ability to assess reliability and make connections. In short, the Kacena et al. research shows the current limits of the “I” in AI. Our job of being human beings, for now, seems safe. (And so is mine!)
