Background:A recent assessment of ChatGPT on a variety of obstetric and gynecologic
topics was very encouraging. However, its ability to respond to commonly asked
pregnancy questions is unknown. Reference verification needs to be examined as
well. Purpose: To evaluate ChatGPT as a source of information for
commonly asked pregnancy questions and to verify the references it provides. Methods: Qualitative analysis of ChatGPT was performed. We queried ChatGPT
Version 3.5 on 12 commonly asked pregnancy questions and asked for its
references. Query responses were graded as “acceptable”
or “not acceptable” based on correctness and completeness in comparison to
American College of Obstetricians and Gynecologists (ACOG) publications,
PubMed-indexed evidence, and clinical experience. References were classified as
“verified”, “broken”, “irrelevant”, “non-existent” or “no references”. Review
and grading of responses and references were performed by the co-authors
individually and then as a group to formulate a consensus. Results: In
our assessment, a grade of acceptable was given to 50% of responses (6 out of
12 questions). A grade of not acceptable was
References
[1]
Hu, K. (2023, February 2) ChatGPT Sets Record for Fastest-Growing User Base—Analyst Note.
https://www.reuters.com/technology/chatgpt-sets-record-fastest-growing-user-base-analyst-note-2023-02-01
[2]
American Medical Association (2023, March 3) ChatGPT Passed the USMLE. What Does It Mean for Med Ed?
https://www.ama-assn.org/practice-management/digital/chatgpt-passed-usmle-what-does-it-mean-med-ed
[3]
Kung, T.H., Cheatham, M., Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., et al. (2023) Performance of ChatGPT on USMLE: Potential for AI-Assisted Medical Education Using Large Language Models. PLOS Digital Health, 2, e0000198.
https://doi.org/10.1371/journal.pdig.0000198
[4]
Ayers, J.W., Poliak, A., Dredze, M., Leas, E.C., Zhu, Z., Kelley, J.B., Faix, D.J., Goodman, A.M., Longhurst, C.A., Hogarth, M. and Smith, D.M. (2023) Comparing Physician and Artificial Intelligence Chatbot Responses to Patient Questions Posted to a Public Social Media Forum. JAMA Internal Medicine, 183, 589-596.
https://doi.org/10.1001/jamainternmed.2023.1838
[5]
Grünebaum, A, Chervenak, J., Pollet, S.L., Katz, A. and Chervenak, F.A. (2023) The Exciting Potential for ChatGPT in Obstetrics and Gynecology. American Journal of Obstetrics and Gynecology, 228, 696-705. https://doi.org/10.1016/j.ajog.2023.03.009
[6]
Nolan, B. (2023) Two Professors Who Say They Caught Students Cheating on Essays with ChatGPT Explain Why AI Plagiarism Can Be Hard to Prove. Business Insider.
https://www.businessinsider.com/chatgpt-essays-college-cheating-professors-caught-students-ai-plagiarism-2023-1
[7]
Sanchez-Ramos, L., Lin, L. and Romero, R. (2023) Beware of References When Using ChatGPT as a Source of Information to Write Scientific Articles. American Journal of Obstetrics and Gynecology, 229, 356-357.
https://doi.org/10.1016/j.ajog.2023.04.004
Declercq, E.R., Sakala, C., Corry, M.P., Applebaum, S. and Herrlich, A. (2014) Major Survey Findings of Listening to Mothers(SM) III: Pregnancy and Birth: Report of the Third National U.S. Survey of Women’s Childbearing Experiences. The Journal of Perinatal Education, 23, 9-16. https://doi.org/10.1891/1058-1243.23.1.9
[10]
Popp, T. (2023, April 26) Alien Minds, Immaculate Bullshit, Outstanding Questions. The Pennsylvania Gazette.
https://thepenngazette.com/alien-minds-immaculate-bullshit-outstanding-questions/
[11]
Kelly, S., Kaye, S.A. and Oviedo-Trespalacios, O. (2023) What Factors Contribute to the Acceptance of Artificial Intelligence? A Systematic Review. Telematics and Informatics, 77, Article ID: 101925. https://doi.org/10.1016/j.tele.2022.101925
[12]
Yeo, Y.H., Samaan, J.S., Ng, W.H., Ting, P.S., Trivedi, H., Vipani, A., et al. (2023) Assessing the Performance of ChatGPT in Answering Questions Regarding Cirrhosis and Hepatocellular Carcinoma. Clinical and Molecular Hepatology.
https://doi.org/10.1101/2023.02.06.23285449
[13]
American College of Obstetricians and Gynecologists (2018) Air Travel during Pregnancy. ACOG Committee Opinion No. 746. Obstetrics & Gynecology, 132, e64-e66.
https://doi.org/10.1097/AOG.0000000000002757
[14]
Wilkerson, E.C., Van Acker, M.M., Bloom, B.S. and Goldberg, D.J. (2019) Utilization of Laser Therapy during Pregnancy: A Systematic Review of the Maternal and Fetal Effects Reported from 1960 to 2017. Dermatologic Surgery, 45, 818-828.
https://doi.org/10.1097/DSS.0000000000001912
[15]
ACOG Practice Advisory (2017) Update on Seafood Consumption in Pregnancy.
[16]
American College of Obstetricians and Gynecologists (2020) Tobacco and Nicotine Cessation during Pregnancy. ACOG Committee Opinion No. 807. Obstetrics & Gynecology, 135, e221-e229. https://doi.org/10.1097/AOG.0000000000003822
[17]
Polis, R.L., Gussman, D. and Kuo, Y.H. (2015) Yoga in Pregnancy: An Examination of Maternal and Fetal Responses to 26 Yoga Postures. Obstetrics & Gynecology, 126, 1237-1241. https://doi.org/10.1097/AOG.0000000000001137
[18]
Silver, R.M., Hunter, S., Reddy, U.M., et al. (2019) Prospective Evaluation of Maternal Sleep Position through 30 Weeks of Gestation and Adverse Pregnancy Outcomes. Obstetrics & Gynecology, 134, 667-676.
https://doi.org/10.1097/AOG.0000000000003458
American College of Obstetricians and Gynecologists (2010) Moderate Caffeine Consumption during Pregnancy. Committee Opinion No. 462. Obstetrics & Gynecology, 116, 467-468. https://doi.org/10.1097/AOG.0b013e3181eeb2a1
[21]
ACOG. Listeria and Pregnancy.
https://www.acog.org/womens-health/faqs/listeria-and-pregnancy
[22]
ACOG. Is It Safe to Dye My Hair during Pregnancy?
https://www.acog.org/womens-health/experts-and-stories/ask-acog/is-it-safe-to-dye-my-hair-during-pregnancy
[23]
American College of Obstetricians and Gynecologists (2017) Marijuana Use during Pregnancy and Lactation. Committee Opinion No. 722. Obstetrics & Gynecology, 130, e205-e209. https://doi.org/10.1097/AOG.0000000000002354
[24]
American Academy of Pediatrics, American College of Obstetricians and Gynecologists (2017) Guidelines for Perinatal Care. 8th Edition.
[25]
Varanasi, L. (2023) AI Models like ChatGPT and GPT-4 Are Acing Everything from the Bar Exam to AP Biology. Here’s a List of Difficult Exams both AI Versions Have Passed. Business Insider.
https://www.businessinsider.com/list-here-are-the-exams-chatgpt-has-passed-so-far-2023-1
[26]
Drake, L. (2023, May 2) We Fact-Checked ChatGPT’s Medical Advice. Healthnews.
https://healthnews.com/news/we-fact-checked-chatgpt-medical-advice
[27]
Alkaissi, H. and McFarlane, S.I. (2023) Artificial Hallucinations in ChatGPT: Implications in Scientific Writing. Cureus, 15, e35179.
https://doi.org/10.7759/cureus.35179
[28]
Gao, C.A., Howard, F.M., Markov, N.S., Dyer, E.C., Ramesh, S., Luo, Y. and Pearson, A.T. (2023) Comparing Scientific Abstracts Generated by ChatGPT to Real Abstracts with Detectors and Blinded Human Reviewers. NPJ Digital Medicine, 6, Article No. 75. https://doi.org/10.1038/s41746-023-00819-6
Texas Instruments Incorporated (1977) TI-30 Student Math Kit.
[31]
World Health Organization (2023, May 23) WHO Calls for Safe and Ethical AI for Health.
https://www.who.int/news/item/16-05-2023-who-calls-for-safe-and-ethical-ai-for-health
[32]
Blagic, D. (2022, December 20) Why Is the User Experience of Chatgpt So Powerful? Medium.
https://uxdesign.cc/why-is-the-user-experience-of-chatgpt-so-powerful-509e803e0122
[33]
Piloto, C. (2022, December 26) Artificial Intelligence vs Machine Learning: What’s the Difference? MIT Professional Education.
https://professionalprograms.mit.edu/blog/technology/machine-learning-vs-artificial-intelligence
[34]
What Is a Chatbot? IBM. https://www.ibm.com/topics/chatbots
[35]
Ji, Z., Lee, N., Frieske, R., et al. (2023) Survey of Hallucination in Natural Language Generation. ACM Computing Surveys, 55, Article No. 248.
https://doi.org/10.1145/3571730
[36]
(2023) Prepare for Truly Useful Large Language Models. Nature Biomedical Engineering, 7, 85-86. https://doi.org/10.1038/s41551-023-01012-6
[37]
What Is Natural Language Processing? IBM.
https://www.ibm.com/topics/natural-language-processing
