The Language of AI and Human Poetry: A Comparative Lexicometric Study
Abstract
This study conducts a lexicometric analysis to compare the lexical richness and diversity in poetry generated by AI models with that of human poets. Employing a robust dataset that includes 1,333 AI-generated poems and 517 human-authored poems across seven distinct poetic eras, six key lexical metrics—Maas Index, MTLD, MATTR, HD-D, Hapax Legomenon Ratio, and Lexical Density—were applied for comparative analysis. The lexical characteristics of the poems were studied through a series of statistical tests and machine learning techniques, including Mann-Whitney U tests, Cliff's Delta, and Random Forest classification. The findings reveal a marked lexical superiority in human poetry, evidenced by significant differences and large effect sizes in all metrics except Lexical Density. HD-D emerged as the most discriminating factor, adeptly differentiating human poetry from its AI-generated counterpart. Further analysis identified the GPT-4 model as exhibiting the closest alignment to human poetry in terms of lexical attributes. The study discusses these outcomes in the context of AI's evolving linguistic competencies, shedding light on the inherent challenges and future prospects of AI in creative writing. Thus, this research provides an empirical framework for assessing AI’s language generation abilities and sets the stage for further interdisciplinary exploration into the frontiers of artificial creativity.
Keywords: artificial intelligence; lexicometry; machine learning; lexical analysis; poetry
Full Text:
PDFReferences
Abu-Rabiah, E. (2023). Evaluating L2 Vocabulary Development Features Using Lexical Density And Lexical Diversity Measures. LLT Journal: Journal on Language and Language Teaching, 26(1), 168–182. https://doi.org/10.24071/llt.v26i1.5841
Alowedi, N. A., & Al-Ahdal, A. A. M. H. (2023). Artificial Intelligence based Arabic-to-English machine versus human translation of poetry: An analytical study of outcomes. Journal of Namibian Studies : History Politics Culture, 33. https://doi.org/10.59670/jns.v33i.800
Atkinson, P., & Barker, R. (2023). AI and the social construction of creativity. Convergence: The International Journal of Research into New Media Technologies, 29(4), 1054–1069. https://doi.org/10.1177/13548565231187730
Attak, E. H. (2023). Tax Policy and Entrepreneurship (pp. 421–442). https://doi.org/10.4018/978-1-6684-8781-5.ch019
Ayinde, L., Wibowo, M. P., Ravuri, B., & Bin Emdad, F. (2023). ChatGPT as an important tool in organisational management: A review of the literature. Business Information Review, 40(3), 137–149. https://doi.org/10.1177/02663821231187991
Belfi, A. M., Vessel, E. A., & Starr, G. G. (2018). Individual Ratings of Vividness Predict Aesthetic Appeal in Poetry. Psychology of Aesthetics, Creativity, and the Arts, 12(3), 341–350. https://doi.org/10.1037/aca0000153
Benoit, K. (2020). The SAGE Handbook of Research Methods in Political Science and International Relations (L. Curini & R. Franzese, Eds.; Vol. 2). SAGE Publications Ltd. https://doi.org/10.4135/9781526486387
Bestgen, Y. (n.d.). Measuring Lexical Diversity in Texts: The Twofold Length Problem. https://doi.org/https://doi.org/10.48550/arXiv.2307.04626
Brglez, M., & Vintar, Š. (2022). Lexical Diversity in Statistical and Neural Machine Translation. Information. https://doi.org/10.3390/info13020093
Chen, R., & Liu, H. (2014). Quantitative Aspects of Journal of Quantitative Linguistics. Journal of Quantitative Linguistics, 21(4), 299–340. https://doi.org/10.1080/09296174.2014.944327
Collentine, J. (2004). The Effects of Learning Contexts on Morphosyntactic and Lexical Development. Studies in Second Language Acquisition. https://doi.org/10.1017/s0272263104262040
Cramer, J. (2016, May 16). Can Robot Artists Create Human-Quality Work? Not Yet. https://home.dartmouth.edu/news/2016/05/can-robot-artists-create-human-quality-work-not-yet
Fergadiotis, G., Wright, H. H., & West, T. M. (2013). Measuring lexical diversity in narrative discourse of people with aphasia. American Journal of Speech-Language Pathology, 22(2). https://doi.org/10.1044/1058-0360(2013/12-0083)
Hakami, A., Alqarni, R., Almutairi, M., & Alhothali, A. (2021). Arabic Poems Generation using LSTM, Markov-LSTM and Pre-Trained GPT-2 Models. Advances in Machine Learning, 139–147. https://doi.org/10.5121/csit.2021.111512
Heng, R., Pu, L., & Liu, X. (2023). The Effects of Genre on the Lexical Richness of Argumentative and Expository Writing by Chinese EFL Learners. Frontiers in Psychology. https://doi.org/10.3389/fpsyg.2022.1082228
Hong, J.-W., & Curran, N. M. (2019). Artificial Intelligence, Artists, and Art. ACM Transactions on Multimedia Computing, Communications, and Applications, 15(2s), 1–16. https://doi.org/10.1145/3326337
Hutson, J., & Schnellmann, A. (2023). The Poetry of Prompts: The Collaborative Role of Generative Artificial Intelligence in the Creation of Poetry and the Anxiety of Machine Influence. Faculty Scholarship, 462. https://digitalcommons.lindenwood.edu/faculty-research-papers/462/
Jarvis, S. (2013). Capturing the Diversity in Lexical Diversity. Language Learning, 63(SUPPL. 1), 87–106. https://doi.org/10.1111/j.1467-9922.2012.00739.x
Kantosalo, A., & Riihiaho, S. (2019). Quantifying co-creative writing experiences. Digital Creativity, 30(1), 23–38. https://doi.org/10.1080/14626268.2019.1575243
Köbis, N., & Mossink, L. D. (2021). Artificial intelligence versus Maya Angelou: Experimental evidence that people cannot differentiate AI-generated from human-written poetry. Computers in Human Behavior, 114. https://doi.org/10.1016/j.chb.2020.106553
Kubi, B. (2018). Bemoaning of Love: An Aspect of Ga Women’s Discourse on Love in Adaawe Song- Texts. International Journal of Comparative Literature and Translation Studies, 6(2), 43. https://doi.org/10.7575/aiac.ijclts.v.6n.2p.43
Labaca-Castro, R. (2023). Generative Adversarial Nets. In Machine Learning under Malware Attack (pp. 73–76). Springer Fachmedien Wiesbaden. https://doi.org/10.1007/978-3-658-40442-0_9
Lee, H.-K. (2022). Rethinking creativity: creative industries, AI and everyday creativity. Media, Culture & Society, 44(3), 601–612. https://doi.org/10.1177/01634437221077009
Lo, K.-L., Ariss, R., & Kurz, P. (2022). GPoeT-2: A GPT-2 Based Poem Generator. http://arxiv.org/abs/2205.08847
Mandják, T., Lavissière, A., Hofmann, J., Bouchery, Y., Lavissière, M. C., Faury, O., & Sohier, R. (2019). Port marketing from a multi-disciplinary perspective: A systematic literature review and lexicometric analysis. Transport Policy, 84, 50–72. https://doi.org/10.1016/j.tranpol.2018.11.011
McCarthy, P. M., & Jarvis, S. (2010). MTLD, vocd-D, and HD-D: A validation study of sophisticated approaches to lexical diversity assessment. Behavior Research Methods, 42(2), 381–392. https://doi.org/10.3758/BRM.42.2.381
Meng, Q. (2021). The Pedagogy of Corpus-Aided English-Chinese Translation From a Critical &Amp; Creative Perspective. Theory and Practice in Language Studies. https://doi.org/10.17507/tpls.1101.04
Obermeier, C., Menninghaus, W., von Koppenfels, M., Raettig, T., Schmidt-Kassow, M., Otterbein, S., & Kotz, S. A. (2013). Aesthetic and Emotional Effects of Meter and Rhyme in Poetry. Frontiers in Psychology. https://doi.org/10.3389/fpsyg.2013.00010
Parsons, L. T., & Pinkerton, L. (2022). Poetry and Prose as Methodology: A Synergy of Knowing. Methodological Innovations. https://doi.org/10.1177/20597991221087150
Pulvirenti, G., & Gambino, R. (2022). Einbildungskraft (Imagination). Goethe-Lexicon of Philosophical Concepts, 2(1). https://doi.org/10.5195/glpc.2022.59
Rahmeh, H. (2023). Digital Verses Versus Inked Poetry: Exploring Readers’ Response to AI-Generated and Human-Authored Sonnets. Scholars International Journal of Linguistics and Literature, 6(09), 372–382. https://doi.org/10.36348/sijll.2023.v06i09.002
Rezwana, J., & Maher, M. L. (2023). Designing Creative AI Partners with COFI: A Framework for Modeling Interaction in Human-AI Co-Creative Systems. ACM Transactions on Computer-Human Interaction, 30(5), 1–28. https://doi.org/10.1145/3519026
Richter, A., Ng, K., & Fallah, B. (2019). Bibliometric and text mining approaches to evaluate landfill design standards. Scientometrics, 118. https://doi.org/10.1007/s11192-019-03011-4
Rockmore, D. (2020, January 7). What Happens When Machines Learn to Write Poetry. The New Yorker. https://www.newyorker.com/culture/annals-of-inquiry/the-mechanical-muse
Sennrich, R., Haddow, B., & Birch, A. (2016). Neural Machine Translation of Rare Words with Subword Units. Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 1715–1725. https://doi.org/10.18653/v1/P16-1162
Sugunan, D. (2022). Multifarious nature in Bharathy’s Lyrical Literature. International Research Journal of Tamil, 4(SPL 2), 1–7. https://doi.org/10.34256/irjt22s21
Sujatna, E. T. S., Heriyanto, H., & Andri, S. (2021). Lexical Density and Variation in Indonesian Folklores in English Student Textbooks: An SFL Study. Leksika Jurnal Bahasa Sastra Dan Pengajarannya. https://doi.org/10.30595/lks.v15i2.11102
Sunico, R. C. (2021). The Poetry of Simple Words. Perspectives in the Arts and Humanities Asia. https://doi.org/10.13185/paha2020.10208
Swett, B. A., Hahn, E. N., & Llorens, A. J. (2021). Designing Robots for the Battlefield: State of the Art. In Robotics, AI, and Humanity (pp. 131–146). Springer International Publishing. https://doi.org/10.1007/978-3-030-54173-6_11
Thorndike, E. L. (1921). The Teacher’s Word Book. Teacher’s College, Columbia University. https://pure.mpg.de/rest/items/item_2395369_2/component/file_2395368/content
Uccelli, P., Galloway, E. P., Barr, C. D., Meneses, A., & Dobbs, C. L. (2015). Beyond Vocabulary: Exploring Cross-Disciplinary Academic-Language Proficiency and Its Association With Reading Comprehension. Reading Research Quarterly. https://doi.org/10.1002/rrq.104
Van de Cruys, T. (2020). Automatic Poetry Generation from Prosaic Text. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2471–2480. https://doi.org/10.18653/v1/2020.acl-main.223
Vitta, J. P., Nicklin, C., & Albright, S. W. (2023). Academic Word Difficulty and Multidimensional Lexical Sophistication: An English‐for‐academic‐purposes‐focused Conceptual Replication of Hashimoto and Egbert (2019). Modern Language Journal. https://doi.org/10.1111/modl.12835
Wassiliwizky, E., Koelsch, S., Wagner, V., Jacobsen, T., & Menninghaus, W. (2017). The emotional power of poetry: neural circuitry, psychophysiology and compositional principles. Social Cognitive and Affective Neuroscience, 12(8), 1229–1240. https://doi.org/10.1093/scan/nsx069
Yi, X., Li, R., & Sun, M. (2018). Chinese Poetry Generation with a Salient-Clue Mechanism. Proceedings of the 22nd Conference on Computational Natural Language Learning, 241–250. https://doi.org/10.18653/v1/K18-1024
Zenker, F., & Kyle, K. (2021). Investigating minimum text lengths for lexical diversity indices. Assessing Writing, 47. https://doi.org/10.1016/j.asw.2020.100505
Zhang, Y., & Wu, W. (2021). How effective are lexical richness measures for differentiations of vocabulary proficiency? A comprehensive examination with clustering analysis. Language Testing in Asia, 11(1). https://doi.org/10.1186/s40468-021-00133-6
DOI: http://dx.doi.org/10.17576/3L-2024-3002-01
Refbacks
- There are currently no refbacks.
eISSN : 2550-2247
ISSN : 0128-5157