Can Artificial Intelligence Write Like Borges? An Evaluation Protocol for Spanish Microfiction

In a groundbreaking study, researchers at Tecnológico de Monterrey and Université Sorbonne Paris Nord unveil GrAImes, an innovative protocol designed to evaluate the literary value of AI-generated microfictions, specifically questioning if such technology can rival human authors, notably those akin to Borges in creativity and depth.

Microfiction, characterized by its brevity and intense literary ambition, serves as the study’s focal point. Traditionally, literary quality assessments hinge on subjective weightings. However, GrAImes strives for an objective standard by incorporating thematic coherence, interpretive depth, and aesthetic merit. The protocol’s development was validated with input from literature experts, and it aims to enhance previous evaluation methods which often rely on quantitative metrics, failing to capture artistic nuances.

In initial experiments, six microfictions—three human-written and three AI-generated—were evaluated by literature experts and enthusiasts. Human evaluators assigned higher ratings to AI-generated fictions from ChatGPT-3.5 over those produced by a GPT-2 fine-tuned model, Monterroso. The AI-generated texts impressed with coherence and engagement but lacked the innovative and interpretative richness characteristic of expert human authors.

High marks were given for narrative engagement, suggesting AI’s potential in maintaining reader interest. However, repeated shortcomings in originality and innovation indicate a nascent stage for AI in creative writing. Despite this, experts found GrAImes a valuable tool, acknowledging its potential in bridging subjective literary critique with objective evaluation.

The study underscores AI’s potential in literary fields but reveals challenges in emulating human creativity’s full breadth, much due to existing limitations in training datasets and depth in narrative generation.