Creativity Support in the Age of Large Language Models: An Empirical Study Involving Emerging Writers

About

The development of large language models (LLMs) capable of following instructions and engaging in conversational interactions sparked increased interest in their utilization across various support tools. We investigate the utility of modern LLMs at assisting professional writers via an empirical user study (n=30). The design of our collaborative writing interface is grounded in the cognitive process model of writing that views writing as a goal-oriented thinking process encompassing non-linear cognitive activities: planning, translating and reviewing. Participants are asked to submit a post-completion survey to provide feedback on the potential and pitfalls of LLMs as writing collaborators. Upon analyzing the writer-LLM interactions, we find that while writers seek LLM's help across all three types of cognitive activities, they find LLMs more helpful in translation and reviewing. Our findings from analyzing both the interactions and the survey responses highlight future research directions in creative writing assistance using LLMs.

Examples

Browse the full directory of stories. Also, you could view the detailed qualitative feedback from our participants here.

Demo Video

Paper

            @inproceedings{10.1145/3635636.3656201,
            author = {Chakrabarty, Tuhin and Padmakumar, Vishakh and Brahman, Faeze and Muresan, Smaranda},
            title = {Creativity Support in the Age of Large Language Models: An Empirical Study Involving Professional Writers},
            year = {2024},
            isbn = {9798400704857},
            publisher = {Association for Computing Machinery},
            address = {New York, NY, USA},
            url = {https://doi.org/10.1145/3635636.3656201},
            doi = {10.1145/3635636.3656201},
            abstract = {The development of large language models (LLMs) capable of following instructions and engaging in conversational interactions has led to increased interest in their use across various support tools. We investigate the effectiveness of contemporary LLMs in assisting professional writers via an empirical user study (n=30). The design of our collaborative writing interface is grounded in the cognitive process model of writing  [17]. This allows writers to obtain model help in each of the three non-linear cognitive activities in the writing process: planning, translating and reviewing. Participants write short fiction/non-fiction with model help and are subsequently asked to submit a post-completion survey to provide qualitative feedback on the potential and pitfalls of LLMs as writing collaborators. Upon analyzing the writer-LLM interactions, we find that while seeking help across all three types of cognitive activities, writers find LLMs more helpful in translation and reviewing. Our findings from analyzing both the interactions and the survey responses highlight future research directions in creative writing assistance using LLMs.},
            booktitle = {Proceedings of the 16th Conference on Creativity \& Cognition},
            pages = {132–155},
            numpages = {24},
            keywords = {Co-Creativity, Computational Creativity, Creativity, Evaluation, Human-AI collaboration, Large Language Models, Natural Language Generation, StoryTelling},
            location = {Chicago, IL, USA},
            series = {C&C '24}
            }

Contact

If you have any questions or comments about this work, please contact Tuhin or Vishakh.