Socially-Driven Prompt Engineering: Harnessing Community Annotations to Map Prompt–Response Surprises

Wang et al. (2024) introduced the concept of “social prompt engineering,” but mainly for collaborative prompt authoring. This research idea takes it further by focusing specifically on collecting and categorizing unexpected or anomalous LLM behaviors encountered by diverse users—essentially crowd-sourcing a living taxonomy of prompt-induced surprises. Each community annotation would include the prompt, the response, and a user-tagged explanation of why the output was unexpected (e.g., factual error, bias, reversal, incoherence). Over time, this data could reveal underexplored model weaknesses across domains and user populations, and inform both model training and prompt design tools. The novelty is in the participatory, distributed “red teaming” that doesn’t rely solely on experts or static benchmarks. This could democratize discovery of LLM edge cases and drive more robust prompt engineering.

References:

1. Wordflow: Social Prompt Engineering for Large Language Models. Zijie J. Wang, Aishwarya Chakravarthy, David Munechika, Duen Horng Chau (2024). Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations).