LLM-Driven Adversarial Scenario Generation for Multi-Agent System Testing

While Ma et al. (2024) made significant strides in MAS testing with their diversity-guided exploration framework, their approach relies primarily on action perturbations at critical states. This research proposes a fundamentally different approach by leveraging the natural language understanding capabilities of LLMs (as surveyed by Wang et al., 2023) to generate semantically rich adversarial scenarios. Instead of just perturbing actions, the system would use LLMs to understand the domain context (e.g., traffic laws for autonomous vehicles, game rules for strategic MAS) and generate scenarios that violate implicit assumptions or create edge cases. For example, in a self-driving car simulation, the LLM could generate scenarios like "pedestrian suddenly reverses direction while crossing at unusual angle" rather than just random perturbations. This approach addresses a key limitation in current MAS testing methods - their focus on syntactic diversity rather than semantic diversity. By incorporating domain knowledge through LLMs, we could discover failure scenarios that are both diverse and realistic, potentially catching critical bugs that traditional fuzzing might miss. The innovation lies in bridging the gap between Pezzè's (2022) vision of NLP in testing and the specific challenges of MAS reliability.

References:

1. Machine learning and natural language processing for automating software testing (tutorial). M. Pezzè (2022). ESEC/SIGSOFT FSE.
2. Enhancing Multi-agent System Testing with Diversity-Guided Exploration and Adaptive Critical State Exploitation. Xuyan Ma, Yawen Wang, Junjie Wang, Xiaofei Xie, Boyu Wu, Shoubin Li, Fanjiang Xu, Qing Wang (2024). International Symposium on Software Testing and Analysis.
3. Software Testing With Large Language Models: Survey, Landscape, and Vision. Junjie Wang, Yuchao Huang, Chunyang Chen, Zhe Liu, Song Wang, Qing Wang (2023). IEEE Transactions on Software Engineering.