Evaluating LLMs' Pragmatic Sensitivity: The Case of (In)Felicity

Felicity conditions, introduced by philosopher of language J.L. Austin, identify the contextual and social requirements that must be satisfied for a speech act to be successfully performed or a sentence to carry out its full communicative force. When these conditions are met, an utterance is considered "felicitous". As such conditions are often deeply embedded in particular contexts and relationships between interlocutors and require a sophisticated understanding of social-linguistic conventions, we are interested in LLMs' function as competent pragmatic speakers by examining how they interact with felicity conditions (which is a theory developed based on human communication):

1. Recognition: Given a particular conversational context, do LLMs recognise the conditions that render an utterance felicitous or infelicitous?
2. Behavioural response: How do LLMs typically respond to infelicitous utterances (which can take on various forms)? Do they ignore the violation, adapt the context to accommodate the utterance, or explicitly flag the infelicity?