Low-latency Multimodal Video Game Assistant

It would be fun and useful to have a video game assistant that can coordinate with the user like a real human teammate. However, most video games have multimodal stimuli that need to be processed to obtain all the required information. While there are models that can interpret multimodal inputs, they are often too slow (over a few seconds), such that the response is not informative and useful enough to the user. Therefore, an AI assistant that uses some more lightweight models and preprocesses the input to extract useful features in the stimuli (such as the sounds of the enemy approaching) may accelerate the process.