Beyond Price and News: Multimodal LLM Trading Agents with Social Media, Audio, and Visual Signals

Research Question: Can integrating multimodal data sources (e.g., social media, audio transcripts, satellite images) into LLM-based trading agents improve real-world trading performance compared to price-and-news-only agents?

Hypothesis: Access to timely, diverse, and multimodal information streams will enable LLM agents to detect market-relevant signals (e.g., sentiment shifts, supply chain disruptions) earlier, leading to superior risk-adjusted returns and faster adaptation to novel events.

Experiment Plan: - Extend LiveTradeBench to include real-time feeds from Twitter/X, earnings call audio (converted to text), and potentially visual data (e.g., shipping port satellite images).

• Fine-tune or prompt LLM agents for multimodal chain-of-thought reasoning, leveraging recent work on vision-language models for sequential decision-making.
• Compare live trading outcomes, adaptation speed, and error types between multimodal and baseline agents.
• Analyze which modalities most frequently contribute to profitable or loss-avoiding decisions.

References: ['Yuexiang Zhai et al. (2024). Fine-Tuning Large Vision-Language Models as Decision-Making Agents via Reinforcement Learning. Neural Information Processing Systems.', 'Hongyang Yang et al. (2024). FinRobot: An Open-Source AI Agent Platform for Financial Applications using Large Language Models. arXiv.org.', 'Mengxi Xiao et al. (2025). Retrieval-augmented Large Language Models for Financial Time Series Forecasting.']