Powered by internet-scale datasets and structures with billions of parameters, Large Language Models (LLMs) are revolutionizing human-machine interaction (HMI) in AVs. Their advanced contextual understanding and reasoning capabilities enable them to process human language, identify intent, and interpret nuanced semantics. In addition, they can translate complex machine signals—such as vehicle-control signals—into user-friendly explanations. Our research explores LLM-enhanced interfaces for AVs, aiming to create seamless, trustworthy communication and interaction between AVs and humans (passengers, nearby human drivers, and pedestrians).

Driver assistance technologies are transitioning from standardized, rule-based warnings to personalized, conversational support. LLMs and Vision Language Models (VLMs), with their advanced capabilities in contextual understanding, knowledge integration, and natural language interaction, provide a novel approach to the development of Personalized Driver Assistance (PDA) systems. By combining multimodal perception with the extensive knowledge of LLMs, we aim to develop in-vehicle agents capable of identifying hazards and engaging in intuitive interactions with drivers. This approach seeks to provide proactive, user-focused support, enhancing driving safety, comfort, and trust on the road.