Research presentation at IEEE IV 2025
Liu, D2 student, presented his research at the international conference IEEE IV2025 held in Cluj-Napoca, Romania.
Publications & Talks
International Conference · 2025
LLM-Driven Adaptive Autonomous Robot Navigation via Multimodal Fusion for Diverse Environments
Abstract
This paper presents a novel autonomous navigation framework that integrates Large Language Models (LLMs) with multimodal sensor fusion to enable dynamic obstacle avoidance and human-aware path planning in diverse environments. The proposed system leverages an FPGA-accelerated fusion pipeline, combining LiDAR and vision data for real-time perception. A Hungarian algorithm-based object matching technique ensures robust tracking, while a bird's-eye view (BEV) representation enhances spatial reasoning and occlusion handling. The fused sensory inputs are processed by a fine-tuned LLM, which contextualizes pedestrian behavior and environmental constraints to generate adaptive, human-centric navigation strategies. Unlike traditional rule-based methods, LLMs provide generalization capabilities to novel scenarios, significantly improving interaction with vulnerable pedestrians such as children, elderly individuals, and wheelchair users. Extensive evaluations in both simulated and real-world scenarios confirm the system's ability to reduce collisions and enhance navigation efficiency in high-density environments. By bridging semantic reasoning and robotic control, this work lays the foundation for next-generation intelligent navigation systems that are both safety-aware and scalable across autonomous platforms.
Research Note
This research addresses the challenges of autonomous robot navigation in dynamic, high-density environments (e.g., train stations and shopping malls) by proposing a novel framework that integrates multimodal sensor fusion (LiDAR and vision) with a Large Language Model (LLM). To overcome the limitations of rule-based methods in handling unpredictable human behavior and dynamic obstacles, our system combines FPGA-accelerated real-time data processing and LLM-driven socially compliant path planning. Specifically, LiDAR point clouds and Triple-RGB camera data are fused on an FPGA using the Hungarian algorithm, while the LLM analyzes pedestrian attributes (age, wheelchair usage) to dynamically adjust navigation priorities. Experimental results demonstrate a 40% reduction in pedestrian prediction error compared to baseline models, with FPGA processing achieving sub-10ms latency. Future work includes enhancing inference accuracy via Q-LoRA and independent FPGA module verification.
Related Lab Activities
Related Projects
LLM-Driven Urban Transportation Simulation Platform
Japan Science and Technology Agency (JST) JST Strategic Basic Research Programs (PRESTO)
[Social Transformation Platform] Co-Creation of the Transformation Platform Technology for Human and Society by Integration of the Humanities and Sciences
