Date of Award

Summer 8-2026

Document Type

Thesis

Degree Name

Master of Science (MS)

Department

Electrical & Computer Engineering

Program/Concentration

Electrical and Computer Engineering

Committee Director

Lee Belfore

Committee Member

Jiang Li

Committee Member

Hicham Chaoui

Abstract

Accurate perception and control systems are essential for autonomous vehicles to safely interpret and navigate complex environments. Individual sensing modalities such as cameras, LiDAR, and inertial measurement units (IMUs) provide valuable environmental and vehicle state information; however, each sensor exhibits inherent limitations when used independently. Camera performance can degrade under varying lighting conditions, LiDAR measurements may be sparse or affected by environmental noise, and IMU measurements are susceptible to drift over time. Little Blue, a research autonomous vehicle platform from ODU (Old Dominion University), previously relied primarily on camera-based data for neural network training, limiting the system’s ability to exploit complementary information available from other sensors.

This thesis introduces FusionNet, a multimodal heterogeneous deep learning sensor fusion framework that integrates camera imagery, LiDAR scans, and IMU measurements to predict vehicle control commands for autonomous navigation. A multimodal dataset was constructed from ROS2 bag recordings collected during testing and competition events involving ODU’s Little Blue platform. A preprocessing pipeline was developed to extract sensor data, synchronize measurements, normalize inputs, and organize the information into a structured dataset suitable for deep learning applications. Camera data were processed using convolutional neural networks based on a ResNet-18 architecture, while LiDAR and IMU measurements were encoded through dedicated neural network branches. Learned feature representations from each modality were combined through a fusion network to generate steering and throttle predictions.

Training and evaluation were conducted using approximately 14,000 synchronized multimodal samples in an open-loop setting. The model demonstrated stable convergence during training and produced consistent steering and throttle predictions during rosbag playback, indicating that the fused sensor representation can be processed reliably within a real-time ROS2 inference environment. These results demonstrate the feasibility of heterogeneous sensor fusion for autonomous vehicle control, though full closed-loop validation and quantitative comparison against single-sensor baselines remain areas for future work.

The proposed framework provides a scalable foundation for real-time autonomous vehicle operation and highlights the potential benefits of multimodal deep learning for robust perception and control. Future work will focus on deployment to embedded computing platforms, expanded datasets, closed-loop vehicle testing, and advanced autonomous navigation capabilities.

Rights

In Copyright. URI: http://rightsstatements.org/vocab/InC/1.0/ This Item is protected by copyright and/or related rights. You are free to use this Item in any way that is permitted by the copyright and related rights legislation that applies to your use. For other uses you need to obtain permission from the rights-holder(s).

DOI

10.25776/8md7-et66

ISBN

9798193216360

Download

Share

COinS