Millimeter-wave and IR Multimodal Sensing System Enabling AI-based Event Recognition with Enhanced Privacy

Technologies for sensing and monitoring human-related activities and vehicles/objects are an important and increasing part of urban management and safety. It is desirable to enable these systems to operate without visible-domain cameras for enhanced privacy and robustness to varying illumination and weather conditions. In response to these requirements, we introduce a multimodal sensing system that comprises 60-GHz phased array TX and RX modules, a passive IR camera, a COTS-based FMCW radar sub-system, a mixed-signal data acquisition module, and an FPGA. The FPGA’s key role is to accurately control radar waveform generation, phased array beam steering, and IR camera data acquisition events, producing a synchronized high-speed stream of 3D radar (>3 Kfps) and IR (9 fps) data. The prototype system and an associated multi-modal DNN are evaluated in two use cases (1) concealed object detection and (2) head gesture classification.