A vision chip with complementary pathways for open-world sensing

• 成果类型: Article
• 署名作者: Yang, Zheyu; Wang, Taoyi; Lin, Yihan; Chen, Yuguo; Zeng, Hui; Pei, Jing; Wang, Jiazheng; Liu, Xue; Zhou, Yichun; Zhang, Jianqiang; Wang, Xin; Lv, Xinhao; Zhao, Rong; Shi, Luping
• 署名单位: Tsinghua University; Tsinghua University; Tsinghua University; Tsinghua University
• 刊物名称: Nature
• ISSN/ISSBN: 0028-3819
• DOI: 10.1038/s41586-024-07358-4
• 发表日期: 2024-05-30
• 关键词: cmos image sensor; pixel; compression

摘要

Image sensors face substantial challenges when dealing with dynamic, diverse and unpredictable scenes in open-world applications. However, the development of image sensors towards high speed, high resolution, large dynamic range and high precision is limited by power and bandwidth. Here we present a complementary sensing paradigm inspired by the human visual system that involves parsing visual information into primitive-based representations and assembling these primitives to form two complementary vision pathways: a cognition-oriented pathway for accurate cognition and an action-oriented pathway for rapid response. To realize this paradigm, a vision chip called Tianmouc is developed, incorporating a hybrid pixel array and a parallel-and-heterogeneous readout architecture. Leveraging the characteristics of the complementary vision pathway, Tianmouc achieves high-speed sensing of up to 10,000 fps, a dynamic range of 130 dB and an advanced figure of merit in terms of spatial resolution, speed and dynamic range. Furthermore, it adaptively reduces bandwidth by 90%. We demonstrate the integration of a Tianmouc chip into an autonomous driving system, showcasing its abilities to enable accurate, fast and robust perception, even in challenging corner cases on open roads. The primitive-based complementary sensing paradigm helps in overcoming fundamental limitations in developing vision systems for diverse open-world applications. Inspired by the human visual system, a vision chip with primitive-based complementary pathways is developed to overcome the power and bandwidth wall of vision systems, achieving fast, precise, robust and high-dynamic-range sensing efficiently in the open world.