Report on the Academic Paper: “Reconfigurable In-Sensor Processing Based on a Multi-Phototransistor-One-Memristor Array: A New Visual Computing Platform Combining Machine Learning and Brain-Inspired Neural Networks”

Academic Background and Problem Identification

Artificial vision systems play a significant role in intelligent edge computing. However, traditional systems, based on CMOS (complementary metal-oxide semiconductor) technology and the von Neumann architecture, are constrained in terms of efficiency due to the physical separation between image sensors, memory modules, and processors. Such separation results in data redundancy, increased signal processing delays, circuit complexity, and power consumption, limiting the system’s real-time processing capabilities. In real-world applications, traditional systems require complex processes from signal capture to image processing, but their efficiency remains insufficient.

In recent years, in-sensor computing, a novel architecture that integrates perception and computation, has emerged as a focus of interest. This architecture incorporates in-memory computing and neuromorphic features to facilitate real-time, low-power processing of complex spatiotemporal images. The integration of optical neural networks (ONNs) and brain-inspired spiking neural networks (SNNs) offers promising opportunities for machine vision. However, current optical computing devices, based on memristors, face functionality limitations in supporting diverse neural network architectures, making it challenging to construct versatile and reconfigurable visual computing systems.

To address these challenges, this paper proposes a new in-sensor processing system based on a multi-phototransistor-one-memristor (MP1R) array. The aim is to explore a universal, reconfigurable visual system that combines machine learning and brain-inspired neural networks. This study demonstrates how to achieve integrated sensing, storage, and computation functions on a single hardware platform.

Paper Source and Contributors

This paper was collaboratively conducted by researchers from Peking University’s School of Integrated Circuits, Institute for Artificial Intelligence, Guangdong Provincial Key Laboratory of In-Memory Computing Chips, and the China Institute for Brain Research. The first author is Bingjie Dang, with Yuchao Yang as the corresponding author. The paper was published online in Nature Electronics (Volume 7, pages 991-1003) on November 12, 2024.

Research Steps and Technical Details

This paper develops a novel optical sensing and neuromorphic vision computing system based on the MP1R array. Below are the main steps and technical details of the research:

1. Development of the MP1R Array

(1) Structural Design and Fabrication

The MP1R array consists of a 20×20 phototransistor array and 20-channel reconfigurable tantalum oxide-based memristors. The phototransistors are made of IGZO (indium gallium zinc oxide) thin films, while the memristors are based on a multi-layer Ta/TaOx/NbOx/W heterostructure. Scanning electron microscopy (SEM) and transmission electron microscopy (TEM) captured the MP1R array’s structure, as well as detailed views of its central units.

(2) Simulation and Experimental Validation

The memristors demonstrated three configurable modes through voltage manipulation: linear resistance response, volatile memory, and threshold switching. Through applied voltage sweeps, the memristors exhibited excellent stability and consistency over multiple cycles, ensuring robust processing and storage of complex optical signals.

2. Exploration of Optical Neural Network Architectures

The MP1R array was integrated into a hardware-based neuromorphic vision system, working in tandem with a 1T1R (one-transistor and one-memristor) non-volatile memory array. The system supported three neural network architectures:

(a) Optical Convolutional Neural Networks (OCNNs)

Using the convolution kernel capabilities of the MP1R array, static optical images were converted to electrical signals for processing. Testing on MNIST handwritten digits demonstrated a hardware experimental accuracy of 85.75%, closely matching the simulation accuracy.

(b) Optical Recurrent Neural Networks (ORNNs)

The MP1R array effectively supported dynamic event image processing. The system demonstrated real-time spatiotemporal integration, tested on the N-MNIST dataset, achieving a classification accuracy of 85.3% without processing delays.

© Optical Spiking Neural Networks (OSNNs)

By configuring the MP1R array for optical sensory and spike encoding functionalities, the platform successfully distinguished complex image targets, such as identical shapes with different colors (e.g., red “0” and blue “0”)—an innovation for real-world image recognition tasks.

3. Data Analysis and Encoding Techniques

The study developed specific hardware and software learning methodologies for the different neural network architectures. OCNNs and OSNNs utilized offline training and 4-bit quantized encoding methods for weights, while ORNNs employed online learning.

4. System Integration and Application Demonstration

The MP1R array and the 1T1R memory array were assembled into a unified hardware test platform equipped with analog/digital circuits for controlling voltage pulses, measuring currents, and executing learning and computation tasks. Experimental results validated the versatility of the platform in supporting various optical neural networks.

Research Outcomes and Scientific Value

1. Key Findings:

   • The study achieved, for the first time, the realization of linear resistance, volatile memory, and threshold switching modes in a single memristor array. These features are critical for achieving multi-functional neuromorphic computing.
   • A novel multimodal vision computing architecture was proposed, based on the MP1R array, supporting ONNs, RNNs, and SNNs.

2. Scientific Significance:

   • This work pioneers the integration of sensing, storage, and computing into a single hardware platform, addressing the limitations of data redundancy and high latency in traditional systems.
   • It provides new insights into combining machine learning with brain-inspired computation, offering a blueprint for future artificial intelligence computing hardware.

3. Application Potential:

   • The system’s ability to handle static, event-based, and color images has applications in areas such as autonomous driving, smart homes, medical monitoring, and robotic vision. Its color-sensitive image processing capabilities broaden its practical utility.

4. Performance Advantages and Innovations:

   • The system achieves complex image recognition tasks with low power consumption and high real-time performance, surpassing existing technologies in architectural compatibility and hardware integration.
   • Efficiently handles static images, dynamic events, and color-sensitive data, representing a significant advancement over conventional memristor-based systems.

Conclusion and Outlook

This study innovatively develops an integrated hardware MP1R array with optical sensing and computational capabilities. Together with the 1T1R non-volatile storage array, it constitutes a highly reconfigurable optical neural computing platform. The results validate its broad applicability to various neural network architectures, offering new paradigms for developing next-generation edge intelligence devices.

However, future research is needed to explore compatibility and scalability for mass-production. This rapidly evolving field could benefit greatly from optimizing array integration and algorithm design, with the potential to significantly enhance the capabilities of complex visual processing systems. This work establishes a solid theoretical and experimental foundation for optical neuromorphic computing technologies, with promising implications for diverse real-world applications.