Prediction error processing and sharpening of expected information

Computer Science Information Science Neurology Clinical Psychology Medical Imaging Life Sciences Medicine Bioengineering
Prediction error Expected information Cognitive Process Face Perception Deep Neural Networks

Research Flowchart

Scientific Report

Background Introduction

Perception and neuronal processing of sensory information are largely influenced by prior expectations. Perception is not merely passive reception, but an active inferential process by combining existing sensory information with prior information based on past experience and current context. The combination of this information can manifest through different mechanisms: one focused on unexpected inputs, i.e., Prediction Error (PE); the other through the sharpening of expected information. This paper investigates these two mechanisms in face perception.

Research Origin

This research was conducted by Annika Garlichs and Helen Blank from the Department of Systems Neuroscience of the University Medical Center Hamburg-Eppendorf, Germany. The paper was published in the April 2024 issue of the journal Nature Communications.

Research Procedure and Methods

Research Design and Steps

  1. Experimental Design: The study is based on Functional Magnetic Resonance Imaging (fMRI) data, combined with computational modelling using Deep Convolutional Neural Networks (DCNN). Participants first learned to associate scene images with subsequently presented face images. Experimental images included four distinct male faces. Ambiguous faces were generated by morphing these faces to introduce uncertainty in face recognition.

  2. Experimental Process: The experiment consisted of four parts, namely the individual calibration of face images, association training, the main fMRI experiment, and the functional localizer experiment. In the main experiment, participants had to press a corresponding button with their right hand to indicate the identity of recognized faces.

  3. Multiple Comparisons Analysis: In data analysis, multiple comparisons analysis of different brain regions was used to distinguish between PE and sharpening representation. Comparing Neural Representational Dissimilarity Matrices (RDMs) under different experimental conditions and combining with the Deep Convolutional Neural Network (DCNN) models (such as vgg-face, vgg-16, and resnet50) were used to optimize the hypothetical model for explaining changes in face representation.

  4. Univariate and Multivariate Data Analysis: Univariate analysis was used to reveal the whole-brain activation differences between expected and unexpected face information, particularly in face-sensitive regions (such as the Occipital Face Area (OFA), Fusiform Face Area (FFA), and Anterior Temporal Lobe (ATL)). Further combining with Multivariate fMRI-based Representational Similarity Analysis (RSA) and computational modelling, different information processing mechanisms in various brain regions were identified.

Specific Research Steps

  • Individualized Face Morph Calibration: First, four different male faces were created using Facegen software, adjusting face features so that they had substantial differences in shape, color, and positional information. These faces were then morphed for each participant to determine their individual threshold for perceiving 50⁄50 morphed faces.

  • Association Training: Participants learned to associate four scene images with four face images. After the training, a scene image was first presented in the fMRI experiment, followed by an expected, unexpected, or ambiguous face image. Participants were required to select the face they recognized based on the cue.

  • Functional Magnetic Resonance Imaging Experiment: Brain activities were recorded via fMRI when participants were viewing and identifying face images. Multivariate fMRI-based Representational Similarity Analysis (RSA) was performed on ambiguous faces. Deep Neural Network models (such as vgg-face) were used to extract neural representations of information processing.

  • Hypothetical Model Generation: Hypothetical representational dissimilarity matrices (RDMs) were generated based on activation data from specific layers of the neural network (such as pool4 and pool5 of vgg-face). These were then compared with the neural data to investigate the roles of PE and sharpening mechanisms in face recognition.

Main Results

  1. Assimilation Effect: Behavioural data indicated that reaction times for recognizing expected faces were shorter (Fig.3b), and ambiguous faces were recognized as expected faces more frequently (Fig.3a), exhibiting a marked assimilation effect.

  2. Reduced fMRI Activation: For expected faces, it was found that fMRI activation was significantly lower compared to unexpected faces, especially in the posterior Fusiform Face Area (pFFA) and Inferior/Middle Temporal Gyrus (ITG/MTG) regions (Fig.4).

  3. PE and Sharpening Representation: RSA results showed that a PE processing mechanism was present throughout the face-processing hierarchy (OFA, pFFA to ATL) (Fig.4a, d-f), while evidence of sharpening representation was also found in the early face processing area OFA (Fig.4b).

Conclusion and Significance

This study unveils the neural representation mechanisms of expected contextual scenarios during face perception. Through multivariate fMRI combined with deep neural network analysis, the following conclusions were drawn: 1. Importance of Prediction Error in Face Processing: Throughout the entire face processing hierarchy, from the Occipital Face Area (OFA), Fusiform Face Area (FFA) to the Anterior Temporal Lobe (ATL), prediction error processing was predominant.

  1. Existence of Sharpening Representation: Evidence of sharpening representation was found in the early stages of face processing (such as OFA).

  2. How the Brain Integrates Prior Knowledge and Sensory Inputs: The study provided evidence supporting the predictive processing mechanism in the brain integrating prior knowledge and sensory inputs, which influences our perception of faces.

Research Highlights

  • Multivariate Analysis Distinguishing PE and Sharpening Mechanisms: Through multivariate fMRI combined with deep neural network activation models, the study successfully distinguished prediction error processing and sharpening representation, providing new insights into face perception.
  • Novel Experimental Design: This study accurately measures neural representational differences under controlled premises through an innovative experimental design using ambiguous faces and scene association training.

Additional Information

  • Accuracy and Repeatability of Methods: Advanced Deep Neural Networks (like vgg-face, vgg-16, and resnet50) were used to simulate and analyze brain mechanisms in face processing. The results show that PE and sharpening representation can coexist, enhancing our understanding of brain mechanisms in visual information processing.
  • Future Research Directions: Future studies could further explore whether these representations are associated with different information processing mechanisms at various levels in the brain, particularly in higher spatial resolution researches.

This study offers in-depth understanding of how the brain processes face information through prediction and sharpening mechanisms, which highlights the roles and importance of these mechanisms in the process of perception.