Learning with Enriched Inductive Biases for Vision-Language Models

Information Science Artificial Intelligence Computer Science
Vision Language Models Inductive Biases Few-shot Adaptation Transformer

Learning with Enriched Inductive Biases for Vision-Language Models

Research Background and Problem Statement

In recent years, Vision-Language Models (VLMs) have made significant progress in the fields of computer vision and natural language processing. These models are pre-trained on large-scale image-text pairs to construct a unified multimodal representation space, demonstrating excellent performance across various downstream tasks. However, in few-shot learning scenarios, effectively fine-tuning these models for specific tasks while maintaining strong generalization capabilities remains an unresolved challenge.

Existing methods typically rely on prompt engineering or Parameter-Efficient Fine-Tuning (PEFT) strategies to optimize pre-trained models. Nevertheless, these approaches often overlook the importance of inductive biases, limiting the model’s generalization ability in complex scenarios. Inductive biases refer to built-in assumptions within algorithms that guide models toward specific solutions. For instance, weight sharing and translation invariance in Convolutional Neural Networks (CNNs) are classic examples of inductive biases, aiding models in learning more efficiently on smaller datasets.

To address these issues, this study proposes a novel framework—Learning with Enriched Inductive Biases (LWEIB)—which aims to enhance the performance of VLMs in few-shot tasks by introducing inductive biases at the text, model, and optimization levels.

Paper Source and Author Information

This paper was co-authored by Lingxiao Yang, Ru-Yuan Zhang, Qi Chen, and Xiaohua Xie, affiliated with institutions such as the School of Systems Science and Engineering at Sun Yat-sen University, the Brain Health Institute at Shanghai Jiao Tong University, and the School of Computer Science and Engineering at Sun Yat-sen University. The paper was published in the prestigious journal International Journal of Computer Vision (IJCV) and officially went online in January 2025.

Research Details and Workflow

a) Research Process and Method Design

The core of this research is the proposal of a novel framework—LWEIB—which optimizes the performance of VLMs by introducing inductive biases at three levels. Below are the details of the research process:

1. Text-Level Inductive Bias

The study first introduces rich descriptive information at the text level. Specifically, the authors supplement traditional handcrafted prompts with customized texts generated by large language models (LLMs). For example, for the category “Shiba Inu,” in addition to the traditional prompt “a photo of a Shiba Inu,” detailed descriptions such as “small,” “compact,” and “fox-like face” were added. This approach aims to bridge the semantic gap between language and visual modalities, thereby enhancing the model’s generalization capability.

2. Model-Level Inductive Bias

To better capture structural information in both language and vision, the authors designed two novel adapters: - Phrase Adapter (PA): Used for the text encoder, it explicitly models relationships between adjacent words through a one-dimensional depthwise convolutional layer (1D Depthwise Convolutional Layer). - Spatial Adapter (SA): Used for the image encoder, it captures local spatial relationships and details through a two-dimensional depthwise convolutional layer (2D Depthwise Convolutional Layer).

These two adapters are inserted into different positions of the Transformer blocks, such as after the multi-head self-attention layer (MSA) and after the first fully connected layer of the feed-forward network (FFN).

3. Optimization-Level Inductive Bias

During the optimization process, the authors propose a dynamic scaling factor α method called the “Slow-Fast Optimization Method.” This method flexibly balances underfitting and overfitting across different tasks by randomly adjusting the value of α. The specific formula is as follows: $$ dy(\alpha) = \begin{cases} s \cdot \alpha, & \text{prob} > 0.5 \ \alpha, & \text{otherwise} \end{cases} $$ Here, s is a hyperparameter used to control the scaling degree.

Experimental Setup

The study conducted experiments on three widely-used benchmark datasets, including ImageNet, Caltech101, and CUB-200. All experiments used a 16-shot setting, meaning only 16 training samples per category were utilized. The model was based on the CLIP (Contrastive Language–Image Pre-training) architecture and evaluated for its performance across multiple tasks.

b) Main Results and Data Analysis

1. Generalization Ability for Base and Novel Classes

The experimental results show that LWEIB outperforms existing methods in both base classes and novel classes. For example, on the ImageNet dataset, LWEIB achieved a novel class accuracy of 78.21%, surpassing the second-best method by 1.35%. Additionally, LWEIB achieved an average Harmonic Mean (HM) score of 81.21% across 11 datasets, significantly outperforming other methods.

2. Cross-Dataset Evaluation

In cross-dataset evaluations, LWEIB also performed exceptionally well, achieving an average accuracy of 68.61%, nearly 2% higher than the second-best method. Particularly on datasets with larger distribution shifts like Eurosat, DTD, and Aircraft, the advantages of LWEIB were even more pronounced.

3. Domain Generalization Ability

In domain generalization tasks, LWEIB performed best on 3⁄4 of the unseen domain datasets. This indicates that the framework has strong robustness and can effectively handle significant domain shifts.

Result Analysis

Through ablation experiments, the authors further validated the effectiveness of each module. For instance, using only the Phrase Adapter or Spatial Adapter resulted in inferior model performance compared to the complete framework; the introduction of the dynamic scaling factor α significantly improved the generalization ability for novel classes. These results demonstrate that LWEIB achieves more efficient model tuning through the synergistic effects of multi-level inductive biases.

c) Research Conclusions and Value

The main contribution of this research lies in proposing a novel framework—LWEIB—which significantly enhances the performance of VLMs in few-shot tasks by introducing inductive biases at the text, model, and optimization levels. Specifically: - Scientific Value: Reveals the importance of inductive biases in few-shot learning, providing new insights for future research. - Application Value: LWEIB demonstrates outstanding performance in multiple practical tasks and can be widely applied in areas such as image classification and object detection.

d) Research Highlights

  1. Design of Multi-Level Inductive Biases: First systematic introduction of inductive biases at the text, model, and optimization levels.
  2. Innovative Adapter Design: The Phrase Adapter and Spatial Adapter target linguistic and visual modalities respectively, capturing rich structural information.
  3. Dynamic Optimization Strategy: The Slow-Fast Optimization Method effectively balances underfitting and overfitting through random adjustments of the scaling factor.

Summary and Significance

This research not only proposes an efficient few-shot learning framework but also provides a new perspective for optimizing vision-language models. By introducing multi-level inductive biases, LWEIB has achieved leading performance in multiple benchmark tasks, showcasing its significant value in both theory and practice. In the future, the research team plans to further explore adaptive optimization strategies to reduce the impact of randomness while improving the stability and generalization ability of the model.