rishii100/knowledge-guided-vqa-models

Knowledge-Guided Visual Question Answering Models

This repository contains trained model checkpoints for Knowledge-Guided Visual Question Answering (VQA) on an augmented subset of the OK-VQA dataset.

📄 Research paper: Rethinking LLM-Guided Visual Question Answering: Does Deeper Knowledge Integration Really Help?

The paper investigates whether deeper integration of LLM-generated knowledge improves VQA performance under data-scarce fine-tuning conditions. We compare decision-level late fusion with attention-matrix injection using a shared ViT + FLAN-T5 framework.

Research Highlights

• Compare shallow, decision-level late fusion against deep attention injection.
• Generate structured guidance with Llama 3.3 70B.
• Encode guidance using a CLIP text encoder.
• Expand the filtered training set from 3,144 to approximately 36,000 samples through question paraphrasing.
• Evaluate the baseline, late fusion, and attention injection using a consistent experimental setup.

The results show that late fusion performs better than attention injection in this data-scarce experimental setting. This does not imply that deeper integration is always inferior.

Architecture

Component	Model
Vision encoder	google/vit-base-patch16-224
Guidance text encoder	openai/clip-vit-base-patch16
Answer generation model	google/flan-t5-small
Benchmark	OK-VQA
LLM used to generate guidance	Llama 3.3 70B

The pipeline generates structured guidance containing visual focus concepts, distractor elements to ignore, and external knowledge statements. A CLIP text encoder represents this guidance as an embedding used by the downstream VQA model.

Integration strategies

1. Baseline — No Guidance

Uses visual features and question representations without external LLM-generated guidance.

2. Late Fusion — Shallow Integration

Combines the global ViT representation with the encoded guidance immediately before answer decoding. The visual feature extraction itself is left unchanged.

3. Attention Injection — Deep Integration

Injects guidance into the query and key projections of the last six ViT layers, modifying visual attention.

Available Checkpoints

Checkpoint	Strategy	Approximate file size
best_baseline_model.pth	No guidance	309 MB
best_fusion_model.pth	Late fusion	313 MB
best_attention_model.pth	Attention injection	325 MB

These files contain trained checkpoint weights. Loading them requires the matching model architecture and compatible pretrained components.

Evaluation Results

The following scores are reported in the paper for the validation split of the augmented, top-500-filtered OK-VQA subset. These are not official full-benchmark OK-VQA leaderboard scores.

Strategy	Exact Match (%)	Edit Similarity (%)	Token F1 (%)	BERTScore
Baseline — No Guidance	29.44	40.57	30.97	91.21
Late Fusion	39.44	48.97	41.34	92.14
Attention Injection	31.80	42.66	33.33	91.00

Main finding

Late fusion achieves 39.44% exact match, a 10.00-percentage-point improvement over the baseline. Attention injection achieves 31.80%, a 2.36-point improvement over the baseline.

These results suggest that, with limited fine-tuning data and largely frozen pretrained components, simpler knowledge integration can be more effective than modifying visual attention.

Training Details

• Dataset: Filtered OK-VQA training data with the top 500 answer classes.
• Data augmentation: LLM-based question paraphrasing.
• Original filtered training set: 3,144 samples.
• Augmented training set: Approximately 36,000 samples.
• Optimizer: Adam.
• Learning rate: 3e-5.
• Batch size: 8–16.
• Maximum epochs: 15.
• Early stopping: Validation exact match, patience 3–4.
• Evaluation metrics: Exact Match, Edit Similarity, Token F1, and BERTScore.

The ViT and CLIP encoders remain frozen except for the explicitly modified and trainable components of the attention-injection strategy.

Usage

You can download a checkpoint from this repository with huggingface_hub:

from huggingface_hub import hf_hub_download

checkpoint_path = hf_hub_download(
    repo_id="rishii100/knowledge-guided-vqa-models",
    filename="best_fusion_model.pth",
)

print("Checkpoint downloaded to:", checkpoint_path)

To run inference, instantiate the matching model class and load the checkpoint:

import torch

# YourLateFusionModel is a placeholder for the exact model class
# used during training. It must match the checkpoint's architecture.
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model = YourLateFusionModel().to(device)

state_dict = torch.load(
    checkpoint_path,
    map_location=device,
    weights_only=True,
)
model.load_state_dict(state_dict)
model.eval()

Important: YourLateFusionModel is a placeholder, not an included implementation. The model class, forward pass, tokenizer configuration, and pretrained components must match the training implementation. The checkpoint alone is not a complete, standalone inference pipeline.

Limitations

• Results are based on the experimental subset and validation protocol described in the paper.
• Performance may differ on the full official OK-VQA benchmark or under different training-data regimes.
• Attention injection can help some question categories while harming others.
• The checkpoint files do not, by themselves, provide the complete training pipeline or guarantee reproducibility without the corresponding implementation and configuration.
• The pretrained components retain their respective upstream licenses and terms.

Paper and Citation

Title: Rethinking LLM-Guided Visual Question Answering: Does Deeper Knowledge Integration Really Help?

Authors: Aneerban Saha, Vijay Kumar, Sameer Madhav D., Priyam Srivastava, and Dhruv Bansal.

Conference: Accepted at ADCIS 2026.

arXiv: 2610.17079

BibTeX

@misc{saha2026rethinkingvqa,
  title         = {Rethinking LLM-Guided Visual Question Answering: Does Deeper Knowledge Integration Really Help?},
  author        = {Saha, Aneerban and Kumar, Vijay and Madhav D., Sameer and Srivastava, Priyam and Bansal, Dhruv},
  year          = {2026},
  eprint        = {2610.17079},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CV},
  url           = {https://arxiv.org/abs/2610.17079}
}

License

This repository is marked as MIT licensed.