Multimodal LLMs See Sentiment

The short version

Ask a multimodal LLM “what is the sentiment of this image?” and it does worse than if you ask it “describe this image” and hand the description to a text classifier. Describing the scene externalises the perceptual cues — decay, isolation, leisure, neglect — that a vision-only classifier has to infer from pixels, and that annotators actually cite when they justify their own sentiment judgments.

How it works

The framework separates two ways of getting a sentiment label out of an image.

Task 1 — direct classification. The MLLM is shown the image and asked for a label, nothing else:

Analyze this image, and classify it as {𝓛} sentiments, do not describe the image, and select only one class.

Task 2 — the description-mediated pipeline. The MLLM is instead asked only to look:

Describe this image in details.

The resulting text is then classified by a text-only LLM — either straight out of the box (Task 2a, pre-trained, only a task head added) or after fine-tuning on sentiment-labeled descriptions (Task 2b). The best configuration of this pipeline — GPT-4o mini descriptions classified by a fine-tuned ModernBERT — is what we call MLLMsent.

The benchmark

Experiments use PerceptSent (5,000 images from Instagram, Flickr and NYC311, 92.7% outdoor) as the in-domain benchmark, and DeepSent (1,269 images from X) for cross-dataset transfer. Every image carries five independent annotator votes.

Two axes run through every experiment:

• $\sigma_l$ — annotator agreement. How many of the five evaluators had to agree for an image to be kept. σ₃ is simple dominance, σ₅ is absolute dominance. Raising the threshold shrinks the set but cleans it.
• $P_C$ — label granularity. P₅ keeps all five categories (positive, slightly positive, neutral, slightly negative, negative); P₃ merges the “slightly” classes into their poles; P₂ is binary, used only for DeepSent.

Results

Mean $F$-score over stratified 5-fold cross-validation, ± 95% confidence interval. Task 1 is the MLLM classifying the image directly; Task 2a and Task 2b classify that same MLLM’s description with a pre-trained and a fine-tuned text LLM respectively. Bold marks the best cell per task within each setup. MiniGPT-4 has no Task 1 entry: prompted for a label alone, it kept describing the image instead.

Three things fall out of this table:

1. Describing beats classifying. In every setup, the best Task 2b cell beats the best Task 1 cell — by 13.9 points at ⟨σ₃, P₅⟩ and still 8.1 points at ⟨σ₅, P₃⟩, where direct classification is already strong.
2. Fine-tuning is what unlocks it. Pre-trained classifiers on descriptions (Task 2a) are only sometimes better than asking the MLLM directly. Fine-tuned ones (Task 2b) are better everywhere.
3. Consensus matters more than granularity. GPT-4o mini + ModernBERT at P₅ goes from 47.9 at σ₃ to 72.2 at σ₅ — a 50.7% relative gain — while collapsing P₅ into P₃ helps less.

What the models actually see

The gap between MLLMs is a gap in description quality, not in classification skill.

Against the baselines

MLLMsent is compared against VADER (lexicon-based, run on the same MLLM descriptions), a ResNet CNN, and a Swin Transformer — the last two operating directly on pixels. VADER is only applicable to the three-class setups, since its rules produce negative / neutral / positive and nothing finer.

The margins in the abstract come from the widest of these gaps: +30.9% over VADER (77.5 vs 59.19 at ⟨σ₃, P₃⟩), +64.8% over the ResNet CNN (84.4 vs 51.2 at ⟨σ₅, P₅⟩), and +42.4% over the Swin Transformer (58.4 vs 41.01 at ⟨σ₃, P₅⟩). A post-hoc analysis — paired $t$-tests over the five folds for Swin and VADER, a Welch-type comparison for ResNet, all Holm–Bonferroni corrected across the 10 comparisons — puts MLLMsent significantly ahead in every applicable configuration ($p_\text{Holm} \le 0.0012$).

VADER is the interesting baseline here. A rule-based lexicon from 2014, given nothing but a good description, matches ResNet and Swin at ⟨σ₃, P₃⟩ and beats both at ⟨σ₅, P₃⟩. That is less a statement about VADER than about the descriptions.

Cross-dataset transfer

The strongest test: train on PerceptSent, evaluate on DeepSent, and never let the model see a DeepSent image. Every baseline below was trained and validated in-domain on DeepSent.

Ground truth

Negative

Campos et al.

Positive

MLLMsent

Negative ✓

Ground truth

Positive

Campos et al.

Negative

MLLMsent

Positive ✓

Ground truth

Positive

Campos et al.

Negative

MLLMsent

Positive ✓

Ground truth

Positive

Oliveira et al.

Negative

MLLMsent

Negative ✗

The last one is a failure, and an instructive one. The description mentions “thick white steam is billowing from the smokestack”, which appears to outweigh “the overall scene captures a blend of industrial heritage and natural beauty”. When neutral context sits beside a localised cue with negative associations, the prediction becomes sensitive to how that cue gets verbalised.

Why it works: the description as an explanation

Because the intermediate representation is text, you can read why a prediction came out the way it did — and compare it against the perception tags human annotators left behind. The examples below are GPT-4o mini descriptions under ⟨σ₅, P₅⟩.

✓ Positive

“beautiful waterfall surrounded by lush greenery” … “people are swimming and enjoying the coolness of the water” … “tranquil and inviting atmosphere”

Cues in the description nature leisure relaxation vibrant colors tourism

Human perception tags Pleasant environment Leisure/Fun/Rest Nature Colors Tourist Attractions

Ground truth Positive

✓ Negative

“overflowing garbage bins” … “the area around the bins is littered with various pieces of garbage” … “neglected or low-maintenance environment” … “sense of abandonment”

Cues in the description trash neglect abandonment urban degradation

Human perception tags Lack of Maintenance Trash Sadness

Ground truth Negative

✓ Positive

“serene coastal scene” … “beautiful shade of blue” … “peaceful solitude in nature” … “perfect for kayaking and enjoying the outdoors”

Cues in the description calmness nature sports leisure panoramic scenery

Human perception tags Pleasant environment Nature Sports Leisure/Fun/Rest

Ground truth Positive

✓ Negative

“partially destroyed and surrounded by debris” … “street littered with rubble” … “smoke and dust” … “overall atmosphere appears somber and tense”

Cues in the description destruction accident tension debris tragedy

Human perception tags Debris/Destruction Morbid Accident

Ground truth Negative

✓ Neutral

“cloudy sky” … “muted tone” … “typical day in a city” … “overall atmosphere is calm and somewhat quiet”

Cues in the description neutrality ordinary urban setting low emotional intensity

Human perception tags Everyday image Meaningless It has positive and negative elements

Ground truth Neutral

Limitations

The pipeline inherits everything wrong with its descriptions. Noisy, incomplete or hallucinated text propagates straight into the classifier, and small differences in emphasis or wording can move the final label — most visibly in ambiguous or context-dependent scenes. Sentiment labels themselves are subjective and carry annotator and cultural bias, which affects training and evaluation alike. And part of the analysis depends on a proprietary model (GPT-4o mini), whose availability and behaviour may change.

The benchmark comparison is also deliberately scoped: the goal is to characterise description-mediated reasoning under one unified protocol, not to exhaustively compare every multimodal fusion architecture in the literature.

Data and models

Everything needed to reproduce or extend the study is released on the Hugging Face Hub under CC-BY-4.0. The PerceptSent source images themselves are not redistributed.

Dataset — descriptions, labels and every result

Files follow percept_dataset_alpha{sigma}_{problem}.csv with columns id, text, sentiment, where text is the MLLM description. The release covers descriptions from six MLLMs — beyond the three studied in the paper it also includes Gemini 2.5 Flash, Phi-4 vision and Gemma-4.

from datasets import load_dataset

data = load_dataset(
    "Neemias/multimodal-LLMs-See-Sentiment",
    data_files="inputs/gpt4-openai-classify/percept_dataset_alpha5_p3.csv",
)

Models — the trained classifiers

The trained sentiment classifiers are published as fp16 safetensors, laid out by the configuration that produced them:

{caption_mllm}/{backbone}/{problem}/sigma{n}/{finetuned|not_finetuned}/
    model.safetensors
    config.json     # base model, id2label, source SHA-256, 5-fold scores

Each config.json records the 5-fold $F$-score that checkpoint achieved, so the model repo and the result CSVs in the dataset repo cross-reference each other. The paper’s headline configuration is gpt4-openai-classify/modernbert/p3/sigma5/finetuned.

Not published, because the weights were never retained: the LLaMA-3 qLoRA adapters, the Swin baseline, and a few BART σ₅ cells. Their result CSVs are in the dataset repo regardless.

Running it

git clone https://github.com/neemiasbsilva/multimodal-LLMs-see-sentiment.git
cd multimodal-LLMs-see-sentiment
uv sync

mllmsent hub pull-checkpoint openai-modernbert-p3-sigma5
mllmsent predict -- \
    --model_name modern-bert \
    --checkpoint_path checkpoints/best_checkpoint_gpt4-openai-classify_modernbert_p3_sigma5_finetuned.pt \
    --model_path answerdotai/ModernBERT-large \
    --input_file your_descriptions.csv \
    --output_file predictions.csv \
    --num_classes 3

Reproducing a cell of the experiment matrix end to end:

mllmsent caption --mllm openai --sigma 5 --resume      # Task 2, descriptions
mllmsent train openai-modernbert-p3-sigma5 --gpu 0     # Task 2b, fine-tune
mllmsent evaluate kfold -- --model openai --tasks 1 2a 2b
mllmsent evaluate stats                                # Holm-corrected t-tests

BibTeX

@misc{dasilva2026multimodalllmssentiment,
      title={Multimodal LLMs See Sentiment},
      author={Neemias B. da Silva and John Harrison and Rodrigo Minetto and Myriam R. Delgado and Bogdan T. Nassu and Thiago H. Silva},
      year={2026},
      eprint={2508.16873},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2508.16873},
}

Credits

Figures and image samples on this page are taken from the paper. PerceptSent images are distributed under CC BY 4.0; DeepSent images come from the DeepSent repository under the MIT License. Faces were blurred in the original figures. Code and Hub artifacts are CC-BY-4.0; the base models keep their own licenses.