IEEE DCOSS-IoT-UrbCom 2026 · arXiv 2604.28048

Stable Behavior, Limited Variation: Persona Validity in LLM Agents for Urban Sentiment Perception

Personas make agents stable. They do not make them different.

Neemias B da Silva1Rodrigo Minetto1Daniel Silver2Thiago H Silva1,2
1 Federal University of Technology – Parana (UTFPR), Brazil2 University of Toronto, Canada
PDFPaperarXivAbstractDOIRecordGITCodeHFDatasetBIBCitation
Phase 1 builds 1,200 persona-driven agents from 24 profiles with 50 agents each. A solid main branch sends them to Phase 2, persona annotation of 50 urban images with Qwen3-VL 8B and thinking enabled. A dashed ablation branch runs the same images with no persona in thinking-enabled and thinking-disabled variants. Both branches feed Phase 3, a convergence and agreement analysis.
Experimental design. A balanced full factorial of gender, economic status, political orientation, and personality gives 24 persona profiles; 50 independent agents per profile annotate the same 50 PerceptSent scenes. A no-persona ablation runs the identical model and images without any persona conditioning, so the contribution of the persona itself can be isolated.

Abstract

Large Language Models (LLMs) are increasingly used as proxies for human perception in urban analysis, yet it remains unclear whether persona prompting produces meaningful and reproducible behavioral diversity. We investigate whether distinct personas influence urban sentiment judgments generated by multimodal LLMs. Using a factorial set of personas spanning gender, economic status, political orientation, and personality, we instantiate multiple agents per persona to evaluate urban scene images from the PerceptSent dataset and assess both within-persona consistency and cross-persona variation. Results show strong convergence among agents sharing a persona, indicating stable and reproducible behavior. However, cross-persona differentiation is limited: economic status and personality induce statistically detectable but practically modest variation, while gender shows no measurable effect and political orientation only negligible impact. Agents also exhibit an extremity bias, collapsing intermediate sentiment categories common in human annotations. As a result, performance remains strong on coarse-grained polarity tasks but degrades as sentiment resolution increases, suggesting that simple label-based persona prompting does not capture fine-grained perceptual judgments. To isolate the contribution of persona conditioning, we additionally evaluate the same model without personas. Surprisingly, the no-persona model sometimes matches or exceeds persona-conditioned agreement with human labels across all task variants, suggesting that simple label-based persona prompting may add limited annotation value in this setting.

Key Results

Agreement with human labels holds up on coarse polarity and falls away as the sentiment scale gets finer. Removing the persona does not cost anything: on every task variant the no-persona model matches or exceeds the persona-conditioned pool.

Macro F1 against human ground truth, agreement subsets at σ = 3
Task granularityPersona agentsNo persona
Binary · 50 imagesPolarity (negative vs. rest)0.8070.862
Binary · 50 imagesPolarity (positive vs. rest)0.7970.836
Three-class · 45 imagesNegative / neutral / positive0.5880.665
Ordinal · 36 imagesFive-point sentiment scale0.3090.433
  • Behavior within a persona is stable. Across the 1,200 image × profile groups, a median 98% of the 50 agents sharing a profile pick the same label; the mean modal ratio is 0.871 [0.860, 0.881].
  • Behavior across personas is not distinct. Economic status is the only dimension with a clear shift — low-income profiles skew negative, high-income profiles positive — while gender and political orientation produce overlapping distributions.
  • Agents avoid the middle of the scale. Negative and Positive absorb 77.4% of the 59,708 annotations, while the human ground truth spreads almost evenly over all five levels.
  • Errors are adjacent, not opposite. Quadratic Cohen κ stays near 0.79 on the five-point task: the ordering is right even where the exact level is wrong.
Heatmap of 24 persona profiles by five predicted sentiment labels. Every profile places roughly 0.40 to 0.47 of its annotations on Negative and 0.21 to 0.45 on Positive, while the three intermediate labels stay at or below 0.16. Low-income profiles occupy the top of the ordering and high-income profiles the bottom.
Predicted sentiment by persona profile. Each row is one of the 24 profiles, ordered by its combined Negative and Slightly Negative share; values are the proportion of that profile's annotations, ± the half-width of a 95% Wilson interval. The columns at the extremes carry nearly all the mass, and the split between the low-income block at the top and the high-income block at the bottom is the clearest persona effect in the study.

Study at a Glance

59,708persona-conditioned annotations
1,200agents, 50 per profile
24persona profiles
50PerceptSent urban scenes

The 24 profiles are the full factorial of gender (2), economic status (2), political orientation (2), and personality (3), with 50 agents drawn per profile and every dimension perfectly balanced. Each agent annotates all 50 images with Qwen3-VL-8B, returning a five-point sentiment label, perception tags, a caption, and a justification. Convergence is measured as the within-group modal ratio over the 50 agents in each image × profile cell. Agreement with humans is evaluated against 12 PerceptSent subsets filtered at three annotator-agreement thresholds, with point estimates from a 60% annotation resample per image and 95% intervals from an image-level bootstrap. The same model and images are rerun without persona conditioning, in reasoning-enabled and reasoning-disabled variants, as the ablation.

Every annotation behind these results is released as UrbanPersona-60K on Hugging Face: the 59,708 persona-conditioned labels with their perception tags, captions, and justifications, the no-persona ablation runs, and the 1,200 persona seed profiles, under CC BY 4.0.

Responsible Interpretation

These results describe how one model behaves under persona prompts; they are not estimates of how real demographic groups perceive cities. The profiles are controlled prompting conditions built from four attribute labels, not complete identities, and synthetic personas can reproduce stereotypical associations. The negative finding is about label-based persona prompting specifically — a richer persona representation, a different model, or a different task may behave differently. Substituting agents for human annotators would require demographically matched human data to justify.

Citation

If this work informs your research, please cite the UrbCom paper.

BibTeX
@inproceedings{urbcom26-neemias,
  title={Stable Behavior, Limited Variation: Persona Validity in LLM Agents for Urban Sentiment Perception},
  author={Neemias B da Silva and Rodrigo Minetto and Daniel Silver and Thiago H Silva},
  year={2026},
  booktitle={Proc. of IEEE DCOSS-IoT-UrbCom},
  address={Reykjavik, Iceland}
}