Time budget analysis with large vision language models for automated pain assessment in horses

Pain assessment in equines is particularly challenging, as they naturally suppress visible signs of discomfort in the presence of potential threats, including humans, which often leads to their pain being underestimated or overlooked. Subtle changes in how a horse distributes its time across key behaviours such as resting or foraging can provide valuable indicators of pain, but reliably capturing these patterns requires continuous observation over extended periods, which is rarely feasible in clinical practice. To address this gap, we propose a novel, non-invasive and scalable approach that uses computer vision to derive time budgets from video. This study aimed to evaluate whether Large Vision Language Models (LVLMs) can automatically analyze equine time budgets from stall videos and, on this basis, distinguish between painful and non-painful states. We hypothesised that zero-shot LVLM-derived time budgets would be sufficiently accurate to support clinically useful pain classification by detection of behaviours including resting, lying, moving, feeding or foraging. We used 24-h stall surveillance footage from 16 hospitalised horses admitted to a university equine hospital for at least three consecutive days and scored for musculoskeletal pain using the Musculoskeletal Pain Scale (MPS). LVLMs (such as GPT-4.1-mini) processed clips in a sliding-window fashion to predict the behaviour occurring in each window; these predictions were then aggregated into clip-level time budgets, which served as input to conventional machine-learning classifiers for binary pain vs no-pain classification. The pipeline achieved an F1 score of 0.8, precision 0.8 and recall 0.8 for pain detection, despite the LVLM operating in a zero-shot regime and without task-specific retraining. These findings demonstrate that LVLMs can extract clinically meaningful time-budget information from unprocessed equine stall videos and, when coupled with simple classifiers, yield pain detection performance comparable to many supervised approaches, without additional annotation or model retraining.

University of Haifa
Jönköping University