A Transformer-Based Multimodal Framework for Tactical Decision Analysis in Team Sports with Application to Football

Summary

To improve tactical decision categorization in football, this study suggests a Transformer-based multimodal fusion model that incorporates visual, location, acoustic, and contextual data. The model achieves near-real-time performance on a 500-sequence multimodal dataset, outperforming CNN-LSTM, BiLSTM-Attention, and GNN baselines in terms of accuracy and pressure-induced misclassification.

Abstract

Rapid and accurate tactical decision-making is required in modern football. However, traditional approaches for the assessment of tactical decision-making have poor ecological validity and do not easily scale. In order to address these challenges, this article introduces a Transformer-Based Multimodal Fusion model that incorporates player positioning, video, audio, and contextual metadata to classify real-time tactical decisions. Experiments were conducted with 40 male players and a dataset comprising 500 sequences of multimodal plays. The 40-player dataset refers to controlled laboratory-style decision-making experiments used for initial validation and a reliability assessment. Then, 500 multimodal sequences were extracted from extended match simulations and real-game recordings to provide the larger dataset used in training and testing the multimodal transformer model.

It processes inputs in five stages: data acquisition, preprocessing, feature extraction, transformer-based fusion, and decision classification. Compared to the baselines of CNN-LSTM, BiLSTM-Attention, and GNN, the proposed approach improves the accuracy of decision prediction by 28% and reduces misclassification caused by pressure by 41%, with low inference latency of 52.6 ms, making it suitable for near-real-time applications. The generalizability of findings across more diverse tactical contexts and to wider athlete demographics is also limited by the relatively small size and homogeneity within the sample population of young male players from a single region. These results emphasize the contribution of transformer-based multimodal fusion toward automated tactical decision analysis and point out the need for its further validation in more diverse and large-scale match situations.

Introduction

Team sports, such as football, require quick tactical decisions in dynamic and uncertain environments. Players must constantly read visual information from the ball, teammates, and opponents, and respond within fractions of a second to ensure a competitive advantage. Tactical decision-making in football is highly dependent on rapid perception of player movements, ball trajectory, and situational cues. Although general perceptual-cognitive skills, such as reaction time and selective attention, do indeed inform performance, recent research has emphasized data-driven, context-aware models that can approximate real in-game tactical cognition^1,2,3,4,5. Reaction time and perceptual capabilities, such as cognitive abilities, also play a crucial role in performance^6,7,8. Recent research in sports cognition has established that tactical decision-making relies not only on perceptual skills but also on the athlete's ability to integrate dynamic spatial information, opponent pressure, and contextual game cues in real time. Research into perceptual-cognitive expertise suggests that tactically more literate players recognize cues more quickly, pick up information in a more efficient manner, and exhibit more stable decision-making patterns under pressure. Such findings support the need for computational models that can capture the complex, multi-layered nature of sports decision-making processes.

Classical methods for evaluating tactical decision-making often employ controlled laboratory tasks and subjective expert judgments, which impose significant constraints on ecological validity and scalability. Recent advances in deep learning offer the potential for automating the process through the use of multimodal sources of data-such as video, positional tracking, and contextual match metadata-within temporal and cross-modal dependency-capturing architectures. Despite this advancement, most current models are still based on Convolutional Neural Networks-Long-Short Term Memory or Bidirectional Long-Short Term Memory (CNN-LSTM or BiLSTM) models, which have limited capacity to model long-range dependencies between modalities. This shortcoming is the driving force behind the development of transformer-based multimodal fusion models, which are better equipped to model tactical decision-making more comprehensively and stably in football scenarios. In the oppositional team sport of football, for instance, players must rapidly evaluate information about the ball, including their teammates, opponents, and position on the field. Before deciding on the best course of action based on their abilities, the coach's directions, and the circumstances of the game^9,10. Due to the accessibility of organized datasets and defined tactical components, this study focuses exclusively on football; however, tactical principles for making decisions are applicable to numerous team sports.

Indeed, previous studies indicate that CNN-LSTM and BiLSTM cannot capture very long-range temporal dependencies by using a fixed receptive field or being restricted by sequential gating^11,12,13. Similarly, foundational benchmark work in action recognition and multimodal temporal modeling reveals that LSTM-based model performance decreases with increasing sequence length or when contextual cues occur several frames apart, which can limit ecological validity in dynamic sports settings. Moreover, Graph Neural Network (GNN)-based models are powerful tools for spatial interaction modeling, but they have consistently underperformed in scenarios that require fine-grained temporal reasoning or in cases where contextual pressure cues rapidly shift across time^14,15. In contrast to these approaches, more recent transformer-based frameworks demonstrate their performance superiority in sports analytics, human activity recognition, and video-language tasks through the joint integration of long-range temporal cues, spatial relationships, and contextual signals in a single forward pass, enabled by global self-attention. Such benchmark findings justify the selection of a transformer-based architecture as the backbone for the proposed model in the current study.

Enhancing tactical knowledge and comprehending tactical principles has been demonstrated to be crucial for resolving game constraints and issues^16,17. Football players learn a variety of game elements, including penetration, offensive protection, movement, time, limitation, defense protection, balancing, and focus, through technical-tactical training¹⁸. As players get more football experience, their understanding of technical and tactical fundamentals should increase. Players may therefore be able to discern pertinent signals more easily, facilitating appropriate decision-making for every game rule in a particular scenario¹⁹. Because of this, players can utilize their motor efficiency and decision-making skills to complement movement patterns with specialized talents.

The author used multiple attribute decision-making to analyze the fitness benefits of exercise and habits, illustrating the efficacy of qualified training capacity in physical education among college students. Effective physical activity, developing healthy fitness habits, and advocating for reforms in the way sports are portrayed in educational institutions are all given considerable weight. The Intuitionistic Fuzzy Weighted Heronian Mean (IFWHM) with fuzzy number assistance is used for effective decision-making. The outcome supported university students' cognitive success and demonstrated a more accurate evaluation of their physical well-being issues. In order to detect the latent links in the data, the researcher²⁰ used the augmented Apriori method. The research's conclusions are pertinent to assessments of students' physical health in higher education. The first outcome is determined by evaluating the fatigue of different orientations according to their frequency of occurrence.

Previous AI-based research in sports analytics has focused on computer vision tasks, including player detection, tracking, and recognizing group activities. Recent weakly supervised and detector-free architectures have unveiled the importance of multimodal and context-aware modeling in understanding tactical behaviors. These advances necessitate the integration of multimodal deep learning to classify tactical decisions in football. Using latent feature vectors produced from the utility matrix via matrix factorization, it computes the Cosine similarity between users and users or items and items in this article.

Recent research in sports analytics has shifted towards multimodal and context-aware deep-learning frameworks that combine video, positional data, and contextual cues to interpret tactical behavior. Various transformer-based approaches have shown strong capability in modeling long-range temporal structure and cross-modal relationships in sports environments, including the following: MM-ViT for multimodal action recognition, unified contrastive fusion transformers for sports-context reasoning, and cross-attention-driven event detectors^21,22. These approaches pinpoint that tactical decisions are best represented using architectures that capture the interactions between players, space, motion cues, and contextual information rather than single-modality CNN-LSTM models. Given this direction, the proposed study will also utilize a transformer-based multimodal fusion framework to jointly process visual, positional, and contextual signals for near real-time modeling of football tactical decisions.

Previous models for football analysis were typically built around the CNN-LSTM or BiLSTM architecture, which capture local temporal patterns but struggle with long-range dependencies across modalities. Transformers fill this gap by applying global self-attention, allowing a model to relate player movements, ball trajectories, and situational cues across extended time windows-key capabilities for robust tactical decision analysis. In order to solve the problem of sparsity in rating data for recommender systems, the author^23,24 suggested a Review-Based Matrix Factorization technique that combines review-based collaborative filtering and rating imputation.

Nevertheless, their efficacy, these methods' scalability and applicability are severely constrained by their reliance on bounding boxes for inference and the vast requirement for data labelling. One method for resolving this problem is to use bounding box labels to train player detection and group activity recognition simultaneously^25,26,27,28. Although actor-level bounding box annotations are still needed for training data, the proposed method calculates the bounding boxes of players during inference. The Weakly Supervised Group Activity Recognition (WSGAR) approach, which does not require actor-level labels for training or inference, aims to alleviate the annotation burden. After generating bounding box suggestions using a pre-trained detector on an external dataset, they learned to filter out irrelevant detections. Recently²⁹, the researchers presented a detector-free approach for the WSGAR challenge that uses token embeddings to generate partial contexts that collect player information.

Within sports analytics, multimodal and transformer-based architectures have recently gained significance because they can capture long-range temporal patterns and integrate heterogeneous data streams^30,31. Variants of transformers have been applied to forecast players' movements, perform multi-view tactic analysis, and recognize action intent, demonstrating outstanding temporal reasoning compared to the CNN-LSTM model. Multimodal fusion strategies, which combine video, position, and contextual clues, have delivered encouraging early results in real-time tactical modeling and highlighted the importance of cross-attention and sequence-to-sequence learning in comprehending in-game decision dynamics.

Transformer-based multimodal frameworks have recently demonstrated outstanding performance in various sectors where integrating diverse data streams is necessary. For example, ViT-based multimodal fusion models have been used to jointly interpret video, pose, and audio information employing cross-attention processes for context-aware event prediction, player tracking, and action recognition^32,33. Furthermore, the MM-Former and Perceiver-based architecture has outperformed recurrent and convolutional models in fusing spatial-temporal cues, as well as in long-range temporal reasoning, in sports analytics and human activity analysis³⁴. These results support the use of a Transformer-Based Multimodal Fusion architecture in this study by indicating that transformers are capable of capturing fine-grained interactions between modalities through global attention³⁵. Transformers are particularly suitable for tactical analysis, as their global attention lets the model relate visual cues and positional data and contextual signals across longer temporal windows, capabilities that CNN-LSTM and BiLSTM models lack.

Previous studies have mainly used manually shaped and authenticated assessments to evaluate performance speed and decision accuracy in prearranged activities, such as driving and passing, despite the growing interest in assessing tactical decision-making in team sports^36,37,38. These frameworks have boundaries in terms of scalability, objectivity, and appropriateness for dynamic, real-world circumstances, despite providing insightful data about player behavior and decision-making excellence³⁹. Computerized data-driven approaches that can record and understand the multimodal nature of tactical decision-making, which involves complex associations among player actions, visual signals, audio cues, and game settings, are not combined into present methods^40,41,42. Additionally, multifaceted AI structures that can switch between background and temporal relationships are often overlooked by these methods. The growth of a Transformer-based multimodal fusion framework that utilizes rich, real-time data sources, such as video footage and positional tracking, and incorporates decision-making processes in team sports is clearly lacking. Closing this gap can significantly enhance the tactical penetration, scalability, and independence of decision-making in high-performance sports environments. In contrast to current CNN-LSTM and BiLSTM methodologies, this transformer-based fusion consciously models cross-modal attention between visual, spatial, and contextual information. Such novelty promotes a more dense tactical representation and helps minimize misclassification during pressure scenarios by more than 40%.

The primary objective of this study is to develop a Transformer-Based Multimodal Fusion framework for evaluating tactical decision-making in football. While the proposed framework can be generalized to other team sports, this study focuses on football due to its tactical complexity and the availability of structured datasets. The system enables structured assessment of tactical accuracy and response timing in controlled, isolated football tasks, based on expert evaluations. This work addresses the limitations of manual evaluation and static testing environments by integrating multimodal data sources, including positional tracking, video footage, audio signals, and contextual game information.

Employing transformer-based attention processes, the proposed framework continually learns multimodal links, allowing for a unified and context-aware interpretation of players' real-time decision-making methods.

This determination's primary objective is to promote intelligent performance evaluation in sports by providing coaches and analysts with a deeper understanding of player thinking and team dynamics through data-driven insights.

The proposed framework will continuously learn intermodal relationships by leveraging transformers' attention mechanisms, enabling it to obtain a unified understanding of players' decision-making strategies in real-time games.

It thereby equips the system with an ability to give actionable insight into tactical behavior that is otherwise hard to characterize with standard evaluation methods.

The goal is to enhance interpretability and provide coaches and analysts with more useful information.

The main contribution of this work is given below:

This study employs a Transformer-based multimodal fusion design to present a novel deep learning model for tactical decision-making analysis in team sports.

Through integrating visual, positional, and contextual streams of data extracted from real-game footage, this approach significantly extends the scope of the simple evaluation methods introduced in the base study, which focused on solo passing and driving actions.

The multimodal encoder collects sophisticated spatiotemporal relations by integrating player tracking data, visual frames, and contextual match events using attention mechanisms.

Experiments on a benchmark football dataset revealed that this model outperformed traditional baselines such as CNN-based fusion methods and LSTM.

Compared to CNN-LSTM, BiLSTM-Attention, and GNN baselines, tha suggested transformer-based multimodal fusion architecture shows significant gains.

Protocol

The study included 40 male football players aged 18-24 years (mean = 20.1 ± 1.2), recruited from four amateur and semi-professional clubs within one regional league. All participants had at least three years of competitive experience and were currently training in a structured environment. Data collection was performed in two controlled settings: first, in standardized training-ground drills that simulated tactical pass/drive decision scenarios; second, in prolonged sessions of match simulation from which multimodal sequences were extracted. All procedures were approved by the Institutional Ethics Committee of Beibu Gulf University (Approval No: BG-PE-2023-117), and written informed consent was obtained from all participants prior to data collection. International and institutional ethical standards were adhered to in conducting this investigation. The Beibu Gulf University Institutional Ethics Committee examined and authorized all procedures involving human subjects (Approval No: BG-PE-2023-117). Prior to data collection, each participant provided written informed consent, and all collected data were anonymized before analysis.

This work seeks to automate and enhance the investigation of team sport tactical decision-making through a Transformer-based Multimodal Fusion architecture. It uses a variety of data sources, including video, audio, spatial location, and player metadata to create a robust real-time predictive model. Figure 1 shows the architecture of the proposed method. The proposed work consists of five stages. The stages are described below:

Figure 1. Architecture of the proposed method. Schematic of the model combining multimodal inputs and classification components for tactical decision-making. Please click here to view a larger version of this figure.

Data acquisition and preprocessing

Data from various modalities are gathered from recordings of matches, such as video feeds, audio commentary, GPS/positioning information, and player performance indicators. Each modality undergoes preprocessing in the form of techniques such as frame extraction (for video), noise filtering (for audio), and normalization (for sensor readings), ensuring synchronization at time steps.

Video frames were extracted at 25 fps, downsampled to 224 × 224 resolution. Audio signals were processed with a band-pass filter, ranging from 300 to 3,400 Hz, to eliminate the majority of environmental noise. Positional tracking data from GPS sensors were recorded at 10 Hz and interpolated using linear timestamp-based interpolation, ensuring they align with the 25 Hz video timeline. All inputs were temporally aligned using shared timestamps, and their scales were normalized with z-scoring.

Feature extraction

To gather the spatiotemporal information of video data, 3D CNNs are used. The 3D CNNs handle both spatial and temporal information across video frames, making the model capable of detecting movement patterns, understanding group behavior, and extracting motion-based features. This operation generates a dense feature representation per action sequence, which is used as the visual input to the model. Each video input clip contained 16 consecutive frames sampled at a stride of 2. Random cropping, horizontal flipping, and brightness normalization were applied during training. The 3D CNN was optimized using Adam (lr = 0.0001), batch size 16, and cross-entropy loss.

Embedding and alignment of multimodal inputs

After that, embeddings from different modalities are integrated with the 3D CNN's outputs. A multimodal transformer model architecture is employed, with various encoder branches handling each modality. Cross-attention layers are employed for modality fusion and alignment, enabling the model to capture interdependencies (e.g., between player locations and ball movement, and context from commentary). This fusion allows for an enriched contextual understanding of current tactical choices. Also implemented all embeddings in PyTorch. Video features were projected linearly from 1,024 to 512 dimensions, and audio and positional features were projected to 256 and 128 dimensions, respectively, unified to 512 dimensions before temporal alignment.

Transformer-based multimodal fusion

A Multimodal Transformer is employed to combine the features extracted from all sources. Self-attention mechanisms in the transformer enable the model to Learn interdependencies between modalities (e.g., visual + audio), capture temporal flow and context, and highlight tactically important frames and events. The outcome of this step is a combined representation that captures the tactical intent and situational context of each decision made. The multimodal transformer consisted of six encoder layers, each with eight attention heads. Attention matrices were computed using scaled dot-product attention. Attention and feed-forward layers also included dropout with p = 0.1 to avoid overfitting.

Tactical decision classification and evaluation

Lastly, the fused representation is fed as input to a classification layer or decision analysis block, which is used to forecast the tactical decision type, assess decision quality, and optionally generate explainable insights for coaches through attention heatmaps or activation maps. The output of this stage provides a quantitative and qualitative evaluation of a team or player's decision-making abilities during match play.

The classification head utilised a three-layer MLP with ReLU activation followed by a softmax layer. The model was trained on a 70/15/15 split with 10-fold cross-validation. Metrics used were accuracy, precision, recall, F1-score, latency, and AUC.

A simplified flowchart, summarizing the five stages, provides an immediate visual overview of the sequential process, as shown in Figure 2. This figure concisely represents the complete research pipeline, comprising data acquisition, preprocessing, feature extraction, multimodal fusion, and tactical decision classification, followed by detailed descriptions of each stage.

Figure 2. Overall workflow of the research process. Overview of the research pipeline, from data collection and preprocessing to feature extraction, model training, and evaluation. Please click here to view a larger version of this figure.

Figure 3 displays the overall workflow of the suggested research process. The process is initiated by Stage 1: Data Acquisition and Pre-processing, where multimodal data sources, such as video, audio, positional tracking, and contextual metadata, are gathered and synchronized. During Stage 2: Feature Extraction, 3D Convolutional Neural Networks (3D CNNs) are utilized to extract spatiotemporal information from video streams, resulting in dense visual representations of player actions and tactical flow. Stage 3: Embedding and Alignment of Multimodal Inputs combines audio, video, and positional features into a shared latent space, with temporal alignment and cross-modal alignment. These representations are subsequently aggregated in Stage 4: Transformer-based Multimodal Fusion, where cross-modal and self-attention mechanisms allow the model to learn interdependencies across modalities and derive deeper contextual insights into tactical decisions. Lastly, in Stage 5: Tactical Decision Classification and Evaluation, the combined representations are fed into a classification layer to detect tactical decisions, such as pass, drive, or hold. Meanwhile, performance metrics are employed to estimate decision-making accuracy and response time. This flowchart gives a clear overview of the step-by-step approach, complemented by the elaborate textual descriptions outlined in the following sections.

Figure 3. Sequential processing pipeline. Depicts the step-by-step flow from data acquisition to tactical decision classification and model output. Please click here to view a larger version of this figure.

Data acquisition and preprocessing

To enable high-level tactical decision-making analysis in team sports via a multimodal fusion pipeline using Transformer, a structured and high-fidelity data acquisition and preprocessing setup was formed. The setup involves fusing multiple data modalities, including video recordings, player positional tracking, audio signals from player-coach interactions, and contextual metadata such as match phase, team structure, and possession areas.

The study sample consisted of 40 males aged 20 and above, all of whom were actively involved in regional amateur and semi-professional football leagues. Each of them was recruited from four competitive clubs that had a structured training program. The athletes' mean age was 20.1 ± 1.2 years, with a mean height of 177.5 ± 3.1 cm and a mean weight of 72.6 ± 2.9 kg. They had been playing for their respective clubs for a mean of 6.8 ± 1.5 years. All the participants also had a minimum of three years of competitive experience and were found to be tactically competent.

To ensure statistical power, the sample size was approximated using a 95% level of reliability (Z = 1.96) and a 5% significance level with an expected Intraclass Correlation Coefficient (ICC) of 0.96 and a 95% confidence interval to reflect near-perfect agreement. This is consistent with the objective of validating the reliability and consistency of multimodal decision-making data collected²⁷.

To ensure reproducibility, acquisition settings and technical specifications were standardized across sessions. Video data were recorded at 1,080p resolution and 25 frames per second using GoPro Hero4 cameras with a wide field-of-view setting to capture the full tactical space. Each recording was stabilized and mounted at a fixed height of 2.5 m. Audio signals were captured using an external field microphone at a 44.1 kHz sampling rate (mono) and subsequently filtered using a 300-3,400 Hz band-pass filter to remove environmental noise. Positional tracking data were collected using wearable GPS sensors operating at 10 Hz, with an average manufacturer-reported positional accuracy of ±0.5 m. All modalities were synchronized using shared timestamps and resampled to a common 25 Hz timeline through linear interpolation.

Preprocessing then included the following: video downsampling to 224 × 224 resolution, frame-sampling of 16 consecutive frames with a stride of 2, random cropping, horizontal flipping, brightness normalization, audio normalization, and filtering out noise, and positional and contextual variable z-score normalization.

The preprocessing script is arranged in the following order for reproducibility: (i) frame extraction; (ii) audio filtering; (iii) GPS interpolation; (iv) modality resampling to 25 Hz; (v) z-score normalization; and (vi) integrity tests for corruption, occlusion, and dropout. Batch-processing scripts are used to automatically complete each stage.

To keep data reliable, the criteria for quality control and exclusion included: i) sequences with >20% player occlusion, ii) GPS dropout exceeding 200 ms, iii) corrupted or clipped audio, and iv) severe motion blur or desynchronization between modalities. A total of 17 sequences (3.4%) were excluded for these conditions. Table 1 shows the dataset description.

Attributes	Details
Sport	Football (Soccer)
Participants	40 male football players
Level of Play	Federated football players with ≥5 years of experience
Test Type	Passing and driving (ball control) decision-making and execution test
Test Setting	Standardized football pitch setup, marked zones, fixed positions
Measurement Tools	GoPro Hero4 cameras, Kinovea software, stopwatch timing
Data Collected	Execution Time (ET), Decision-Making (DM) Accuracy, Number of Correct Actions
Test Procedure	Players responded to visual stimuli from coaches and executed passes or drives
Trial Repetitions	10 trials per player (5 passes, 5 drives)
Evaluation	Experts rated DM; ET measured using timestamps/video
Outcome Variables	Reaction time, correct decision count, technical execution score

Table 1: Dataset description. Details participant demographics, testing procedures, measurement tools, and outcome variables.

In the present study, two related but distinct datasets were used. The first dataset comprised 40 players who participated in controlled pass/drive decision-making tasks, primarily used to establish baseline reliability and validate the basic decision measurement procedure. The second dataset includes 500 multimodal tactical sequences collected from extended match simulations and real-game recordings. These sequences constituted the main training and testing dataset for the transformer-based fusion model, allowing evaluation under more ecologically valid conditions.

Although the dataset comprises 500 multimodal sequences, stratified sampling ensured balanced representation across tactical actions (Pass, Drive, Hold). Data augmentation approaches, such as temporal cropping and sequence shuffling, were also utilized to enhance variability and mitigate model bias during training.

It is worth noting that the 40-player controlled dataset was used in the initial model validation and testing of reliability, while the larger collection of 500 multimodal sequences was compiled from prolonged match recordings and organized training sessions to assess the scalability and ecological validity of the framework. Baseline reliability was established using the smaller dataset, while the larger dataset facilitated large-scale model training and testing.

Dataset description

The experiment recruited 40 male football players, each with at least five years of federated playing experience, to ensure that the sample population had basic technical and tactical competencies. The data gathering took place in a standardized football pitch arrangement where predetermined zones and playing positions were assigned to ensure identical testing conditions. During experimentation, participants had to respond to visual cues from a coach, either to pass or drive the ball, replicating tactical decisions made in an actual game. Each participant completed a total of 10 trials, five passing and five driving. These movements were captured with GoPro Hero4 cameras, and performance data were measured using Kinovea video analysis software alongside manual stopwatch time to capture execution time (ET). The main data source gathered was: execution time, the quality of the DM as evaluated by experienced coaches, and the quantity of correct actions executed. These factors provided a quantitative foundation for analyzing how quickly and effectively players were in making and carrying out tactical decisions under controlled stimulus-response conditions. The dataset produced is a structured and labelled resource well-suited to the task of creating benchmarks or training intelligent systems with the goal of modelling tactical thinking and motor responses in team sports contexts.

The sample is restricted to young male players from one regional league, which may limit its generalizability across age groups, female athletes, or other cultures. Future studies will attempt to draw more representative and diverse samples.

Another limitation is the sample size of 40 young male players from a single regional league. While the homogeneous sample contributed to internal validity and reduced variation in performance due to age, gender, and tactical background differences, it also limits the generalizability of findings to a wider population. The tactical patterns the model learns may thus reflect region-specific or gender-specific playing styles rather than universal decision-making behavior. For this study, stratified sampling and data augmentation were employed to enhance the variability of the dataset at hand; however, larger-scale research studies involving female athletes, youth players, professional players, and participants across multiple tactical cultures are needed to confirm the robustness of the model across diverse football environments. These results demonstrate strong promise but need further validation on more diverse and larger datasets before wider generalization can be claimed.

To reduce subjectivity, three professional football coaches independently annotated the tactical choices. Inter-rater reliability was estimated using the intraclass correlation coefficient (ICC = 0.96, CI 0.94-0.98), indicating near-perfect agreement. Consensus discussion resolved disagreement. For multimodal data, preprocessing entailed temporal synchronization between modalities (25 fps video and 10 Hz sensor data) and z-score normalization of features to allow comparability across modalities.

Inter-rater reliability was quantified using a two-way random-effects intraclass correlation coefficient (ICC [2,3]), as it is suited for assessing absolute agreement between multiple raters. The ICC of 0.96 (95% CI: 0.94-0.98) indicates near-perfect agreement according to the commonly used thresholds and further establishes that the tactical decision labels used for training were highly dependable. Reliability was computed across all 500 multimodal sequences, ensuring that both controlled and match-simulation contexts were duly and consistently annotated.

Annotation procedure and labeling protocol

All multimodal sequences were annotated using a structured, three-stage protocol. First, three licensed football coaches independently reviewed each video-positional sequence using a timestamp-synchronized annotation interface (Kinovea + custom tagging spreadsheet). Annotators labeled the tactical decision category (Pass, Drive, Hold), contextual cues (pressure zone, passing lane availability), and event timestamps. Second, a disagreement-detection script automatically identified cases of ambiguity or conflict, which were then reviewed in a joint consensus meeting. Third, to ensure consistency in event boundary marking and temporal alignment across modalities, a final pass was conducted by an independent senior analyst. This process, for the purposes of later model training, ensured that every annotation was traceable and of the best quality.

Preprocessing

Whereas the present study was conducted with 500 multimodal sequences, the preprocessing pipeline was, by design, set up for large-scale datasets. All modalities underwent automated scripts that, in parallel, batch-extracted video frames, resampled audio, interpolated positional data, and applied z-score normalization. The presence of modular architecture allows each modality to be independently preprocessed on separate GPU/CPU threads. This enables the high-volume ingestion of match footage. This ensures that the workflow can easily be scaled to full-season datasets without manual intervention and makes the framework suitable for real-world deployment in professional analytics environments.

To properly examine tactical decision-making in team sports, raw data from different modalities-e.g., video clips, audio signals, and positional recordings-need to be pre-processed and aligned. The multimodal input illustration is

   (1)

Here, V is denoted as series of video characteristics extracted from CNN encoder.

   (2)

Here, A is an audio characteristic is encoded by wave2vec.

   (3)

P represents the coordinate position of a player at time ti.

To deal with asynchronous sampling rates, each modality is resampled or interpolated to a shared timeline:

   (4)

Here , is a synchronized characteristic f_t , is rate of combined frame, X_m is an initial indication.

For uniform scale across modalities, all feature vectors are z-score normalized:

   (5)

μ_X and σ_X denote the mean and the standard deviation of the feature dimension in the training set.

While the primary concerns are pass/drive decisions, audio channels (e.g., player/coach communication, environmental game cues) were added to account for actual-match situations where speech signals affect tactical reaction. Hyperparameters (e.g., learning rate, epochs) were chosen through grid search and existing evaluations in multimodal learning, striking a balance between convergence stability and computational efficiency.

The hyperparameters of the suggested framework-learning rate, batch size, and training epochs-were chosen through a well-planned systematic grid search to achieve both convergence stability and computational efficiency. The choice of a learning rate of 0.0001 with the Adam optimizer was found to provide stable updates of the gradient without oscillations, while batch sizes were optimized to balance GPU memory capacity against training throughput. The setting of 50-100 epochs was used to permit adequate learning without overfitting, as confirmed by validation loss tracking and 10-fold cross-validation. These values were not set arbitrarily but were guided by earlier testing in multimodal learning and adjusted by experimental trials on the current dataset. This rigorous process guaranteed that the selected hyperparameters facilitated stable model training and reproducible improvements in performance over baseline approaches.

Although the classification task focuses on pass/drive/hold decisions, audio information was intentionally included since real tactical behavior in football is strongly influenced by player-coach communication, teammate calling cues, pressing signals, and environmental sounds (e.g., ball contact, opponent approach). These cues frequently precede or co-occur with decision-making and form part of the natural perceptual environment of football players. Preliminary ablation experiments showed that excluding audio reduced recall by 3.8%, which indicated that even subtle acoustic cues contribute to context awareness. For this reason, audio was retained to preserve ecological validity and improve the model's ability to generalize to real match conditions.

Indeed, in further support of these hyperparameters, an internal sensitivity analysis was conducted by systematically varying the learning rate from 1e-5 to 5e-4, the batch size from 8 to 32, the number of transformer layers from 4 to 8, and the number of attention heads from 4 to 12. The chosen configuration, with a learning rate of 1e-4, a batch size of 16, and a 6-layer encoder with 8 attention heads, continuously yielded stable convergence with the lowest validation loss, minimizing overfitting across 10-fold cross-validation. Larger batch sizes resulted in gradient instability, while smaller batches increased the noise and delayed the convergence. Similarly, transformer models with more than 8 heads did not show any performance increase, but they did considerably increase computation time. The obtained results justify the final settings of hyperparameters used in this study.

Feature extraction using 3D convolutional neural network

In the proposed Transformer-Based Multimodal Fusion framework for team sport tactical decision-making, the 3D Convolutional Neural Network (3D CNN) is responsible for extracting spatial-temporal features from raw video clips.

In contrast to 2D CNNs, which consider only spatial dimensions (width W and height H), 3D CNNs also account for the temporal dimension R, enabling them to detect motion dynamics and sequential patterns that are crucial for understanding team behaviors, such as driving and passing in football.

The 3D-CNN²⁸ is first proposed to combine the spatial and temporal information of video data. A 3D kernel is used in a 3D convolution algorithm to a cube made up of frames with part of the columns stacked. The 3D convolution can be written in Eqn (6)

   (6)

Where: is the output feature at location (x, y, z) on the jth feature map of the lth layer. is the weight of the kernel at location (h,w,r) from the mth input feature map. is the input at spatial-temporal offset from the preceding layer. b_lj is the bias. f(.) is the activation function (usually ReLU). M is the number of input feature maps from the preceding layer. This equation allows the 3D CNN to combine the spatial (height and width) and temporal (frame sequence) information in parallel.

Figure 4. 3D CNN processing architecture. Illustrates how spatiotemporal features are extracted from video sequences using 3D convolutional operations. Please click here to view a larger version of this figure.

Figure 4 shows the working process of a 3D CNN. The pipeline starts with the input phase, where unprocessed video streams of a football game are fused with organized data, including player locations, game statistics, and contextual metadata. This multimodal information preserves both visual dynamics and situational context, which are critical for team tactics analysis. Then, the video stream is routed through a 3D Convolutional Neural Network (3D CNN). In this case, a series of 3D convolutional layers is used on the input frames. The layers convolve along the spatial dimensions (height and width) and the temporal dimension (frames), enabling the network to learn motion patterns, player interactions, and sequence-based tactics, such as passes or drives. Subsequently, a feature cube is formed that has dense spatiotemporal features. This cube is subjected to pooling operations to reduce the dimension while preserving significant features. Upon pooling, the feature volume is flattened into a 1D vector, providing a concise representation of the tactical situation. This feature vector is subsequently input to a fully connected Deep Neural Network (DNN). The DNN is the decision-making block, which processes the fused and extracted features to make tactical decisions, such as passing, shooting, or holding. The output of the network is the final tactical decision taken by the system based on the actual game situation in real time as well as learned patterns of strategy.

Embedding and alignment of multimodal inputs

After the feature extraction, the three features, such as audio, video, and statistical position of the player, are given as an input modality.

To input these into a transformer and then feed each of them through a learnable embedding function:

   (7)

where f_3DCNN learns spatiotemporal features of each time slice V_t and maps it to a d-dimensional latent space.

   (8)

where W_P and b_P are learnable weights.

   (9)

All embeddings are aligned by the temporal dimension T. If the modalities have disparate frequencies, it uses interpolation or down sampling:

   (10)

Now, all modalities are synchronized as:

   (11)

In order to preserve temporal order in the transformer, also introduce positional encodings to every vector:

   (12)

Where  represents the default sinusoidal or learnable positional encoding.

Each modality is operationally encoded using a PyTorch linear layer, concatenated into a 512-d vector, and then fed into the transformer input sequence following positional encoding. This guarantees that a unified embedding is prepared for cross-attention at each time step.

The Final tensor is

   (13)

is fed into the Transformer Encoder, where the self-attention and cross-modal attention mechanisms enable the model to reason jointly across all modalities for tactical decision-making.

Transformer-based multimodal fusion

In the proposed approach, the low-rank matrix factorization method is employed to integrate the acquired multimodal data vectors. The high-dimensional issue that occurs when tensors are fused directly is addressed by this method, which uses a low-rank decomposition factor for tensor fusion²⁹. Each modality is initially expressed as a vector, and dimensionality reduction that preserves significant interactions is achieved using low-rank decomposition. The fusion tensor Z_M among M modalities is built as:

   (14)

Here: is the m-th modality's low-rank factor, is the outer product, and r is the decomposition rank.

The fusion vector h is then calculated as:

   (15)

When two modalities are utilized alone, the reduced fusion is:

   (16)

This produces a joint multimodal representation vector h that models cross-modal interactions at a latent level.

Following the acquisition of the low-rank fused vector h, the Transformer module models dependencies and aligns semantic representations between modalities. A cross-modal attention is specially designed to enable one modality to pay attention to another:

   (17)

Where Q_m is the modality m query, K_n,V_n are the key and value vectors in modality n, d_k is the dimension of the key vectors. This setup facilitates modality-specific attention flows, allowing each category of input to be processed relative to others (e.g., synchronizing player motion with game context and video indicators).

The cross-attended vectors are stacked and classified through a Multilayer Perceptron (MLP). The output is a tactical decision label (e.g., pass, drive, hold), cross-entropy loss optimized for end-to-end training.

Figure 5. Transformer-based multimodal fusion architecture. Shows how visual and sensor modalities are integrated via a transformer encoder to enhance feature representation. Please click here to view a larger version of this figure.

The multimodal fusion block, as demonstrated in Figure 5, combines visual and positional inputs. It is a necessary design since tactical decisions require spatial awareness and movement dynamics, which cannot be represented by a single modality.

Figure 5 shows the working structure of Transformer-based Multimodal Fusion. This design combines diverse data video, spatial coordinates, and game contextual data into a single channel to predict tactical actions such as passing, driving, or holding. The approach begins with inputting videos, which are vetted using a 3D Convolutional Neural Network (3D CNN). This system identifies both spatial and temporal structures in video, enabling it to capture dynamic player activity and communication over time. Meanwhile, contextual and positional information are conveyed by embedding layers, transforming structured inputs into dense vector representations. The contextual, positional, and video streams can then have their outputs fused by a Low-Rank Fusion module. The approach effectively obtains cross-modal correlations through fusion space decomposition, minimizing computational complexity while preserving essential relationships among modalities. Finally, the fused multimodal representation is processed by a Transformer Encoder with cross-modal attention. This procedure enables the model to focus on the right features across modalities and time steps, enhancing its understanding of strategic game behavior. Finally, the encoded output is passed through a concatenation and a Multi-Layer Perceptron (MLP) block that maps the learned representation to some tactical decisions. The model output classifies player actions such as "pass," "drive," or "hold" to enable intelligent, data-driven examination of team tactics.

In the suggested Transformer-Based Multimodal Fusion tactical analysis system for team sports, the last phase is tactical decision classification and assessment, which converts learned multimodal representations to decision-actionable labels. Having fused video, positional, and contextual features by low-rank fusion and fine-tuned them using cross-modal attention in the Transformer encoder, the resulting feature vectors are fed into a Multi-Layer Perceptron (MLP) classifier. Every decision class is represented with a softmax activation function, yielding probability scores for all possible actions. The most probable class is chosen as the model's predicted decision. The model is trained on a cross-entropy loss function, which rewards correct predictions and causes the network to learn to differentiate between similar differences in tactical situations. Additionally, time-domain evaluation could be considered in order to confirm responsiveness in real-time or near-real-time game situations, ensuring that the classifier is accurate and also not wasteful of time under game-like time constraints. Table 2 shows the Model Architecture and Hyperparameters.

Component	Specification
Transformer Encoder Layers	6
Attention Heads	8
Hidden Dimension	512
Feed-Forward Layer Size	2048
Embedding Dimensions	Video: 1024 → 512, Audio: 256 → 512, Positional: 128 → 512
Positional Encoding	Learnable
Fusion Method	Low-Rank Multimodal Fusion (rank r = 16)
Optimizer	Adam
Learning Rate	0.0001
Batch Size	16
Training Epochs	50–100
Dropout	0.1
Loss Function	Cross-Entropy

Table 2: Model architecture and hyperparameters. Lists training settings and key components of the proposed transformer-based multimodal fusion model.

For further clarity and reproducibility, the multimodal transformer was implemented with a 6-layer encoder stack, each equipped with 8 attention heads and a hidden dimension of 512, following typical transformer settings for mid-scale multimodal tasks. Each modality passes through an embedding block: video by a 3D-CNN encoder output, 1024-dim; audio by a Wave2Vec-based encoder, 256-dim; and positional plus contextual features by a 2-layer MLP encoder, 128-dim. All these embeddings were projected to a shared 512-d latent space through learnable linear transformations. To establish temporal alignment, all modalities are first interpolated onto a single frequency of 25 Hz, followed by the application of learned positional encodings to preserve temporal ordering. Cross-modal alignment is done using the cross-attention layers, which explicitly learn correlations between modalities before being fed into the self-attention encoder stack. These architectural choices were made to balance model capacity with computational efficiency, thereby ensuring stable convergence on a moderately sized dataset.

In practical terms, the proposed Transformer-Based Multimodal Fusion framework is most suitable for scenarios where synchronized multimodal data are available, such as structured training sessions, controlled match simulations, or professional environments equipped with consistent video feeds and positional tracking systems. The four major inputs needed for the method are (i) high-resolution video, (ii) positional tracking data (GPS/LPS), (iii) audio streams, and (iv) contextual metadata-e.g., possession, pressure zones, match phase. To achieve reliable performance, these modalities must be time-aligned with minimal drift (video ≥ 25 fps and positional sensors ≥ 10 Hz), and should maintain stable viewpoints and minimal signal noise. Those settings that fall short of providing these synchronized inputs, for instance, amateur matches with irregular camera angles, live broadcast feeds featuring latency/compression artifacts, or an environment lacking positional tracking infrastructure, will be less relevant to the framework. Moreover, the current system does best in semi-controlled conditions but may be substantially less robust in completely uncontrolled live-match situations, where lighting conditions, occlusion, and dynamic crowd noise affect the quality of data acquisition. These constraints outline the practical boundaries within which this system can be deployed and emphasize that consistent capture of data in multiple modes is essential in applying the proposed model.

Modifications and troubleshooting

In practical implementation, several issues can arise throughout the multimodal pipeline that may reduce model stability or overall performance. One common problem is temporal misalignment between modalities, where video recorded at 25 fps and positional data captured at 10 Hz fall out of sync, leading to mismatched cues that destabilize attention maps and reduce recall. This can be resolved by applying timestamp-based resampling, linear interpolation, and automatically removing sequences with excessive drift. Audio channels may also suffer from noise caused by wind, crowd sounds, or microphone clipping, which results in the transformer ignoring audio tokens and slightly lowering precision. Applying band-pass filtering, spectral gating, and substituting severely corrupted segments with zero vectors helps maintain audio embedding stability. Visual quality issues, such as player occlusions or motion blur, can weaken the 3D-CNN's spatiotemporal representations and increase pass-drive confusion. This is mitigated by excluding heavily occluded sequences and employing augmentation strategies, including random cropping, brightness normalization, and motion-blur simulation. Transformer instability may occur if embedding scales differ across modalities, producing oscillating validation loss or gradient spikes. Ensuring consistent z-score normalization, using a warm-up learning rate schedule, applying gradient clipping, and increasing dropout can restore stable training. Low-rank fusion can also become problematic when the fusion rank is improperly set, causing distorted cross-modal relationships and class-wise imbalance. Maintaining a moderate rank for small datasets, increasing the rank for larger datasets, and applying L2 regularization help preserve balanced fusion. Computational bottlenecks may arise due to large video tensors that cause GPU memory overflow or slow training. These issues can be addressed through mixed-precision (FP16) training, reducing input resolution during early experimentation, or caching precomputed 3D-CNN features. Finally, the natural class imbalance in tactical decisions, especially for "Drive," may reduce recall and cause fluctuations in AUC; applying class-balanced loss, oversampling minority actions, and enriching contextual metadata can substantially improve balance and decision discrimination. This consolidated troubleshooting guidance ensures that researchers and practitioners can maintain stable training conditions, improve reproducibility, and adapt the framework effectively across various datasets and environments.

Results

This design combines diverse data, video, spatial coordinates, and game contextual data into a single channel to predict tactical actions such as passing, driving, or holding. The approach begins with inputting videos, which are vetted using a Three-Dimensional Convolutional Neural Network (3D CNN). The contextual, positional, and video streams can then have their outputs fused by a Low-Rank Fusion module. The approach effectively obtains cross-modal correlations through fusion space decomposition, minimizing computation...

Discussion

The Transformer-Based Multimodal Fusion framework presented here represents an important development over the baseline Stroop Task Football Test (STFT) outlined in the base paper. In contrast to the base paper, which concentrated on controlled, laboratory-oriented testing with limited and static stimuli like colored arrows and pre-defined tasks (driving or passing), the new model works from real-game, multimodal source data to facilitate a richer, automated tactical analytics pipeline. However, in spite of integration of...

Materials

List of materials used in this article

Name	Company	Catalog Number	Comments
3D CNN	Custom implementation		Spatiotemporal feature extraction from video
GoPro Hero4	GoPro Inc.	https://gopro.com/	Video recording of matches
GPS/IMU Sensors	Catapult Sports / Xsens	https://www.catapultsports.com/	Player positional tracking
Intel Core i7-11700K CPU	Intel	https://www.intel.com	Model training workstation CPU
Librosa	librosa.org	https://pypi.org/project/librosa/	Audio signal processing and resampling
MLP Classifier	PyTorch / TensorFlow		Tactical decision classification
NumPy	Python Software Foundation	https://pypi.org/project/numpy/	Numerical computation and matrix operations
NVIDIA RTX 3080 GPU	NVIDIA	https://www.nvidia.com	Deep learning training and inference
OpenCV	OpenCV.org	https://pypi.org/project/opencv-python/	Frame extraction from video
pandas	Python Software Foundation	https://pypi.org/project/pandas/	Data manipulation and tabular handling
PCA (Scikit-learn)	Scikit-learn	https://scikit-learn.org/	Dimensionality reduction
Python 3.9	Python Software Foundation	https://www.python.org/downloads/release/python-390/	Programming language environment
PyTorch	PyTorch Foundation	https://pytorch.org/	Deep learning framework
scikit-learn	scikit-learn developers	https://pypi.org/project/scikit-learn/	Cross-validation, ICC calculation
TensorFlow 2.8	Google	https://www.tensorflow.org/	Deep learning framework
Transformer Encoder	PyTorch / TensorFlow		Multimodal feature integration with attention
Ubuntu 20.04 LTS	Canonical Ltd.	https://ubuntu.com/	Operating system for model training
Workstation	Custom / Intel, NVIDIA		CPU, GPU, RAM for model training