IEEE Control Systems Society SBC Presents Virtual Distinguished Lecture Program
The The IEEE Control Systems Society Student Branch Chapter at IIST, in collaboration with IEEE CSS SBC CET and IEEE CSS Kerala Chapter, successfully organised an online Distinguished Lecture titled "Modelling Self-Attention in Transformers as a Multi-Agent Dynamical System: Equilibria, Stability, and Asymptotic Behaviour." The session was delivered by Dr Claudio Altafini, Professor, Division of Automatic Control, Dept. of Electrical Engineering, Linköping University, Sweden, and a renowned researcher in nonlinear dynamical systems and complex network theory. The session commenced with initial coordination among the organising team, followed by a formal welcome address delivered by Anjana K., chairperson of IEEE CSS SBC IIST. She welcomed the participants and introduced the lecture's theme, emphasising its interdisciplinary relevance in control theory, dynamical systems, and artificial intelligence. Followed by an introduction of the speaker by the Shiv Charan Bhoi, Vice Chair, IEEE CSS SBC IIST. The technical session began with an overview of transformer architectures and their importance in modern machine learning systems. The speaker highlighted that transformers, although widely used, are often treated as black-box models and emphasised the need for theoretical understanding. The speaker explained the concept of tokens as vector representations and described how they are processed through multiple layers. The self-attention mechanism was introduced using query, key, and value matrices, and attention coefficients were explained using inner product and softmax operations. A key contribution of the lecture was interpreting transformers as dynamical systems. By treating layers as time steps, the evolution of tokens was modelled dynamically. Tokens were shown to evolve on a unit sphere, preserving their norm. The speaker introduced Euler flow and extended it to a multi-agent framework, identifying consensus, bipartite, and polygonal equilibria. It was shown that only the principal eigenvector leads to stable consensus. The analysis was extended to full self-attention dynamics, introducing clustering and metastability. Simulations showed convergence to consensus or structured configurations. Further discussion highlighted that, despite complexity, systems tend to converge toward consensus. Real-world models like ALBERT were discussed, showing token alignment. Experimental observations demonstrated that token representations tend to converge and align as they pass through multiple layers of the transformer. This behaviour was shown through analysis of token distances and correlations, confirming that tokens tend to collapse towards a common direction. This phenomenon, widely known in machine learning literature as over-smoothing or rank collapse, was discussed in detail. The speaker emphasised that although this behaviour is mathematically consistent with the dynamical system model, it is undesirable in practical applications because it reduces the diversity of token representations. The session also had a lively discussion where participants asked thoughtful questions about how self-attention mechanisms relate to traditional multi-agent systems, especially those that use adjacency and Laplacian matrices. The speaker clarified that while there are similarities, the transformer dynamics are fundamentally different due to their state-dependent interactions and the presence of multiple attention heads, which complicate direct analogies with traditional consensus models. The event concluded with a vote of thanks delivered by Saroj B. Zachariah, Chairperson, IEEE CSS SBC CET. As a token of appreciation, a virtual memento was presented to the speaker by Dr. Sourav Burmik, Assistant Professor, IIST and Chapter Advisor, IEEE CSS SBC IIST. Participants particularly benefited from the systematic explanation of self-attention mechanisms, stability analyses, and equilibrium behaviour. The use of simulations and practical examples further enhanced the learning experience, making complex concepts more accessible. The interactive question-and-answer session also contributed to clarifying key doubts and encouraging critical thinking. The lecture successfully bridged control theory and machine learning, providing a theoretical understanding of transformers and encouraging interdisciplinary research. The event successfully achieved its objective of presenting a theoretical framework for understanding self-attention mechanisms in transformers using dynamical systems theory. By interpreting transformer layers as time-evolving systems, the lecture provided valuable insights into their behaviour, stability, and convergence properties. The session highlighted the importance of combining theoretical analysis with practical applications, thereby fostering interdisciplinary research and learning among participants.
Dr. Claudio Altafini
Professor, Division of Automatic Control, Dept. of Electrical Engineering, Linköping University, Sweden
Website Url
Event Details
-
Mode:Online
