Dr Hossein Adeli | Brain Encoding and Decoding with Transformer Attention

The attention mechanism, central to the transformer architecture, has emerged as a powerful and scalable computational motif underlying many recent advances in machine learning. Beyond its success in AI, attention offers a compelling framework for building interpretable, mechanistically grounded models of brain function. In this talk, I will present a line of work exploring this potential across encoding and decoding. I will begin with Transformer brain encoders, in which attention is used to model how retinotopic visual features are dynamically routed to category-selective areas in high-level visual cortex, achieving state of the art encoding performance. I will then describe how this framework was extended to construct a digital twin of the visual system, enabling in silico experiments that reveal categorical visual selectivity across the whole brain. A third line of work leverages the in-context learning property of transformers to build encoding models that generalize across subjects and datasets. Finally, I will present our work on interpretable decoding models that reconstruct images from brain activity using attention-based architectures.