The act of foveation is a fundamental cornerstone of active vision in the human visual system, because it allows processing images with very high acuity.
However, foveating eye movements pose a significant challenge in their own right. Namely, because a large number of visually-sensitive brain areas are retinotopically organized, foveal neurons in such areas experience new afferent image patches after every eye movement that are completely devoid of any explicit context about the global scene organization. For example, when making a rapid eye movement from a traffic light to the road ahead, the foveal image before the eye movement is a circular green light, whereas it is a gray pavement right after the movement. Thus, in order to maintain a subjective experience of perceptual stability across rapid saccades, the brain needs to bridge visual representations across eye movements, and this need is particularly important for foveal visual representations.
In a new paper just published in PLOS Biology, we found that foveal neurons in the superior colliculus (SC) are informed about the expected appearance of post-saccadic foveal visual features, even though they never directly experience the pre-saccadic peripheral features; thus, foveal neurons signal a “prediction error” if feature continuity between the periphery and fovea is disrupted across the eye movements. Our discovery makes a significant advance in our understanding of perceptual stability, and it strongly motivates exploring the role of foveal visual representations in such stability. This discovery is also integral to concepts of internal monitoring and the sense of agency, which are critical in several neurological and psychiatric disorders like schizophrenia. Therefore, learning about the mechanisms of internal monitoring is additionally clinically relevant.
The idea of trans-saccadic foveal prediction has appealed to psychologists and theoreticians for many decades. However, the underlying neurophysiological mechanisms were not documented. Past neurophysiological work has primarily focused on “predictive remapping of spatial representations”; this remapping refers to the updating of peripheral stimulus spatial locations in retinotopic representations across saccades. For example, a visual stimulus in the right retinotopic hemifield can end up being in the left retinotopic hemifield after a large rightward eye movement. However, we studied a remapping from the pre-saccadic peripheral saccade target visual representation to the post-saccadic foveal one. In many ways, such foveal remapping is more fundamental than spatial updating, especially because it is foveal vision that is most relevant for perception in the immediate wake of foveating eye movements. Most importantly, we focused on remapping of visual feature representations rather than spatial locations. We found that foveal neurons change their responses to a particular visual feature entering into their response fields (via eye movements) as a function of the pre-saccadic visual appearance; this happens even when the pre-saccadic visual feature was never inside the foveal neurons’ response fields pre-saccadically.
The neural modulations that we observed in foveal SC neurons are a remarkable instantiation of predictive coding. That is, the visual system may “detect” unpredictable visual signals that deviate from the “predicted” consequences of the image flow disruptions caused by eyeball rotations, and therefore create a percept of a changed environment. Thus, in this framework, the visual environment may be perceived as being stable, despite major disruptions of retinal images, only when no “unpredictable” visual signals are detected across the eye movements. Our foveal SC neurons contributed exactly to detecting “unpredictability”. In retrospect, this idea makes sense given how saccades induce highly systematic sensory-motor contingencies every time that they occur, and it can also generalize to more universal frameworks for building invariant object detection mechanisms in the brain.
Our results are additionally appealing because they demonstrate a noteworthy, and timely, functional use for an equally surprising discovery that we recently made a couple of years ago about peripheral SC representations. Specifically, when we studied saccade-related motor bursts in peripheral SC neurons, we found that these motor bursts exhibited paradoxical visual feature tuning properties in them, such that the same eye saccade (both in metrics and kinematics) to two different visual images was associated with two distinct motor bursts by the very same SC neuron. Clearly, such visual feature tuning of motor bursts was independent of saccade control, leading to the question of what it might be good for at all. Our discovery in the current study, that foveal neurons are sensitive to peripheral pre-saccadic target appearance (even without explicitly experiencing the pre-saccadic targets in their response fields), suggests that sensory tuning in peripheral motor bursts may act as a peri-saccadic store of visual appearance, exactly at the time of sensory uncertainty in the retina. This could aid in foveal prediction. Consistent with this, we found that foveal prediction in our experiments was contingent on saccade generation and also strongly dependent on the visibility and dimensions of peripheral visual features being tracked across saccades. Thus, in the span of only a couple of years, we went from a curious observation about peripheral motor bursts to a potential framework for its use trans-saccadic integration.
Our discovery motivates studying foveal visual representations in multiple brain areas during active vision scenarios, especially because these representations are a fundamental goal of foveating eye movements, and also because they can additionally impact subsequent peripheral orienting.