Visual working memory plays a fundamental role in visual perception and visually guided behavior, and much has been learned about the nature of this memory system by studies using arrays of artificial but easily controlled stimuli (e.g., arrays of colored disks or oriented Gabor patches). However, current quantitative models of visual working memory based on these artificial stimuli cannot be readily extended to the kinds of complex, structured scenes that humans face in daily life. The central goal of the proposed research is to develop and test a new quantitative approach to understanding the representation of complex objects and scenes in working memory, which will lead to a better understanding of real-world vision. In our model, a scene is represented in visual working memory as a noisier version of the pattern of activation that was produced during the perception of that scene. We model this by feeding the scene into a neural network model of the ventral object recognition pathway and using the resulting pattern of activation across the population of units (the population vector) as a model of the working memory representation. We then use this model to make predictions about both behavioral performance and neural activity in human subjects. For example, the change detection task involves presenting a sample scene followed after delay by a test scene and asking subjects to report whether the two scenes are the same or different. We can model this task by feeding the sample and test scenes into the model and computing the distance between the population vectors for the sample and test scenes. In our preliminary data, we find that the distance between the vectors can predict behavioral change detection performance extremely well. Moreover, we find that the vector for the sample scene can predict the pattern of neural activity during the delay between the sample and test scenes (measured via event-related potentials). Note that previous quantitative models of visual working memory cannot make any predictions at all for the natural scenes used in these preliminary studies. We propose testing and extending this model in several ways. First, we will conduct several experiments to assess the ability of the model to predict behavioral performance and neural activity across a broad range of natural stimuli. Second, we will compare the ability of population vectors from different cortical regions (as estimated from the model and from fMRI data) to predict behavioral performance and delay-period activity, providing new insights into the specific brain regions that underlie visual working memory. Third, we will determine whether our model can predict performance in visually guided tasks (e.g., visual search) that rely on visual working memory. Finally, we will assess different versions of our model that implement competing mechanisms for producing capacity limitations, and we will compare the ability of these models to account for behavioral performance for both natural scenes and classic examples of artificial stimuli. Together, these experiments will provide a new and broader understanding of visual working memory.

Visual perception and visually guided behaviors require the ability to rapidly store information in memory (e.g., so that information can be integrated across blinks and eye movements). The proposed research will provide a new understanding of how this visual memory system works in the context of the kinds of scenes that people encounter in daily life. This will be important for understanding, diagnosing, and treating the many disorders that involve impaired visual processing.