Visual search and FEF

With: Hugo Fernandes, Pat Lawlor, Josh Glaser, Daniel Wood, Mark Segraves, Konrad Körding

To bring objects of interest in the visual environment into focus, we shift our gaze up to three times a second. Deciding where to look next among a large number of alternatives is thus one of the most frequent decisions we make. Why do we look where we look?

Studies have shown that both bottom-up (e.g. luminance contrast, saliency, edge–energy) and top-down factors (such as target similarity or relevance) influence the guidance of eye movements. Predictive models of eye movements are derived from priority maps composed of one or more of these factors. Evidence for such maps have been reported in the lateral intra parietal (LIP) cortex, the frontal eye field (FEF), primary visual cortex (V1) and/or ventral visual area V4, but computational maps of priority have not been used to model neural activity.

In this project, we attempt to unify computational models and neurophysiology of gaze priority maps. We developed primate models of gaze behavior in natural scenes by rewarding monkeys to find targets embedded in scenes. We build predictive models of gaze using computational definitions of visual priority and quantify model predictions on monkeys' fixation choices.

To ask if FEF neurons represent computational descriptions of priority including saliency, relevance and energy, we build generalized linear models (GLMs) of Poisson-spiking neurons (see Fig. 2).

We find that the majority of variance in FEF firing rates are explained by the direction of upcoming movement. However, we find firing rate enhancement for saccades to targets within the receptive field, suggesting that FEF neurons encode expected reward during search.

Rapid scene categorization and MEG

With: Sebastian Pannasch, Bruce Hansen, Lester Loschky

To make effective decisions in our environment, our brains must be able to effectively recognize and comprehend real-world scenes. Human are remarkable at recognizing scene categories from the briefest of glimpses (< 20 ms). The holistic information that can be extracted in such short durations has come to be known as scene gist.

Computational models of scene gist, such as the spatial envelope (SpEn) model, as well as behavioral studies provide useful suggestions for what visual features the brain might use to categorize scenes. However, they do not inform us about when and where in the brain such information is represented and how scene-categorical judgments are made on the basis of these representations.

Here, we investigate the brain-behavior relationship underlying rapid scene categorization. We use whole-scalp magnetoencephalography (MEG) to track visual scene information flow in the ventral and temporal cortex, using spatially and temporally resolved maps of decoding accuracy. To investigate the time course of visual representation versus behavioral category judgment, we then use neural decoders in concert with decoders based on SpEn features to study errors in behavioral categorization (Fig. 3). Using confusion matrices, we track how well patterns of errors in neural decoders are be explained by SpEn decoders and behavioral errors. We find that both SpEn decoders and behavioral errors explain unique variance throughout the ventrotemporal cortex, and that their effects are temporally simultaneous and restricted to 100-250 ms after stimulus onset. Thus, during rapid scene categorization, neural processes that ultimately result in behavioral categorization are simultaneous and colocalized with neural processes underlying visual information representation.

Movement chunking and basal ganglia

With: Daniel Acuna, Max Berniker, Rob Turner, Scott Grafton, Konrad Körding

We routinely execute complex movement sequences with such effortless ease that the computational complexity of planning them optimally is often under-appreciated. When movements are learned from external cues, they start out highly regular, progressively become more varied until the become habitual and more regular again (Fig. 4).

A common facet of many such complex movements is that they tend to be discrete nature, i.e. they are often executed as chunks (Fig. 5a). Here, we framed movement chunking as the result of a trade-off between the desire to make efficient movements and minimize the computational complexity of optimizing them (Fig. 5b). We show that monkeys adopt a cost-effective strategy to deal with this tradeoff. By modeling chunks as minimum-jerk trajectories (Fig. 5c), we found that kinematic sequences are best described as progressively resembling locally optimal trajectories, with optimization occurring within chunks (Fig. 5d). Thus, the cumulative optimization costs are kept in check over the course of learning.

To study the underlying neural basis of these tradeoffs, we also record from globus pallidus (GP) — motor output structures in the basal ganglia — implicated in habit learning and movement sequencing. Our hypothesis is that neurons encode complexity of a movement, and that this encoding depends on whether the movement is novel or habitual. To test this hypothesis, we train monkeys to execute habitual overlearned (OL) sequences comprising consecutive center–out–and–back reaches always in the same order, as well as novel random (RN) sequences comprising a set of four cued reaches whose directions varied from trial to trial. For each neuron and sequence class, we fit Poisson generalized linear models (GLMs) to account for firing rate variability as a function of multiple regressors: (1) the cue type, (2) the order of the movement in the sequence, (3) the upcoming reach direction, (5) the instantaneous hand kinematics, (6) the computational complexity of executing the rest of the sequence as a single chunk relative to the complexity of executing each movement as a separate chunk, (7) the efficiency of the upcoming movement, defined as the negative squared jerk, (7) a reward event at the end of each movement, and (8) a trial-end regressor.

We found that GP encoded the order of the movement in the context of the sequence, as well as the relative complexity and efficiency of movements (see example neuron in Fig. 6a). When we compared the habitual (OL) and the novel (RN) sequences, a significantly larger proportion of GP neurons encoded the order of the movement, but a significantly smaller proportion encoded the relative complexity (Fig. 6b). The complexity of optimal control is thus an important driver of habit learning, both behaviorally and neurally.

Over and above the novel role of GP neurons in indexing computational complexity, we also found that a significantly larger proportion of GP neurons encode instantaneous arm kinematics during the habitual relative to the novel condition. Furthermore, a significantly smaller proportion of GP neurons encoded the cue and the reward during the habitual relative to the novel condition, suggesting that as movements become habituated with learning, GP firing rates are less modulated by response to an external movement cue or an external reward. Our findings suggest a dynamic functional role for GP — starting with bracketing movement sequences for goal-directed movements, and transitioning to a more direct control of movements once they are habituated.

Sensorimotor uncertainty and PMd

With: Brian Dekleva, Paul Wanda, Lee Miller, Konrad Körding

Movements in the real world are often planned with uncertain information about where to move. Understanding the role of uncertainty in movement plans is important to improve rehabilitation therapies and brain-based prostheses. Bayesian estimation theory, which combines sources of information in proportion to their uncertainty, predicts movement behavior when uncertainty about subjective beliefs (priors) and sensory observations (likelihoods) is varied. By manipulating sensory uncertainty of the target in each trial of a reaching task, as well as the prior uncertainty of the target location during the task, we show that monkeys' reaches can be predicted using a Bayesian model that weights the different sources of uncertainty appropriately (Fig. 7).

To perform probabilistic inference of this nature, the brain must represent/reflect uncertainty. We asked how sensory uncertainty is represented in population activity in dorsal premotor cortex (PMd) and primary motor cortex (M1) during movement preparation. We found that greater sensory uncertainty led to increased firing rates and broad recruitment of neurons in PMd but not M1 (Fig. 8). This broad recruitment suggests that multiple movement plans are represented in PMd activity until uncertainty-reducing feedback about the target location is received.

Reward in premotor and motor cortices

With: Brian Dekleva, Sam Cooler, Lee Miller, Konrad Körding

Reward is an important feedback signal for motor learning. How reward information reaches the motor cortex to influence movement planning and execution is unknown. Therefore, we asked whether premotor and motor cortices encode reward. We found a strong and robust encoding of reward outcome in premotor (PMd) and motor (M1) cortices. In particular, neurons increased their firing rate following trials that were not rewarded. We further investigated the nature of this signal and established that it is unlike any previously reported reward signal in the brain. It is unrelated to reward magnitude expectation or prediction error, it is not influenced by history of reward, and it is not modulated by error magnitude. Using generalized linear modeling of spikes we also carefully verified that the signal is not explained away by differences in kinematics or return reach planning activity. Thus, we found a categorical reward signal in PMd and M1 signaling the presence or absence of reward at the end of a goal-directed task.

V4 color tuning in natural scenes

With: Hugo Fernandes, Matt Smith, Konrad Körding

How does the subjective experience of color arise from oscillations in the visible electromagnetic spectrum? We know that color perception begins in the L, M and S cone cells in the retina, which have broad preferred sensitivities to greenish-yellowish, greenish, and bluish frequencies in the electromagnetic spectrum. Following this stage, tuning to opponency — red-green, blue-yellow, light-dark — begins to emerge in the retina, the lateral geniculate nucleus (LGN), and the primary visual cortex (V1) where specialized color cells in V1 are clustered into "blobs".

The ventral visual area V4 is the first stage at which tuning to hue — subjectively experienced colors — begins to emerge. However, most experiments measuring hue tuning have been done with highly controlled artificial stimuli such as bars or gratings carefully optimized for preferred orientation and spatial receptive fields. Therefore, not much is known about the neural representation of perceived hue in naturalistic conditions.

In this project, we build encoding models of V4 spiking activity using a generalized linear model (GLM) framework when monkeys freely viewed natural scenes. We take advantage of the fact that hue can be represented on a circle by using the HSI color space and assuming a broad cosine tuning (Fig. 10). We show that these assumptions are sufficient to rapidly fit encoding models of V4 spikes — in response to naturalistic free viewing — as a function of luminance, hue and orientation, and saturation with very few parameters (Figs. 11, 12).

Synthetic Neurophysiology with CNNs

With: Hugo Fernandes, Matt Smith, Konrad Körding

The ventral visual stream in the visual cortex consists of several brain areas that progressively represent more abstract aspects of visual information, going from orientations and spatial frequencies in the primary visual cortex (V1) to object identities in the inferior temporal (IT) cortex. The exact computational nature of this feed-forward transformation is yet to be completely characterized. Visual area V4 — being an intermediate are in this pathway — is a prime candidate to understand the computations underlying our visual percepts.

Deep convolutions neural networks (CNNs) have recently seen tremendous success in real-world visual tasks such as object recognition. It has also been shown that they capture the representational properties of the inferior temporal cortex (IT) in both humans and monkeys across a wide range of object categories, suggesting that they are excellent biophysical and computational approximations of the feed-forward part of the ventral visual stream. Thus, pre-trained CNNs can be used as substrates to study the computational and representational properties of neurons all over the ventral stream. This is a promising new technique to bridge computational and physiological approaches to understanding vision and has come to be known as synthetic neurophysiology.

Here, we focus on the visual area V4. We present artificial stimuli such as colored checkerboards (picture above) to CNNs and study the tuning properties of synthetic neurons in the CNN. Preliminary analysis suggests that synthetic neurons are hue-tuned, and further, the distribution of preferred hue across the population of synthetic neurons resembles those of real V4 neurons characterized during the free-viewing of natural scenes. Both these distributions also mirror the highly non-uniform distribution of hues in natural scenes.

Natural scene psychophysics with CNNs

With: Claire Chambers, Hugo Fernandes, Pat Lawlor, Josh Glaser, Konrad Körding

During visual search in natural scenes, targets can only be fixated after they are detected in our visual periphery. Why is the same target detected in some contexts but undetected in others? This question has been difficult to address with standard psychophysical techniques due to the interaction between natural scene statistics, visual crowding, and feature-based attention.

Here, we use a large natural scene search database of monkeys searching for a Gabor wavelet embedded in natural scenes. We train convolutional neural networks (CNNs) to distinguish between detected and undetected contexts. The CNN uses fixation–centered and target–centered multi-scale glimpses as inputs and learns to predict whether the animal saccaded to the target or not. The CNN-based approach complements models based on distance-to-target, peak velocity, fixation duration, local multiscale contrast, and saliency. By visualizing preferred inputs of the most predictive CNN units, we aim to provide additional insight into the nature of detectability and crowding in natural scenes.