Incorporating intrinsic suppression in deep neural networks captures dynamics of adaptation in neurophysiology and perception

See allHide authors and affiliations

Science Advances  14 Oct 2020:
Vol. 6, no. 42, eabd4205
DOI: 10.1126/sciadv.abd4205
  • Fig. 1 Neural network architecture and incorporation of activation-based intrinsic suppression.

    (A) Architecture of a static deep convolutional neural network, in this case AlexNet (35). AlexNet contains five convolutional layers (conv1 to conv5) and three fully connected layers (fc6, fc7, and the decoder fc8). The unit activations in each layer, and therefore the output of the network, are a fixed function of the input image. Photo credit: Kasper Vinken, Boston Children’s Hospital, Harvard Medical School. (B) Intrinsic suppression was implemented for each unit using an intrinsic adaptation state s(t) (orange), which modulates the response r(t) (blue) and is updated at each time step based on the previous response r(t − 1) (Eqs. 1 and 2). The parameter values α = 0.96 and β = 0.7 were chosen to impose a response suppression (β > 0) that gradually builds up over time: For constant input (gray shaded areas), the value of the intrinsic state s(t) gradually increases, leading to a reduction in the response r(t). The intrinsic adaptation state recovers in the absence of input (nonshaded areas). (C) Expansion over time of the network in (A), where the activation of each unit is a function of its inputs and its activation at the previous time step (Eqs. 1 and 2).

  • Fig. 2 Activation-based intrinsic suppression in a neural network captures the attenuation in neurophysiological responses during repetition suppression.

    (A) Face stimuli (created with FaceGen: were presented in repetition trials (adapter = test) and alternation trials (adapter ≠ test). (B) Responses in IT cortex (n = 97, shown normalized to average peak activity) are suppressed more for a repeated stimulus (blue) than for a new stimulus [orange, data from (37)]. Black bars indicate stimulus presentation. (C) The same experiment as in (A) and (B) produced similar repetition suppression in the model with intrinsic suppression (black, blue, and orange lines; gray: no adaptation mechanism; average activity after ReLU of all N = 43,264 conv5 units). The x-axis units are time steps, mapping to bins of 50 ms in (B). (D) Example oddball sequence (top) with a high-probability standard (blue) and a low-probability deviant (purple) and example equiprobable sequence (bottom) as control (green, texture images from (E and F) Average neural responses in rat V1 [n = 55, (E)] and LI [n = 48, (F)] (12) for the standard (blue), deviant (purple), and control (green) conditions (normalized by the response at trial one). (G) Deviant − standard (blue) and deviant − control (green) response differences increase from V1 to LI [error bars: 95% bootstrap confidence interval (CI), assuming no inter-animal difference]. (H to J) Running the experiment in the model captures response dynamics similar to rat visual cortex. (H) and (I) show the results for conv1 and fc7 [indicated by larger markers in (J)], respectively. Green and blue horizontal lines and shading in (J) indicate the neural data averages of (G).

  • Fig. 3 A neural network incorporating intrinsic suppression produces the perceptual bias and enhanced discriminability of aftereffects.

    (A) Examples of the face-gender morph stimuli (created with used in our simulated experiments. After exposure to a male adapter face, the gender decision boundary shifts toward the adapter and an observer perceives a subsequent test face as more female, and vice versa (36). The example adapt, test, and perceive morph levels were picked on the basis of the estimated boundary shift shown in (B). (B) Decision boundaries before (blue) versus after (orange) exposure to a male (0%) adapter based on the top layer (fc7) of the model with intrinsic suppression. Markers show class probabilities for each test stimulus, full lines indicate the corresponding psychometric functions, and vertical lines denote the classification boundaries. Adaptation to a 0% (male) face leads to a shift in the decision boundary toward male faces, hence perceiving the 20% test stimulus as gender-neutral (50%). (C) Decision boundary shifts for the test stimulus as a function of the adapter morph level per layer. The round marker indicates the boundary shift plotted in (B). (D) Relative face-gender discriminability (Materials and Methods, values >1 signify increased discriminability and values <1 denote decreased discriminability) for fc7 as a function of adapter and test morph level. See color scale on the right. The red diagonal indicates that face-gender discriminability is increased for morph levels close to the adapter. (E) Average changes in face-gender discriminability per layer as a function of the absolute difference in face-gender morph level between adapter and test stimulus.

  • Fig. 4 Response enhancements and tuning shifts emerge in deeper layers of a network incorporating intrinsic suppression.

    (A) Effects of adapting to female/male faces on the activation strength of single units. Left: Heatmap showing the activation normalized to the maximum of all 556 responsive fc7 units (rows) for all face-gender morph images (columns). See the color scale on the bottom left. Rows are sorted according to the SIg (Eq. 3). The remaining five heatmaps show the difference (post − pre adaptation) in single-unit activations after adapting to five different adapters. See the color scale on the bottom right. (B) Mean response change (activity post − activity pre) across responsive units for each layer (shaded area = 95% bootstrap CI). For highly gender-selective units (red), the magnitude change (averaged across stimuli) was taken after adapting to a gender stimulus opposite to the unit’s preferred gender [0% adapter for SIg > 0.6, 100% adapter for SIg < −0.6; black rectangles in (A)]. For less gender-selective units (blue), the magnitude change after both 0 and 100% adapters was used. (C) Proportion of adapters causing the preferred morph level to shift toward (attractive, magenta) or away (repulsive, green) from the adapter, averaged across units (shaded area = 95% bootstrap CI). (D) An example unit showing a repulsive shift in tuning curves for the 25% (left) and 75% (right) adapters [the y axes depict activation in arbitrary units (a.u.); black, preadaptation tuning curve; green, postadaptation tuning curve; yellow marker, adapter morph level]. (E) An example unit showing an attractive shift in tuning curves [magenta, postadaptation tuning curve; same conventions as (D)].

  • Fig. 5 Response magnitude and tuning changes in the model differentially explain perceptual boundary shifts and discriminability changes.

    (A) Face-gender boundary shifts toward the adapter were produced both by magnitude changes without tuning changes (top) and by tuning changes without magnitude changes (bottom). Gray shading indicates the range of original layer effects shown in Fig. 3C. (B) Face-gender discriminability enhancement for morph levels close to the adapter was produced by tuning changes without magnitude changes (bottom), but not by magnitude changes without tuning changes (top). Gray shading indicates the range of original layer effects shown in Fig. 3E.

  • Fig. 6 Adapting to prevailing but interfering input enhances object recognition performance.

    (A) Representative examples for each of the five doodle categories from the total set of 540 selected images (63). (B) Schematic illustration of the conditions used in the doodle experiment. In each trial, participants or the model had to classify a hand-drawn doodle hidden in noise (test), after adapting to the same (middle), a different (right), or no (left) noise pattern. The trials with different or no noise adapters were control conditions where we expected to see no effect of adaptation. (C) Participants showed an increase in categorization performance after adapting to the same noise pattern. Gray circles and lines denote individual participants (n = 15). The colored circles show average categorization performance; error bars indicate 95% bootstrap CIs. Chance = 20%.

  • Fig. 7 Intrinsic adaptation can be trained by maximizing recognition performance and is more robust to over-fitting than a recurrent neural network.

    (A) A convolutional neural network with an AlexNet-like feedforward architecture. For the adaptation version, an exponentially decaying hidden state was added to each unit according to Eqs. 1 and 2 (except for the decoder). For the recurrent version, fully recurrent weights were added for the fully connected layer and convolutional recurrent kernels for the three convolutional layers (see drawings in blue; Materials and Methods). (B) Average fitted parameters α and β for each layer after training 30 random initializations of the network with intrinsic adaptation state on same noise trials (SEM bars are smaller than the markers). (C) Test categorization performance on trials with the same Gaussian noise distribution as during training. Full markers: average categorization performance after training 30 random initializations on the same noise trials without intrinsic adaptation state (black), after training with intrinsic adaptation state on same noise trials (blue) or on different noise trials (orange). Empty markers: same as full markers but for the recurrent neural network. SEM bars are smaller than the markers. Chance = 20%, indicated by the horizontal dotted line. (D to F) Average generalization performance of the networks with an intrinsic adaptation state (magenta), recurrent weights (blue), or neither (gray) for same noise trials under noise conditions that differed from training. Performance is plotted as a function of increasing standard deviations (x axis) of Gaussian noise [(D), the vertical line indicates the SD = 0.32 used during training] and uniform noise (E) or as a function of increasing offset values added to Gaussian noise [(F), SD = 0.32, same as training]. Error bounds indicate SEM.

Supplementary Materials

  • Supplementary Materials

    Incorporating intrinsic suppression in deep neural networks captures dynamics of adaptation in neurophysiology and perception

    K. Vinken, X. Boix, G. Kreiman

    Download Supplement

    This PDF file includes:

    • Sections S1 to S7
    • Figs. S1 to S13

    Files in this Data Supplement:

Stay Connected to Science Advances

Navigate This Article