Research ArticleCOGNITIVE NEUROSCIENCE

The value of what’s to come: Neural mechanisms coupling prediction error and the utility of anticipation

See allHide authors and affiliations

Science Advances  19 Jun 2020:
Vol. 6, no. 25, eaba3828
DOI: 10.1126/sciadv.aba3828
  • Fig. 1 The utility of anticipation drives a preference for advanced information.

    (A) Task. Participants were presented with an immediate-information target (“Find out now”) and a no-information target (“Keep it secret”), as well as two central stimuli signaling the probability of reward and the duration of a waiting period until reward or no-reward delivery. A symbolic image cue was presented for the entire waiting period until a rewarding image or an image signaling no reward appeared. (B) The immediate-information target was followed by cues that predict upcoming reward or no reward (reward predictive cue or no-reward predictive cue). The no-information target was followed by a cue that implied nothing about the reward outcome (no-information cue). (C) Average behavior. Participants showed a stronger preference for advanced information under longer delay conditions [two-way analysis of variance (ANOVA), F4,950 = 10.0]. The effect of reward probability (F4,950 = 0.35, P > 0.05) showed heterogeneous dependencies (fig. S4). (D to G) Computational model (8). (D) Following (1), the value of each cue is determined by the sum of (i) the utility of anticipation that can be consumed while waiting for reward (red) and (ii) the value of reward consumption itself (green). (E and F) If a reward predictive cue is presented, then the anticipation is boosted throughout the delay period (orange upward arrows). The boosting is quantified by surprise, proportional to the absolute value of aRPE Eq. 1. (G) The model predicts that the value difference between the two targets is larger under longer delay conditions (8). (H) The average of modeled preferences, using a hierarchical Bayesian fitting procedure (8). (I) The model (blue) captures the effect of delay conditions in data (black). The error bars indicate the mean and SEs of participants (n = 39). See fig. S2 for the effect of probability conditions, and fig. S1 for how other classical models fail to explain behavior.

  • Fig. 2 Neural representation of our computational model’s anticipatory utility signal in the vmPFC.

    (A) The anticipatory utility signal at time t is an integral of discounted future anticipation (urgency signal) at t′ > t (red curve). This signal is different from a well-studied expected value of future reward, which we included in the same GLM. (B) The model’s prediction for fMRI signals (solid red) is computed by convolving the model’s signal (dotted red) with a canonical HRF (light blue). (C) BOLD in vmPFC positively correlated with an anticipatory utility signal. This survived our phase-randomization test (whole-brain FWE P < 0.001; see fig. S8) and SPM’s standard whole-brain FWE (P < 0.05). A cluster surrounding the peak [10,50,16] (cFWE, P < 0.05 with height threshold at P < 0.001) is shown for display purposes. (D) The temporal dynamics of the BOLD signal in the vmPFC [shown in (C)] matched the model’s anticipatory utility signal during the anticipation period. Changes in activity following receipt of a reward predictive cue (red) and a no-information cue (magenta), as well as the model’s prediction for each of these conditions (black) are shown. The error bar indicates the SEM over participants. (E) A confirmatory analysis shows that activity in vmPFC is more strongly correlated with our model’s anticipatory utility signal than an expected reward value signal. The average regression weights in the vmPFC for the anticipatory utility signal were significantly greater than the expected reward signal (***P < 0.001, permutation test). The former was also significantly larger than zero (***P < 0.001, t test, t38 = 4.07), but the latter was not. The error bars indicate the mean and SEM. A.U., arbitrary units; N.S., not significant.

  • Fig. 3 Neural representation of our computational model’s aRPE signals.

    (A) The ventral tegmental area and substantia nigra (VTA/SN) and medial posterior parietal cortex (mPPC) BOLD positively correlated with the model’s aRPE at the time of advanced information cue presentations [VTA/SN, P < 0.05 FWE small volume correction (39); mPPC, P < 0.05 whole-brain FWE, cluster-corrected at P < 0.001]. Voxels at P < 0.005 (uncorrected) are highlighted for display purposes. (B) Our confirmatory analysis shows that both the VTA/SN and the mPPC show paradigmatic correlations with aRPE. At the time of advanced information cue presentations, BOLD in the VTA/SN and the mPPC positively correlated with the model’s actual cue value signal and negatively with the model’s expected cue value signal, indicating that both regions express canonical prediction errors. The difference was significant in the VTA/SN (P < 0.001, permutation test) and in the mPPC (P < 0.001, permutation test). The positive correlation with cue outcome values and the negative correlation with expected values were all significant (received cue value, P < 0.01 for the VTA/SN and the mPPC by t test, t38 = 3.24 and t38 = 3.40; expected cue value, P < 0.01 for the VTA/SN and P < 0.001 for the mPPC by t test, t38 = 2.82 and t38 = 4.37). (C) Our confirmatory analysis shows that both regions express stronger correlations with our model’s full aRPE than with standard prediction error with discounted reward (RPE) at advanced information cues. The difference was significant between the VTA/SN and in the mPPC cluster (P < 0.05, permutation test). ***P < 0.001, **P < 0.01, and *P < 0.05.

  • Fig. 4 Neural correlates of our computational model’s surprise that can boost anticipation utility in our computational model.

    (A) Our model predicts that a surprise, quantified by the absolute value of aRPE, can boost the utility value of anticipation. The model predicts the effect of boosting to be sustained during the anticipatory period, in contrast to the phasic, short, aRPE signal. (B) A surprise at advanced information cues, quantified by the absolute value of aRPE, significantly correlated with BOLD in the hippocampus [FWE, P < 0.05, small volume correction (46)]. (C) The temporal dynamics of fMRI signal in the hippocampus. Changes in activity averaged over participants after receiving a reward predictive cue (orange), and after receiving a no-information cue (magenta), are shown. The phasic response confirmed in (B) is apparent in the early phase of the delay period (blue). Still, the coding of boosting-related value is sustained throughout the entire waiting period (blue and yellow), which is what our model predicted. The error bar indicates the SEM. Please also see fig. S14 for responses to a no-reward predictive cue.

  • Fig. 5 Functional connectivity analysis suggestive of a neural network associated with our model’s anticipation utility computation.

    (A) Functional coupling between the VTA/SN and the hippocampus is positively modulated by the model’s anticipation utility signal [P < 0.05, FWE small volume correction (46)]. PPI regressor: BOLD signal in VTA/SN modulated by model’s anticipation utility signal. (B) Functional coupling between the vmPFC and the hippocampus is positively modulated by the model’s aRPE signal [P < 0.05 FWE small volume correction (46)]. PPI regressor: BOLD signal in vmPFC modulated by the model’s aRPE signal. (C) The functional coupling strength between the vmPFC and the hippocampus mediated by the model’s prediction error signal is positively correlated with the model’s boosting coefficient parameter estimated by the behavior of participants (r = 0.37, P < 0.05). (D) Three distinctive regions contributed to construct the anticipation utility, in a manner that is predicted by our computational model. The three-dimensional brain image was constructed by the mean T1 brain images, which were cut at y = 34 and z = 15.

Supplementary Materials

  • Supplementary Materials

    The value of what’s to come: Neural mechanisms coupling prediction error and the utility of anticipation

    Kiyohito Iigaya, Tobias U. Hauser, Zeb Kurth-Nelson, John P. O’Doherty, Peter Dayan, Raymond J. Dolan

    Download Supplement

    This PDF file includes:

    • Figs. S1 to S17
    • Table S1

    Files in this Data Supplement:

Stay Connected to Science Advances

Navigate This Article