全部 标题 作者
关键词 摘要

OALib Journal期刊
ISSN: 2333-9721
费用：99美元
投递稿件

查看量下载量

相关文章

更多...
PLOS ONE  2014 

Optogenetic Stimulation in a Computational Model of the Basal Ganglia Biases Action Selection and Reward Prediction Error

DOI: 10.1371/journal.pone.0090578

Full-Text   Cite this paper   Add to My Lib

Abstract:

Optogenetic stimulation of specific types of medium spiny neurons (MSNs) in the striatum has been shown to bias the selection of mice in a two choices task. This shift is dependent on the localisation and on the intensity of the stimulation but also on the recent reward history. We have implemented a way to simulate this increased activity produced by the optical flash in our computational model of the basal ganglia (BG). This abstract model features the direct and indirect pathways commonly described in biology, and a reward prediction pathway (RP). The framework is similar to Actor-Critic methods and to the ventral/dorsal distinction in the striatum. We thus investigated the impact on the selection caused by an added stimulation in each of the three pathways. We were able to reproduce in our model the bias in action selection observed in mice. Our results also showed that biasing the reward prediction is sufficient to create a modification in the action selection. However, we had to increase the percentage of trials with stimulation relative to that in experiments in order to impact the selection. We found that increasing only the reward prediction had a different effect if the stimulation in RP was action dependent (only for a specific action) or not. We further looked at the evolution of the change in the weights depending on the stage of learning within a block. A bias in RP impacts the plasticity differently depending on that stage but also on the outcome. It remains to experimentally test how the dopaminergic neurons are affected by specific stimulations of neurons in the striatum and to relate data to predictions of our model.

 References

[1]  Redgrave P, Prescott TJ, Gurney K (1999) The basal ganglia: a vertebrate solution to the selection problem? Neuroscience 89: : 1009–1023. Available: http://linkinghub.elsevier.com/retrieve/？pii/S0306452298003194. Accessed 29 January 2014.
[2]  Hwang EJ (2013) The basal ganglia, the ideal machinery for the cost-benefit analysis of action plans. Front Neural Circuits 7: : 121. Available: http://www.pubmedcentral.nih.gov/article？render.fcgi?artid=3717509&tool=pmcentrez？&rendertype=abstract. Accessed 6 August 2013.
[3]  McGeorge a J, Faull RL (1989) The organization of the projection from the cerebral cortex to the striatum in the rat. Neuroscience 29: : 503–537. Available: http://www.ncbi.nlm.nih.gov/pubmed/24725？78. Accessed 17 February 2014.
[4]  Alexander G, DeLong M, Strick PL (1986) Parallel organization of functionally segregated circuits linking basal ganglia and cortex. Annu Rev Neurosci 9: : 357–381. Available: http://www.annualreviews.org/doi/pdf/10.？1146/annurev.ne.09.030186.002041. Accessed 25 October 2011.
[5]  Albin RL, Young AB, Penney JB (1989) The functional anatomy of basal ganglia disorders. Trends Neurosci 12: : 366–375. Available: http://linkinghub.elsevier.com/retrieve/？pii/016622368990074X. Accessed 6 April 2011.
[6]  McHaffie JG, Stanford TR, Stein BE, Coizet V, Redgrave P (2005) Subcortical loops through the basal ganglia. Trends Neurosci 28: : 401–407. Available: http://www.ncbi.nlm.nih.gov/pubmed/15982？753. Accessed 27 January 2014.
[7]  Surmeier DJ, Ding J, Day M, Wang Z, Shen W (2007) D1 and D2 dopamine-receptor modulation of striatal glutamatergic signaling in striatal medium spiny neurons. Trends Neurosci 30: : 228–235. Available: http://www.ncbi.nlm.nih.gov/pubmed/17408？758. Accessed 23 January 2014.
[8]  Reynolds JNJ, Wickens JR (2002) Dopamine-dependent plasticity of corticostriatal synapses. Neural Networks 15: : 507–521. Available: http://linkinghub.elsevier.com/retrieve/？pii/S089360800200045X. Accessed 25 January 2011.
[9]  Lau B, Glimcher PW (2007) Action and outcome encoding in the primate caudate nucleus. J Neurosci 27: : 14502–14514. Available: http://www.ncbi.nlm.nih.gov/pubmed/18160？658. Accessed 19 July 2011.
[10]  Doya K, Samejima K, Katagiri K, Kawato M (2002) Multiple model-based reinforcement learning. Neural Comput 14: : 1347–1369. Available: http://www.ncbi.nlm.nih.gov/pubmed/12020？450. Accessed 24 January 2014.
[11]  Samejima K, Ueda Y, Doya K, Kimura M (2005) Representation of action-specific reward values in the striatum. Science 310: : 1337–1340. Available: http://www.ncbi.nlm.nih.gov/pubmed/16311？337. Accessed 24 January 2014.
[12]  Kim H, Sul JH, Huh N, Lee D, Jung MW (2009) Role of striatum in updating values of chosen actions. J Neurosci 29: : 14701–14712. Available: http://www.ncbi.nlm.nih.gov/pubmed/19940？165. Accessed 22 July 2011.
[13]  Pawlak V, Kerr JND (2008) Dopamine receptor activation is required for corticostriatal spike-timing-dependent plasticity. J Neurosci 28: : 2435–2446. Available: http://www.ncbi.nlm.nih.gov/pubmed/18322？089. Accessed 25 January 2014.
[14]  Pawlak V, Wickens JR, Kirkwood A, Kerr JND (2010) Timing is not Everything: Neuromodulation Opens the STDP Gate. Front Synaptic Neurosci 2: : 146. Available: http://www.pubmedcentral.nih.gov/article？render.fcgi?artid=3059689&tool=pmcentrez？&rendertype=abstract. Accessed 6 July 2011.
[15]  Izhikevich EM (2007) Solving the distal reward problem through linkage of STDP and dopamine signaling. Cereb Cortex 17: : 2443–2452. Available: http://www.ncbi.nlm.nih.gov/pubmed/17220？510. Accessed 20 July 2011.
[16]  Montague P, Dayan P, Sejnowski TJ (1996) A framework for mesencephalic dopamine systems based on predictive Hebbian learning. J Neurosci 16: : 1936–1947. Available: http://www.jneurosci.org/content/16/5/19？36.short. Accessed 20 October 2011.
[17]  Schultz W, Dayan P, Montague PR (1997) A neural substrate of prediction and reward. Science 275: : 1593–1599. Available: http://www.ncbi.nlm.nih.gov/pubmed/90543？47. Accessed 17 February 2014.
[18]  Schultz W, Dickinson A (2000) Neuronal coding of prediction errors. Annu Rev Neurosci 23: : 473–500. Available: http://www.ncbi.nlm.nih.gov/pubmed/19176？813. Accessed 22 January 2014.
[19]  Daw ND, Doya K (2006) The computational neurobiology of learning and reward. Curr Opin Neurobiol 16: : 199–204. Available: http://www.ncbi.nlm.nih.gov/pubmed/16563？737. Accessed 21 January 2014.
[20]  Montague PR, Hyman SE, Cohen JD (2004) Computational roles for dopamine in behavioural control. Nature 431: : 760–767. Available: http://www.ncbi.nlm.nih.gov/pubmed/15483？596. Accessed 20 January 2014.
[21]  Hollerman JR, Schultz W (1998) Dopamine neurons report an error in the temporal prediction of reward during learning. Nat Neurosci 1: : 304–309. Available: http://www.ncbi.nlm.nih.gov/pubmed/10195？164. Accessed 10 February 2014.
[22]  Smith Y, Bevan M, Shink E, Bolam J (1998) Microcircuitry of the direct and indirect pathways of the basal ganglia. Neuroscience 86: : 353–387. Available: http://europepmc.org/abstract/MED/988185？3. Accessed 17 May 2013.
[23]  Kravitz A V, Tye LD, Kreitzer AC (2012) Distinct roles for direct and indirect pathway striatal neurons in reinforcement. Nat Neurosci: 1–4. Available: http://www.ncbi.nlm.nih.gov/pubmed/22544？310. Accessed 20 July 2012.
[24]  Kreitzer AC, Berke JD (2011) Investigating striatal function through cell-type-specific manipulations. Neuroscience 198: : 19–26. Available: http://www.pubmedcentral.nih.gov/article？render.fcgi?artid=3221791&tool=pmcentrez？&rendertype=abstract. Accessed 6 March 2013.
[25]  Ball KT, Combs TA, Beyer DN (2011) Opposing roles for dopamine D(1)- and D(2)-like receptors in discrete cue-induced reinstatement of food seeking. Behav Brain Res 222: : 390–393. Available: http://www.ncbi.nlm.nih.gov/pubmed/21497？172. Accessed 27 May 2011.
[26]  Tai L-H, Lee AM, Benavidez N, Bonci A, Wilbrecht L (2012) Transient stimulation of distinct subpopulations of striatal neurons mimics changes in action value. Nat Neurosci 15: : 1281–1289. Available: http://www.ncbi.nlm.nih.gov/pubmed/22902？719. Accessed 28 January 2013.
[27]  Kravitz AV, Freeze BS, Parker PRL, Kay K, Thwin MT, et al.. (2010) Regulation of parkinsonian motor behaviours by optogenetic control of basal ganglia circuitry. Nature 466: : 622–626. Available: http://www.nature.com/doifinder/10.1038/？nature09159. Accessed 7 July 2010.
[28]  Tsai H-C, Zhang F, Adamantidis A, Stuber GD, Bonci A, et al.. (2009) Phasic firing in dopaminergic neurons is sufficient for behavioral conditioning. Science 324: : 1080–1084. Available: http://www.ncbi.nlm.nih.gov/pubmed/19389？999. Accessed 18 July 2011.
[29]  Watabe-Uchida M, Zhu L, Ogawa SK, Vamanrao A, Uchida N (2012) Whole-brain mapping of direct inputs to midbrain dopamine neurons. Neuron 74: : 858–873. Available: http://www.ncbi.nlm.nih.gov/pubmed/22681？690. Accessed 21 May 2013.
[30]  Chaudhury D, Walsh JJ, Friedman AK, Juarez B, Ku SM, et al.. (2013) Rapid regulation of depression-related behaviours by control of midbrain dopamine neurons. Nature 493: : 532–536. Available: http://www.pubmedcentral.nih.gov/article？render.fcgi?artid=3554860&tool=pmcentrez？&rendertype=abstract. Accessed 17 September 2013.
[31]  Cohen MX, Frank MJ (2009) Neurocomputational models of basal ganglia function in learning, memory and choice. Behav Brain Res 199: : 141–156. Available: http://www.ncbi.nlm.nih.gov/pubmed/18950？662. Accessed 31 January 2014.
[32]  Kress GJ, Yamawaki N, Wokosin DL, Wickersham IR, Shepherd GMG, et al.. (2013) Convergent cortical innervation of striatal projection neurons. Nat Neurosci 16: : 665–667. Available: http://www.ncbi.nlm.nih.gov/pubmed/23666？180. Accessed 26 May 2013.
[33]  Crittenden JR, Graybiel AM (2011) Basal Ganglia Disorders Associated with Imbalances in the Striatal Striosome and Matrix Compartments. Front Neuroanat 5: : 59. Available: http://www.ncbi.nlm.nih.gov/pubmed/21941？467. Accessed 24 September 2011.
[34]  Fujiyama F, Sohn J, Nakano T, Furuta T, Nakamura KC, et al.. (2011) Exclusive and common targets of neostriatofugal projections of rat striosome neurons: a single neuron-tracing study using a viral vector. Eur J Neurosci 33: : 668–677. Available: http://www.ncbi.nlm.nih.gov/pubmed/21314？848. Accessed 12 August 2011.
[35]  Houk JC, Adams JL, Barto AG (1995) A model of how the basal ganglia generate and use neural signals that predict reinforcement. Houk JC, Davis JL, Beiser DG, editors MIT Press. Available: http://books.google.com/books?hl=en&lr=&？id=q6RThpQR_aIC&oi=fnd&pg=PA249&dq=A+mod？el+of+how+the+basal+ganglia+generate+and？+use+neural+signals+that+predict+reinfor？cement&ots=zPWtYfGo7n&sig=_hCSGDbgX1FQhU？LRxep68OL14cE.
[36]  Sheth SA, Abuelem T, Gale JT, Eskandar EN (2011) Basal ganglia neurons dynamically facilitate exploration during associative learning. J Neurosci 31: : 4878–4885. Available: http://www.pubmedcentral.nih.gov/article？render.fcgi?artid=3486636&tool=pmcentrez？&rendertype=abstract. Accessed 24 May 2013.
[37]  O'Doherty J, Dayan P, Schultz J, Deichmann R, Friston K, et al.. (2004) Dissociable roles of ventral and dorsal striatum in instrumental conditioning. Science 304: : 452–454. Available: http://www.ncbi.nlm.nih.gov/pubmed/15087？550. Accessed 18 July 2011.
[38]  Lobo M, III HC (2010) Cell type-specific loss of BDNF signaling mimics optogenetic control of cocaine reward. Science (80-) 330: : 385–390. Available: http://stke.sciencemag.org/cgi/content/f？ull/sci;330/6002/385. Accessed 26 June 2013.
[39]  Oyama K, Hernádi I, Iijima T, Tsutsui K-I (2010) Reward prediction error coding in dorsal striatal neurons. J Neurosci 30: : 11447–11457. Available: http://www.ncbi.nlm.nih.gov/pubmed/20739？566. Accessed 22 July 2011.
[40]  Lenz JD, Lobo MK (2013) Optogenetic insights into striatal function and behavior. Behav Brain Res: 1–10. Available: http://www.ncbi.nlm.nih.gov/pubmed/23628？212. Accessed 3 May 2013.
[41]  Adamantidis AR, Tsai H–C, Boutrel B, Zhang F, Stuber GD, et al.. (2011) Optogenetic interrogation of dopaminergic modulation of the multiple phases of reward-seeking behavior. J Neurosci 31: : 10829–10835. Available: http://www.pubmedcentral.nih.gov/article？render.fcgi?artid=3171183&tool=pmcentrez？&rendertype=abstract. Accessed 28 May 2013.
[42]  Szydlowski SN, Pollak Dorocic I, Planert H, Carlén M, Meletis K, et al.. (2013) Target selectivity of feedforward inhibition by striatal fast-spiking interneurons. J Neurosci 33: : 1678–1683. Available: http://www.jneurosci.org/content/33/4/16？78.abstract. Accessed 27 January 2014.
[43]  Shen W, Flajolet M, Greengard P, Surmeier DJ (2008) Dichotomous dopaminergic control of striatal synaptic plasticity. Science (80-) 321: : 848–851. Available: http://www.sciencemag.org/content/321/58？90/848.short. Accessed 28 September 2011.
[44]  Matsumoto M, Hikosaka O (2007) Lateral habenula as a source of negative reward signals in dopamine neurons. Nature 447: : 1111–1115. Available: http://www.ncbi.nlm.nih.gov/pubmed/17522？629. Accessed 24 January 2014.
[45]  Bromberg-Martin ES, Matsumoto M, Hong S, Hikosaka O (2010) A pallidus-habenula-dopamine pathway signals inferred stimulus values. J Neurophysiol 104: : 1068–1076. Available: http://www.pubmedcentral.nih.gov/article？render.fcgi?artid=2934919&tool=pmcentrez？&rendertype=abstract. Accessed 1 March 2012.
[46]  Steinberg EE, Keiflin R, Boivin JR, Witten IB, Deisseroth K, et al.. (2013) A causal link between prediction errors, dopamine neurons and learning. Nat Neurosci 16: : 966–973. Available: http://www.nature.com/doifinder/10.1038/？nn.3413. Accessed 26 May 2013.
[47]  Graybiel A, Aosaki T, Flaherty A, Kimura M (1994) The basal ganglia and adaptive motor control. Science (80-) 265: : 1826–1831. Available: http://www.sciencemag.org/content/265/51？80/1826.short. Accessed 15 March 2012.
[48]  Nambu A (2011) Somatotopic organization of the primate Basal Ganglia. Front Neuroanat 5: : 26. Available: http://www.pubmedcentral.nih.gov/article？render.fcgi?artid=3082737&tool=pmcentrez？&rendertype=abstract. Accessed 13 March 2013.
[49]  Van Zessen R, Phillips JL, Budygin EA, Stuber GD (2012) Activation of VTA GABA neurons disrupts reward consumption. Neuron 73: : 1184–1194. Available: http://www.pubmedcentral.nih.gov/article？render.fcgi?artid=3314244&tool=pmcentrez？&rendertype=abstract. Accessed 17 September 2013.
[50]  Platt ML, Glimcher PW (1999) Neural correlates of decision variables in parietal cortex. Nature 400: : 233–238. Available: http://www.ncbi.nlm.nih.gov/pubmed/10421？364. Accessed 20 January 2014.
[51]  Wang AY, Miura K, Uchida N (2013) The dorsomedial striatum encodes net expected return, critical for energizing performance vigor. Nat Neurosci 16: : 639–647. Available: http://www.ncbi.nlm.nih.gov/pubmed/23584？742. Accessed 21 May 2013.
[52]  Charlesworth JD, Warren TL, Brainard MS (2012) Covert skill learning in a cortical-basal ganglia circuit. Nature 486: : 251–255. Available: http://www.ncbi.nlm.nih.gov/pubmed/22699？618. Accessed 12 July 2012.
[53]  Ostendorf F, Liebermann D, Ploner CJ (2013) A role of the human thalamus in predicting the perceptual consequences of eye movements. Front Syst Neurosci 7: : 1–12. Available: http://www.frontiersin.org/Systems_Neuro？science/10.3389/fnsys.2013.00010/abstrac？t. Accessed 26 April 2013.
[54]  Mengual E, de las Heras S, Erro E, Lanciego JL, Giménez-Amaya JM (1999) Thalamic interaction between the input and the output systems of the basal ganglia. J Chem Neuroanat 16: : 187–200. Available: http://www.ncbi.nlm.nih.gov/pubmed/10422？738. Accessed 17 February 2014.
[55]  Sandberg A, Lansner A, Petersson KM, Ekeberg O (2002) A Bayesian attractor network with incremental learning. Network 13: : 179–194. Available: http://www.ncbi.nlm.nih.gov/pubmed/12061？419. Accessed 17 February 2014.
[56]  Berthet P, Hellgren-Kotaleski J, Lansner A (2012) Action selection performance of a reconfigurable basal ganglia inspired model with Hebbian-Bayesian Go-NoGo connectivity. Front Behav Neurosci 6: : 65. Available: http://www.pubmedcentral.nih.gov/article？render.fcgi?artid=3462417&tool=pmcentrez？&rendertype=abstract. Accessed 1 February 2013.
[57]  Yang T, Shadlen MN (2007) Probabilistic reasoning by neurons. Nature 447: : 1075–1080. Available: http://www.ncbi.nlm.nih.gov/pubmed/17546？027. Accessed 24 July 2010.
[58]  Doya K, Ishii S, Pouget A, Rao RPN (2007) Bayesian Brain. MIT Press.
[59]  Deneve S (2008) Bayesian spiking neurons I: inference. Neural Comput 20: : 91–117. Available: http://www.ncbi.nlm.nih.gov/pubmed/18045？002. Accessed 17 February 2014.
[60]  Buesing L, Bill J, Nessler B, Maass W (2011) Neural Dynamics as Sampling: A Model for Stochastic Computation in Recurrent Networks of Spiking Neurons. PLoS Comput Biol 7: : e1002211. Available: http://dx.plos.org/10.1371/journal.pcbi.？1002211. Accessed 4 November 2011.
[61]  Wickens J, Begg A, Arbuthnott G (1996) Dopamine reverses the depression of rat corticostriatal synapses which normally follows high-frequency stimulation of cortex in vitro. Neuroscience 70: : 1–5. Available: http://www.sciencedirect.com/science/art？icle/pii/030645229500436M. Accessed 20 October 2011.
[62]  Reynolds J, Wickens J (2000) Substantia nigra dopamine regulates synaptic plasticity and membrane potential fluctuations in the rat neostriatum, in vivo. Neuroscience 99: : 199–203. Available: http://www.sciencedirect.com/science/art？icle/pii/S0306452200002736. Accessed 28 September 2011.
[63]  Lerner TN, Kreitzer AC (2011) Neuromodulatory control of striatal plasticity and behavior. Curr Opin Neurobiol 21: 322–327. Available: http://www.pubmedcentral.nih.gov/article？render.fcgi?artid=3092792&tool=pmcentrez？&rendertype=abstract. Accessed 27 January 2014.

Full-Text

Contact Us

service@oalib.com

QQ:3279437679

WhatsApp +8615387084133