Report

Computational and Physiological Models Part 1 Ekaterina Lomakina Computational Psychiatry Seminar: Computational Neuropharmacology 7 March, 2014 Entropy and the theory of life “The general struggle for existence of animate beings is not a struggle for raw materials – these, for organisms, are air, water and soil, all abundantly available – nor for energy which exists in plenty in any body in the form of heat, but a struggle for [negative] entropy, which becomes available through the transition of energy from the hot sun to the cold earth.” Ludwig Boltzmann, 1875 “[...] if I had been catering for them [physicists] alone I should have let the discussion turn on free energy instead. It is the more familiar notion in this context. But this highly technical term seemed linguistically too near to energy for making the average reader alive to the contrast between the two things.” Erwin Schrödinger, 1944 Let’s go a bit more formally – 1 • The defining characteristic of biological systems is that they maintain their states and form in the face of a constantly changing environment Minimize entropy • Mathematically, this means that the probability of sensory states in which biological agent can be present must have low entropy. • Entropy is also the average self information or Minimize surprise surprise. Low surprise means that agent is likely to be in one of the very few states, so observing each of this states would cause low surprise (both emotionally and mathematically). • So to stay ‘alive’ biological agents must therefore minimize the long-term average surprise to ensure that their sensory entropy remains low. Let’s go a bit more formally – 2 • So system must avoid surprise. But how? It does not know everything what is going to happen… • System can only evaluate what is available to it: it’s sensory experience and it’s own properties, i.e. energy efficiency and robustness. • That’s where free-energy comes into play. • Thermodynamical (Helmholtz) free-energy is the work obtainable from a closed system at a constant temperature. FE = Energy + Entropy • Statistical Free-energy is a negative sum of the predictive accuracy of the model (energy of the model) and the entropy of the model. • As we show now free-energy can be seen as an upper-bound for surprise, which however system can optimize. Minimize entropy Minimize surprise Minimize free-energy And a bit more mathematically – 1 Let’s look at the log version Bayes formula, where s(t) are the sensory states and ϑ are representational parameters of the agent: Surprise! If we would have known the true generative model of the sensorium then we would have been able to compute model evidence of the model or it’s average surprise. It is also called sometimes the evidence of the agent’s self existence. And a bit more mathematically – 2 • However, the true model is almost always unavailable or at least computationally expensive to compute. Which means that brain is unlikely to be able to perform such an operation. • Instead, one can propose simpler model q and optimize it to be as similar to the true model as possible. • D is the KL divergence between proposed model q and true model p, which is always positive. • The smaller D becomes (the closer q gets to p) the lower becomes F and the closer F gets to model evidence. Thus F becomes the upper-bound for surprise. And a bit more mathematically – 3 • However, we can’t optimize D directly as it requires knowledge of the true model p. But after some magic… Expected energy (accuracy) Entropy (complexity) • This now can be efficiently optimized, using Variational Bayes technics. • They (usually) provide us with fast and efficient update and decision rules which are likely to be implementable within brain, contrary to sampling or numerical integration. And one last bit of formulas Free energy can be minimized in two ways: • By changing the mental representation (optimizing it such that it better explains data and becomes more compact) • By performing actions which would reduce potential surprise. Bayesian brain hypothesis P(ϑ|x) – posterior belief about causes Probabilistic Causality Sensations X Update beliefs • The Bayesian brain hypothesis uses Bayesian probability theory to formulate perception as a constructive process based on internal or generative models, where brain is presented as an inference machine that actively predicts and explains its sensations. • Probabilistic model generates predictions, against which sensory samples are tested to update beliefs about their causes. • The brain is an inference engine that is trying to optimize probabilistic representations of what caused its sensory input. • This optimization can be finessed using a (variational free-energy) bound on surprise. Complexity p(ϑ) – prior beliefs Accuracy p(x|ϑ) – generative model (likelihood) Bayesian brain hypothesis Two key questions in this hypothesis are: 1. How to choose the form of generative model and the choice of prior beliefs? Answer: to use hierarchical models in which the priors themselves are optimized. 2. How to choose the form of the proposed model or distribution q? Answer: it can take take any form driving the choice of optimization procedure, but the simplest assumption as it to be Gaussian (Laplace approximation). Then minimizing free energy simply explains away prediction error. Bayesian brain hypothesis • In case of Gaussian assumption the scheme is known as predictive coding. • It is a popular framework for understanding neuronal message passing among different levels of cortical hierarchies. • This scheme has been used to explain many features of early visual responses and can plausibly explain repetition suppression and mismatch responses in electrophysiology. The principle of efficient coding • The principle of efficient coding suggests that the brain optimizes the mutual information (that is, the mutual predictability) between the sensory states and its internal representation, under constraints on the efficiency of those representations. • The infomax principle says that neuronal activity should encode sensory information in an efficient and parsimonious fashion Representing variables Efficient coding Sensory states The principle of efficient coding • The infomax principle might be presented as a special case of the free-energy principle, which arises when we ignore uncertainty in probabilistic representations. • The infomax principle can be understood in terms of the decomposition of free energy into complexity and accuracy: mutual information is optimized when conditional expectations maximize accuracy (or minimize prediction error), and efficiency is assured by minimizing complexity. Complexity Accuracy Coding length Mutual information The cell assembly theory • ‘Cells that fire together wire together’. • Conditional expectations about states of the world are encoded by synaptic activity. • Learning under the free-energy principle is the the optimization of the connection strengths in hierarchical models of the sensory states. • It appears that a gradient descent on free energy is formally identical to Hebbian plasticity. The cell assembly theory • When the predictions and prediction errors are highly correlated, the connection strength increases, so that predictions can suppress prediction errors more efficiently. • Synaptic gain of prediction error units is modulated by the precision of units. • The most obvious candidates for controlling gain are classical neuromodulators like dopamine and acetylcholine. Neural Darwinism Epigenetic mechanisms Primary repertoire of neuronal connections Experience-dependent plasticity • • • • Secondary repertoire of neuronal connections • This theory focuses on how selection and reinforcement of action policies is performed within the boundaries of cell assembly theory. Only neuronal assembly which increase evolutionary value get reinforcement through the interaction with the environment (natural selection). Plasticity is thus modulated through value. Neuronal value systems reinforce connections to themselves, thereby enabling the brain to label a sensory state as valuable if, and only if, it leads to another valuable state. This theory has deep connections with reinforcement learning and related approaches in engineering, such as dynamic programming and temporal difference models. Neural Darwinism • Value is inversely proportional to surprise: the probability of an agent being in a particular state increases with the value of that state. • The evolutionary value of an agent is the negative surprise averaged over all the states it experiences, which is simply its negative entropy. • Prior expectations (that is, the primary repertoire) can prescribe a small number of attractive states with innate value. They can also affect the way world is sampled, i.e. force agent to explore until states with innate value are founded. • Neural Darwinism exploits the selective processes in order to explain brain evolution. • Free energy formulation considers the optimization of ensemble or population dynamics in terms of entropy and surprise. Complexity Priors on small amount of innate ‘good’ states + on exploration Accuracy ‘Survival’ or rate change of value Optimal control theory • • • • • • Optimal control theory describes how optimal actions should be selected to optimize expected cost. Free energy is an upper bound on expected cost. According to the principle of optimality cost is the rate of change of value, which depends on changes in sensory states. Optimized policy ensures that the next state is the most valuable of the available states. Priors specify small amount of fixed-point attractors, and when the states arrive at the fixed point, value will stop changing and cost will be minimized. Additional priors on motion through state space enforce exploration until an attractive state is found. Action under the free-energy principle is meant to suppress sensory prediction errors that depend on predicted (expected or desired) movement trajectories. Complexity Priors on small amount of fixed-point attractors where system should arrive + on motion Accuracy Rate of change of value Overview of free-energy principle • Many global theories of brain function can be united under a free energy principle. • The commonality is that brain optimizes a (free-energy) bound on surprise or its complement, value. • This manifests as perception (so as to change predictions) or action (so as to change the sensations that are predicted). • Crucially, these predictions depend on prior expectations (that furnish policies), which are optimized at different (somatic and evolutionary) timescales and define what is valuable. Dopamine as a prediction error encoder • Measures show that dopamine sensitively react to the reward itself or to the conditional stimuli predicting reward, when such a connection is learnt. • However dopamine level decreases in case of lack of predicted reward. • That gave rise to the hypothesis that dopamine encodes prediction error. • Optimal control theory might provide a framework to explain. Temporal difference algorithm • • • • • The computational goal of learning is to maximize expected discounted reward V (value function proportional to surprise). It can be computed dynamically knowing rewards at the previous time points. Prediction error δ (TD error) can be defined as a linear combination of reward and change in surprise. M1 and M2 are two cortical modalities which input (as a derivative of value V(t)) arrives at VTA. Reward r(t) also converges on the VTA. VTA output as a prediction error is taken as a simple linear sum informing structures constructing the prediction. Representing a stimulus through time • The ability to predict in time within the model is encoded as events happening at the different time points being different stimuli. • The predicted value is then modeled as a weighted linear combination of the sensory input. • The weights are updated using the information about the prediction error. Simulations • The conditional stimuli were presented at time step 10 and time step 20 followed by reward on time step 60. The prediction of the model. • Absence of reward on one of the intermediate trials causes a large negative fluctuation of the prediction error. • However overall model is able to learn well the dependency between conditional stimulus and reward, while also blocking the repetition of redundant in terms of information secondary stimulus. • The behavior of the prediction error mimics accuratly the measured dopamine response in monkeys in similar situation. Input Perception Volatility Probability Learning Hierarchical Gaussian filtering • One particular model within Bayesian Brain hypothesis is Hierarchical Gaussian filtering. • It consists of a hierarchical Bayesian model of learning through perception. • The response model can deal with states and inputs that are discrete or continuous, uni- or multivariate, and as well as with deterministic and probabilistic relationships between environment and perception. • Parameters of the model can account for individual differences between agents and can be used to simulate and explain maladaptive behavior. Hierarchical Gaussian filtering • • • • • This is the simplest example (with univariate, binary deterministic response model). Each layer of the model performs Gaussian random walk with parameters guiding layer coupling and width of the Gaussian walks. As many layers as needed can be added on top. Priors on the parameters χ = {κ, ω, ϑ} allow to perform full Bayesian inference. Inverting this model corresponds to optimizing the posterior densities over the unknown (hidden) states x = {x1, x2, x3} and parameters χ. This corresponds to perceptual inference and learning, respectively. Inversion under free energy principle • Exact inference of such a model would involve expensive numerical optimization. • Instead was used Variational Bayesian approach (which involves minimization of free energy). • The key assumption which was made is a factorization of the recognition distribution q (mean-field approximation) Update equations The resulting update equations are not only efficient and easy to compute but also resemble results from the field of reinforcement learning. Rescorla-Wagner model: prediction(k) - prediction(k-1) = learning rate x prediction error Precision updates • Variances σ1, σ2 and σ3 are also updated on every step in form of the precision (inverse variance) • The precision updates account for two type of uncertainty ‘informational’ (the lack of knowledge about x2) and ‘environmental’ (the volatility on the third level). • It has been proposed that dopamine might encode the value of prediction error, i.e., the precision-weighting of prediction errors. This is encoded by parameters κ and ω. Meaning and the effect of the parameters Reference scenario: ϑ = 0.5, ω = −2.2, κ = 1.4 Meaning and the effect of the parameters Reduced ϑ = 0.05 (unchanged ω = −2.2, κ = 1.4) Meaning and the effect of the parameters Reduced ω = −4 (unchanged ϑ = 0.5, κ = 1.4) Meaning and the effect of the parameters Reduced κ = 0.2 (unchanged ϑ = 0.5, ω = −2.2) Potential for real data Conventional observed behavioural data fail to detect difference between subjects However computational model reveal striking difference of the underlying mechanisms fraction of correct responses success 1 [running average] 0.5 0 20 40 60 80 100 120 140 160 trial-wise reward score 10 [running average] 5 0 RT [ms] 20 20 40 60 80 100 140 160 reaction time [s] 2 10 0 120 20 40 60 80 trials 100 120 4 140 6 160 S_3364P S_2978P S_2947P S_3040P S_2411T S_2855P S_1021T S_3232P S_3515P S_3327P S_3504P S_2862P S_3031P 1 0.8 healthy participant 0 50 4 volatility 2 1 0 -2 0.8 -4 0 50 4 1 2 probability 0 0.5 -2 -4 0 0 50 trials 1 1 0.5 prodromal schizophrenia 0.8 0 50 0 4 50 20 volatility 1 0 -2 0.8 -40 50 0 50 4 1 probability 2 0 0.5 -2 -4 00 50 0 50 trials 1 100 150 100 150 100 150 100 100 150 150 100 100 150 150 100 100 150 150 100 150 0.5 0 0 50 Outcome • Using free-energy principle we derive fast & efficient update equations which can be implemented by brain. • The resulting update equations present a clear connection to the field of reinforcement learning. • Parameterization of these equations accounts for individual differences and can model the whole variety of maladaptive behavior. • There is evidence that some of the parameters may correspond to neuromodulators, in particular dopamine. Conclusions • Computational models inspired by expert knowledge about the brain can provide us with powerful insights how brain works in healthy and maladaptive ways. • Free-energy principle provides a powerful framework, which generalizes many of the existent theories about the brain functioning. • However, “all models are wrong but some are useful”. To prove the correctness of model we have to see whether it predicts real data – more next week. • Neuromodulators can be predicted to explain certain parameters of the models. I.e. dopamine seems to play key role in the processes of reward-driven learning. Studies with pharmacological manipulations together with computational models can provide a deeper mechanistical insight on the particular role of it.