Negative Valence

[rcoreilly]

Here's the discussion thread for trying to figure out how to integrate negative valence outcomes (pain, bad taste / smell, predators, negative social outcomes, etc) into the existing BOA model in a better way.

So far, BOA has been built around positive-valence (appetitive) outcomes that drive goal engaged approach behavior, within an overall consumatory (e.g., foraging) framework free from existential threats ("ordinary" life). This BOA framework naturally suggests two different categories of negative valence signals, which align with the central ACC (action cost) and OFC (outcome value) distinction, and logically affect different aspects of decision making. However, existential-level decision making is likely qualitatively different in important ways, as discussed subsequently.

• Action costs: effort (time), pain (physical impact, etc), frustration, etc (what else?)

  • These are a function of the dlPFC (ALM) action plan, and are optimized by choosing the best plan to achieve a given outcome, minimizing costs.
  • ACCcost can learn about these costs in relation to different action plans, and ACCutil can integrate this cost estimate relative to OFCval value, in the course of deciding on an overall goal state, in the same way that effort costs are currently processed.
  • Do all of these factors sum together (pan + effort) into the same kind of multiplicative discount factor, like we do with effort currently?
  • Time-wise, these negatives arise during the course of performing an action plan (as opposed to being a discrete outcome at the goal end state) -- they are more continuous, ongoing states instead of discrete outcomes. Presumably there is an online tracking and integration of these costs, which can feed into an "abandon current goal" action (LHb DipSum in the current model).
  • It is also important to capture the core Rubicon dissociation that costs are down-weighted during online performance, and then emphasized during subsequent goal selection decision making. Given that LHb is distinct from ACC inputs during decision making, there is a nice anatomical dissociation here but we need to do more to explore this dynamic in the model.

• Negative outcomes: rotten / sour / bitter taste, negative social outcomes, etc (what else?)

  • These modulate the outcome value of a possible positive approach goal, and should be integrated within the BLA / OFC assessment of outcome value.
  • For outcomes directly associated with an otherwise positive outcome (bad taste), the negative can simply compete / subtract from the value of that outcome directly, within the existing PVLV-based BLA framework, which has separate pathways for negative USs vs. positive.
  • BOA has not extended this negative valence opponency into OFC -- does OFC even represent these negative outcomes, or just the net subtraction from the positive? Maybe OFC is only representing the net positive goals, to drive goal-engaged behavior?

• Existential threat: at some point, pain transitions into mortal threat, at which point it should probably be more than a discount factor. This is a discrete outcome, not a continuous cost. It should drive active avoidance behavior directly instead of just representing a cost.

  • Probably this is a separate pathway through the same brain systems, that is competing with approach goals and driving avoidance goals. Biologically, threat will engage the sympathetic nervous system, in a way that effort or low-level pain will not.
  • How are avoidance goals represented? Approach is easy: follow the gradient toward the thing you want. How does avoidance work? Invert the gradient? This is highly underspecified relative to approach.
  • What else goes in this category? Probably anything long-lasting that affects overall life quality: changes in home territory (nest, house, etc), social standing, social group dynamics (war, etc), mate selection.
  • These have both positive and negative valence components, and go beyond ordinary daily survival behavior within an existing "status quo".
  • The positive aspect could function like an approach goal, but it is much more loaded in terms of costs: if you want to claim a new bigger territory, you likely need to fight / attack. The risks are not just discount factors on the plan, but major life-altering negative outcomes, that need to be carefully weighed.
  • Whereas ordinary appetitive-focused decision making is likely governed by RL-like value optimization, existential decision making is unlikely to be informed by a history of prior experience (these are once-in-a-lifetime kinds of scenarios) and are governed more by innate, sympathetic arousal-driven threat / opportunity assessments. This is where rationality goes out the window, as evident in political decision making when people get their existential threat system activated.
  • We could decide to postpone consideration of this level of function?

Biological data

The amygdala is the core system for all of the "valenced" signals, and most research has traditionally focused on negative valence instead of positive. Having all of these signals going through the amygdala (BLA) allows for competitive "attention" dynamic so that the most important ones dominate: if there are existential issues at hand, those will presumably out-compete others.

To accommodate the above factorization, we likely need to organize BLA into existential vs. ordinary categories, and route these into different pathways in other brain systems.

[thazy]

Summary Commentary

CAVEAT: To be crystal clear re: my use of the term “negative valence”, I want to point out the distinction between the mere omission of a positive US in the context of APPETITIVE processing (excluded) — VERSUS everything else negative relevant for behavior (my definition for “negative valence”). I feel this bears pointing out because the literature in RL and behavioral economics, and even much of neuroscience is sloppy about this distinction and actually generally uses reward omission as the paradigmatic case of negative outcomes — which IMO it should NOT be considered. Instead, reward omission should be considered EITHER just a part of the world’s being probabilistic, OR a potential signal for me to pay more attention to what I’m doing or to consider that my world’s contingencies have changed.

The 3-way taxonomy @randy outlines makes a lot of sense to me. At the highest level is the distinction between the negative outcomes that serve to define processing in an AVERSIVE CONTEXT versus negative events/experiences/considerations that accompany processing in what is otherwise an APPETITIVE CONTEXT. The former might be defined as processing where the dominating consideration is the escape/avoidance/mitigation of the occurrence of seriously bad events, e.g., @randy’s “existential threats” but also other really bad stuff like substantial loss of property, etc. as a human/economic example, while the latter might be defined as less serious negative experiences that occur within the context of what is otherwise an appetitive-dominant situation. One way to define a distinction between these two might be in terms of the magnitude of the stakes involved for the positive vs. negative experiences. Appetitive context means there’s a lot to gain while the negative stuff is less consequential - and vice versa.

• ASIDE: Of course, there are also other cases, particularly the one in which the stakes are high in both directions as when a very hungry prey animal has to venture out for food in the presence of a potential predator. See THE DUAL HIGH-STAKES EXPERIMENTAL PARADIGM.

Within the appetitive context, it also seems important to distinguish between negative stuff like the unpleasantness of physical and/or mental effort that comes with pursuing appetitive goals — VERSUS negative stuff that happens “out there in the world” as a result of my appetitive behavior, either along the way to goal-realization, or that ALSO occurs in conjunction with goal-the realization; the former are kinda hurdles to be overcome, the latter “side effects” of the primary (desired) outcome. A possible dimension for distinguishing between these two kinds of negative events might be the sensory (or cognitive?) source of the information: for the former the source seems to (always?) be the subject’s own body (or brain?), i.e., it is interoceptive; while for the latter the source seems to be events that happen in the world (exteroceptive). This distinction could be useful/relevant because these processing modes tend to involve separate neural structures and could thus serve to motivate model architecture/organization.

Addressing each of the three taxonomical branches in turn:

APPETITIVE VS. AVERSIVE CONTEXT PROCESSING AS LEFT- VS. RIGHT-DOMINANT LATERALIZED PROCESSING: I believe that the long-standing distinction made between appetitive-as-left-hemisphere vs. aversive-as-right still holds up empirically (BIS/BAS and all that) although it tends to be largely underemphasized in much of the cognitive neuroscience literature. Note that such a framing does not need to be to the exclusion of other lateralization schemes commonly proposed: left-as-on-task vs. right-as-monitoring; left-as-primary-task vs. right-as-secondary, etc.

• TODO: Assuming that others agree that this might be a viable modeling direction I will do lit search to examine current thinking about the left-appetitive vs. right-aversive hemispheric specialization. See also RIGHT- VS. LEFT-LATERALIZATION OF OFC(?).

OBLIGATORY COST/INTEROCEPTIVE(?) PROCESSING AS VMPFC (mostly excludes OFC): Generally, the VMPFC constitutes a large chunk of neural real estate that is not yet very well characterized, especially in primates. A lot of VMPFC seems to process interoceptive and/or affectively-charged information that is central to motivation, etc. By hypothesis, my mental model for VMPFC has several subareas, each specialized for processing various dimensions of “cost” associated with various categories and anticipated instantiations of specific actions, partially hierarchically organized so as to feed into a kind of integrated utility area (ACCutil?) where the expected outcome value of contemplated actions and their accompanying costs combine to form expected utility reps.

• TODO: I will re-visit the old OngurPrice primate OFC/VMPFC neuroanatomy papers + forward search.
• TODO: Also, there was that grant proposal document @randy prepared with a detailed model of VMPFC specialization that I will dig that up and forward-search/update from there.

Regarding how to combine multiple cost dimensions, including how to weigh them relative to one another, Prospect Theory has generally used different exponent parameters for discounting positive (benefits) vs. negative (costs) outcomes as a function of their respective magnitude. I am aware, however, if and/or how they handle multiple cost dimensions. There is a body of work in behavioral economics having to do with multi-featural decision making, typically involving consumers’ purchasing behavior, that could be relevant here as well. One approach might be for each subject to have a vector of such exponent-parameters, one for each cost dimension, along with scaling parameters that reflect the relative strength of each dimension for that subject and combine them linearly.

• TODO: If this seems like a fruitful direction, I will search the behavioral economics and related literature for how multiple cost dimensions might be combined in a formal, mathematical way so as to inform decisions about neural architecture.

Also, I have only recently come to realize just how intimately integrated the VMPFC, etc. are with the HC/MTL and associated subcortical structures (anterior thalamus, nuc. reuniens, ventral striatum-defined BG pathways, etc.). I’ve come to think of this system (also including OFC) as a kind of “rodent brain” (because that seems to be all they have cognitively) that serves as kind of core system “around which” the additional cognitive processing substrates are layered on in the primate (i.e., subsumption).

• TODO: (tentative) I will pursue lit search on current thinking re: VMPFC-HC/MTL interactions as a “system” — just how intimately tied together are these functionally, etc.

Finally, Josh Brown recently published an update to his PRO model of ACC function, which also seems relevant here. Although the original PRO had ACC doing only outcomes, and predictions thereof, he may have since added the cost side. In any case, it’s worth looking at his latest.

• TODO: I will review the latest PRO model paper

NEGATIVE EXTERNAL EVENTS IN APPETITIVE CONTEXT HANDLED BY OFC: To a first approximation the OFC seems to serve as a substrate for a kind of forward-looking working memory goal-state representational space that serves to motivate behavior towards affectively-charged desired states of the world, e.g., searching/working for food when hungry, etc. While there was some early thought that positive vs. negative valence was represented along a medial-to-lateral gradient in OFC (refs in FrankClaus06?), more recent thinking is that appetitive (food) vs. aversive (electric shock) USs are represented in the OFC in a mosaic-like pattern, and that the medial-lateral gradient may contribute to the OFC’s serving as a kind of “task-representational-space” perhaps extending into adjacent VLPFC in primates (my hypothesis).

• TODO: I will perform lit search re: current thinking regarding the representation of appetitive vs. aversive USs in the OFC

ADDITIONAL THOUGHTS:

RIGHT- VS. LEFT-LATERALIZATION IN OFC(?): We know from the motor system that right vs. left lateralization exhibits a ubiquitous pattern whereby contralateral action is dominant but ipsilateral action is always also represented in each hemisphere and that there is extensive interhemispheric communication. Speculatively, a potentially useful hypothesis re: OFC functional organization might be that while both appetitive and aversive US outcomes are represented in each hemisphere, the left OFC is appetitive-dominant while the right is aversive-dominant (analogous to contra- vs. ipsi- in the motor system). Such an arrangement would allow for the representation of relatively low-stakes negative outcomes, like occasional thorn pricks, etc., to be represented in the left hemisphere alongside a currently dominant appetitive goal-state, like foraging on a berry bush. That is, while I stay in persistent appetitive-processing-mode, I am nonetheless able to “keep in mind” the need to keep the thorn pricking to a minimum.

• TODO: I will also consider during the above OFC-centric lit search whether there’s anything new that might bear on the left-appetitive- vs. right-aversive-dominant processing story as it might pertain to OFC (and VMPFC?).

PREDATOR-PREY INTERACTIONS: For what seems like obvious reasons, predator-prey interactions have exerted an especially powerful influence on evolution. The prey’s perspective, in particular, can be particularly informative as it serves to highlight the importance of distinguishing between processing under aversive vs. appetitive context. And, the aversive context here can be especially enlightening.

Consider a prey that encounters a predator in its immediate environment. As a first priority it must avoid capture, which entails first of all maintaining its attention on the predator and its location at all times, and then continually working to keep optimal practical distance between itself and the predator. Think of the ubiquitous case of one person chasing another with a table between them. Something to notice here is how the predator serves as a kind of anti-goal for the prey — I’ve got to maintain focus on the predator in order to keep my distance; that is, I CANNOT afford to treat the predator as something unpleasant and so put it out of my mind. From an OFC-as-working-memory-for-goals perspective, the maintenance of the predator rep and its location has to be my main focus. This is of course all happening in an aversive context.

On the other hand, going through life in perpetual fear/avoidance mode is a terrible way to live. Thus, if I can identify a state-of-the-world for myself that precludes my predator’s access to me then I don’t have to focus on the predator for the foreseeable future. In the behavioral literature, this is studied by so-called safety-signal or security learning, and such learned safe states acquire strong positive valence — think about being inside a shelter all warm and dry while a cold rain pours outside. An important thing to note here is that once such safe states are learning/identified, their pursuit can serve to convert an aversive context situation into an appetitive one — instead of perpetually having to avoid the predator, I can instead pursue the quasi-appetitive goal of finding somewhere that I know they can’t reach me for whatever reason and then I can forget about them. Problem solved!

• TODO: There has been considerable work showing that the posterior dorsomedial striatum (pDMS) appears to be specialized for so-called safety-signal learning, along with its cortical and subcortical processing partners. If desired, I will review the current state of knowledge on this body of work with an eye towards incorporating safety signal learning into the model architecture.

THE DUAL HIGH-STAKES EXPERIMENTAL PARADIGM: There is a really nice paradigm, I think from the behavioral ecology literature, in which rodents are required to venture out from a safe area at one end of the experimental environment in order to forage for food placed at various points in an open field that has a predator residing at the opposite end of the environment. The predator’s range is limited somehow such that it’s speed relative to the subject means there is a danger gradient that goes from safe foraging (near the subject’s safe haven) towards the predator’s lair. This allows for quantitative analysis of individual subject’s behavior in order to explore causal factors affecting risk-taking behavior, individual differences, etc.

• TODO: If interested, I can dig up references regarding this paradigm that might be used as a target for (future?) modeling work.

REQUEST: @randy and anyone: Please indicate which of the above-mentioned lit search questions you’d like me to prioritize; otherwise, I will follow my own muse. Also, let me know if I missed any issues brought up in the post above, or if there are additional, specific questions you’d like me to address.

[randy]

Just a heads-up, you're tagging the wrong user...

[rcoreilly]

I will perform lit search re: current thinking regarding the representation of appetitive vs. aversive USs in the OFC

this would be particularly helpful soon -- specifically in relation to ACC cost representations. Here's one plausible model:

• BLAneg <-> USneg <-> ACCcost (summary of time/effort and overt negative outcomes)

Alternatively, BLAneg might (also) have short-circuit directly into ACCcost?

[rcoreilly]

@thazy tag. tag me back at @rcoreilly so as to not bug "randy" :)

[thazy]

Specific Inline Comments

…Action costs... what else

• Add: opportunity costs, uncertainty costs (i. epistemic, ii. aleatoric)

…dlPFC (ALM)

• ALM is more like M2, not really “dlPFC”. Is prelimbic (PL), maybe even IL(?), a consideration here for rodents?

do all these factors sum together

• Great question! There is sorta the mathematical framework, and then the neural implementation thereof. Certainly lots of individual differences to consider as well. Prospect Theory formalisms may be informative here. See main discussion above under OBLIGATORY COST/INTERCEPTIVE PROCESSING AS VMPFC
• typo: pan -> pain

…Rubicon

• I'm not sure the online discounting is always the case. Often it is, but sometimes it seems like the effort involved is MORE than I anticipated it to be.
• There is also the issue of anticipated effort for some project well in advance, say next week, versus how it can sometimes feel immediately before. I.e., the whole present self, future self distinction.

…continuous, ongoing

• These continuously cumulative costs do seem a significant challenge in that they must be “factored in” on an ongoing basis at performance/runtime, but then get integrated into a kind of scalar value for long-term storage to be used for future decisions. Currently reflected in some sort of ACC vs. LHb division-of-labor, I guess?

…Negative outcomes

• There is also a probabilistic case here, e.g., same action, sometimes food, sometimes mild shock

…negative USs vs. positive

• As implemented in PVLV, these were originally conceived in terms of dominant, context-defining US outcomes. In the case of concurrent activations, the idea would be for the same context sometimes I get food, sometimes a shock. Thus, I am developing co-existing expectations based on the occurrence of a given CS.

…does OFC even

• OFC definitely represents negative outcomes in the traditional aversive conditioning sense. See main discussion above under NEGATIVE SIDE-EFFECTS/ACCOMPANYING OUTCOMES PROCESSING AS OFC and RIGHT- VS. LEFT-LATERALIZATION IN OFC(?)

…sympathetic nervous system

• Effort, for example, definitely activates the sympathetic nervous system, even the anticipation thereof activates it. See main discussion above under APPETITIVE VS. AVERSIVE CONTEXT PROCESSING AS LEFT- VS. RIGHT-DOMINANT LATERALIZED PROCESSING for an idea for implementing the “in a way that…” difference.

…avoidance goals represented…invert the gradient

• I think this basically has to be the way it works while under the aversive processing mode. See main discussion above under PREDATOR-PREY INTERACTIONS for a concrete example of how “avoidance goals” and an “inverted gradient” kinda has to work.

• …What else in this category

• In general, my sense is that rather than necessary having an exhaustive list to start, if we end up with a robust architecture motif for the whole existential threat category we ought to be able to add individual cases on a one-by-one basis.

…”status quo”

• I don’t quite follow this bullet.

…once in a lifetime

• Definitely! I’ve come to think in terms of evolution having come to use innate-like threat representations that serve as quasi-USs in order to avoid having to actually experience death in order to learn from it.

…different pathways in other brain systems

• BLA does definitely communicate with VMPFC (ACC, etc.) quite extensively, i.e., not just OFC.

[rcoreilly]

Basic USneg, BLAneg, OFCneg, ACCcost etc connectivity and logic in BOA

Here's some notes on basic computational logic for organizing layers within the first two "approach goal compatible" forms of negative US outcomes.

Graded USs

• whereas current USpos are binary, USneg needs to be graded, because their whole point is to provide a graded modulation against positives. Can switch to Drives-like layers for USs with popcodes. Pos can still use binary or start using graded..

Gate OFCneg along with OFCus as part of the goal rep

• maintain expected negative as well as positive outcomes. makes a lot of sense for Effort, pain etc kinds of things.

Action costs: Effort = 1st pool in USneg

summary of discussion from today:

• OFCus -> OFCposUS

• OFCval -> OFCposVal

• add OFCnegUS

• ACCcost -> ACCnegVal

• add crossover point for negative outcome where it counts as a overall negative vs. discount factor

Existing Effort logic provides a good template for how to generalize to all USneg:

• With graded US reps, 1st USneg pool replaces the current Effort / EffortP layer.
• This then feeds into BLANegAcqD2 (1st pool) -- will get a direct estimate of amount of effort there.
• OFCneg (similar to OFCus) (1st pool) will take over job of predicting effort costs using standard CT / PTp predictive learning, instead of having ACCcost do that.
• ACCcost is single summary of all costs (i.e. predicts OFCneg), in same way that OFCval is summary of all OFCus positive values.
• If you fail to achieve goal (give up), you already have the effort cost sitting there -- no need to add additional USneg input etc.

This same logic works well for pain (e.g., bumping into walls in eboa), social negative outcomes, etc.

CS -> BLANeg

• Same as CS -> BLAPos, BLA can learn about stimuli that predict negUS outcomes.

• But learning logic is a bit different: no discrete goal-satisfying phasic US outcomes -- instead more continuous, graded signals, without big phasic neuromodulatory signals..

• big salient phasic changes (sudden shock after tone, etc) should work same as positive phasic cases..

• but how do graded cases work? learn at end (pos US or give up) about accumulated Effort / Pain? learn on error pred - actual * abs(DA)?

• Extinction is same in pushing down Acq, context sensitive.

VsMatrix

• BLANegAcq -> VsMatrixNo: negative CS->US drives Nogo gating

  • not connecting BLANegExt -> VsMatrixGo -- double negative but not clear it drives Go.. worse boa perf.

• usNeg -> VsMatrixNo: not doing this -- VsMatrix is for positive goal selection; usPos -> VsMatrix drives US-goal clearing; not clear what role usNeg might play.

OFCneg -> ACCcost

Model OFCneg, ACCcost on OFCus, OFCval layers -- same basic logic, connectivity with corresponding BLAneg etc.

[rcoreilly]

Bidirectional (positive / negative) net value as a function of total reward (pos US) and total cost (neg US):

Goals:

• When cost is high and reward is low, result is negative, proportional to cost (but with some impact of reward)
• When reward is high and cost is low, result is positive, proportional to reward (but with some impact of cost)
• Avoid having lots of values around 0 -- those are basically invisible from network. Typical values are likely to be around .5 on both pos and neg side, so subtraction would just give small values around 0, whereas discount multiplication gives values around .25 if each is .5: .25 = .5 * (1 - .5)
• Normalized units
• Effort is open-ended sum that increments step-by-step; probably same thing with pain. Do same for all negUS?

    // [i] below is each individual US -- aggregate across all then normalize!
    pvNeg= 1 - (1 / (1 + sum(weight[i] * negUSRaw[i]))) // raw = unbounded linear sum, negUS is normalized value across all
    pvPos = 1 - (1 / (1 + sum(weight[i] * drive[i] * posUSRaw[i]))) // same

    var pvNet float32 // pv is overall primary value -- + / - 
    if pvPos > threshold * pvNeg { // threshold [default 1] = factor for relationship between pos / neg
        // todo: threshold can be adaptive -- depending on longer-term rate of positive reward etc..
	pv = pvPos * (1-pvNeg) // negative discounts against posUS 
    } else {
        pv = -pvNeg * (1 - pvPos) // converse
    }

    da = pv  // da = dopamine; + = burst above baseline tonic; - = dip below baseline tonic

[rcoreilly]

this kind of non-linear relative computation looks ideal for what the LHb should be doing and why you need an LHb in the first place, to implement this kind of thing. will need to fold this into the LHb code in the PVLV model.

Also, need to figure out when to do this! presumably VS inputs to LHb are inhibited outside of ACh disinhib at PV, so this all happens at the time of US, or give-up.

otherwise the intervening DA dynamics are all in the DLS and goverened by the local motor-learning dynamics?

[rcoreilly]

graded vs. discrete USs

The new USneg is predicated on the US being graded, and accumulating over time, with effort and pain being the prototypes. Pain may come in more discrete bursts but is also cumulative in weighing whether to give up.

Thus, when you drive a new USneg input to PVLV, it adds to existing USnegRaw and updates a normalized value representation of that for each USneg pool to represent (a separate weighting factor is needed there).

There is a question as to whether USpos should also be more cumulative and graded? e.g., as you eat, at some point the cumulative effect gets over some threshold?? For now, let's keep it discrete, given that we need a discrete event ultimately to count as goal completion, but keep this issue in mind going forward.

[rcoreilly]

rationale for why we need to use globals in GPU-friendly way: basically need to read / write lots of values to / from different layers on a cycle-by-cycle basis.

There are a few vars that don't need to be visible but more that are -- so on balance, better to just have in one place.

Also, we run 10 cycles of updating entirely on the GPU without any copying back/ forth to CPU, so all of the computation has to also run on the GPU.

[rcoreilly]

By shifting the VSPatch to be predictive from previous trial to next trial, we can make the entire PV side of the computation computable at the start of a trial: USs, etc are all available at that time. This is done in PVLVStepStart now, called on CPU in sim's ApplyPVLV method. The results of this computation however still need to be GPU accessible in the Globals.

If this all holds up, we can probably reorganize so PVLV and pvlv.go no longer lives in Context, and thus does not need to be GPU computable. The remaining cycle-by-cycle VTA.DA computation does not require any params.

Where does it go? we don't have another global location! could just live on the Network presumably. just change the Sim ApplyPVLV and ConfigPVLV to operate on it there.

[rcoreilly]

This major reorg is now complete and it really cleans up the code organization, which was very messy in many ways owing to an initial PVLV import and lots of stuff accumulated since then without a cleanup.

VTALayer does have VTA params for gain factors on LV (CeM) and PV.

[rcoreilly]

ps. fixed the dreaded FloatFromInt bug (net negative DA gain) and it also wasn't properly resetting globals in NewRun.

[rcoreilly]

When a negative US becomes its own outcome: ACh salience

Key point: ACh gates (inhibits / disinhibits) activity in VS (ventral striatum), so that goal gating only happens on "salient" events, such as positive US onset, or CSs. The new LHb code has this logic.

Thus, the ACh response determines whether a negative US is sufficient to drive VS gating on its own:

• If a negative US is slowly accumulating like effort or muscle pain, it doesn't drive VS gating, and the accumulated amount is only "registered" in terms of dopamine when another salient US happens.

• However, if there is a salient, phasic negative US like a sudden bump, shock, etc, this will activate the ACh and drive VS gating so the negative DA is actually registered and drives learning etc.

• If the slowly accumulating negUS exceeds a threshold, then it becomes sufficient to drive VS gating by itself. This is the GiveUp situation. Thus, in some ways, a single salient negUS drives an "instant GiveUp" dynamic.

[rcoreilly]

Negative CS Learning and avoidance

• If NegUS gets over threshold and drives negative DA, then the inverted D2 dynamic for BLANegAcqD2 kicks in and it learns to "recognize" any associated CS as having a negative outcome.

• For graded NegUS (e.g., effort) that doesn't trigger negative DA -- what kind of CS learning happens in this case? There is positive DA, so BLANegAcqD2 doesn't learn. The only way BLA could learn is when the negUS is stronger than usual, so that there is a negative RPE DA dip relative to expectations..

If there is a negCS -> BLANegAcqD2, it should drive VSMtxNo to avoid approaching that stim again.

It is also possible that negCS -> BLAPosAcqD1 can learn to activate more VSMtxNo if there are some cases of negative RPE DA.

Should think about this further.

[rcoreilly]

Effort vs Urgency

Now that Effort is a "proper" negative US, it always activates BLANegAcq, like any other neg US. This then competes with novelty activity at the start in BOA, preventing the existing default model from engaging the curiosity drive.

We might want to make some kind of exception here, but it also suggests an important clarification between urgency and effort: they should be mutually exclusive "bins" that accumulate the same quantity ("effort") depending on whether the goal is engaged or not:

• Urgency = effort is expended in the absence of a goal-engaged state, resulting in increased probability of engaging (VSMatrixGo gating) as effort / time passes. Unlike current impl, Urgency should reset at the Go firing, and only start accumulating again after US / GiveUp.

• Effort as USneg[0] = effort expended during the goal-engaged state. It will be absent at the start so it won't interfere with curiosity gating, and will just accumulate greater BLANeg activity over time, which then drives OFCNeg to encode the prediction of accumulated negative for the resulting Goal state. This BLA -> OFC path is the established route for OFC activation, so probably makes sense also in this context.

This will fix the weird situation where Urgency could interfere with an already-engaged state (@garymm pointed out this problem) and will make Go firing much more robust, as it was originally intended to.

[rcoreilly]

The logic of Giving Up

The following negative valence signals drive giving up on an engaged goal:

• If a negative US input on any trial exceeds NegUSOutcomeThr (see Negative Valence #221 (comment) above) then it becomes a primary US outcome and drives VS gating, etc.

• If the accumulated PVneg (normalized summed negative outcomes) exceeds the expected positive reward (PVpos), then it is time to give up. This could be modulated by future estimates in addition to current accumulated values -- the PVpos level always has to be estimated until it is encountered, and the rate of PVneg may be variable and if estimated to decrease significantly ("just over the next little hill") then it may make sense to persist.

  • This is related to the marginal value theorem (MVT) in foraging (Charnov, 1976; Stephens & Krebs, 1986; Hayden et al, 2011), where it is optimal to give up on a patch when the rate of return goes below the average expected rate of return. Areas in the OFC and ACC have been associated with these foraging decision variables (dACC representing cost, OFC / vmPFC representing benefits of current patch (Kolling et al, 2012), compatible with the overall BOA model.
  • The existing model just uses raw effort levels, but we need to transition to PVneg normalized values to include all sources of negative outcomes (accumulated pain etc).

• If the VSPatch predicts a positive outcome and it doesn't occur, this drives DA dips via LHb -- when these accumulate it suggests that the expected outcome is not coming.

It is possible to integrate the first two neg-US based values into a single threshold, but the ACh salience mechanism for strong sudden-onset USs suggests that they might engage distinct mechanisms, and it is simpler to set thresholds on single US values rather than deal with the complexities of normalized values.

The VSPatch mechanism is distinct because of the way that we integrate negative and positive PV values: first there is a comparison between PVpos and PVneg, and if pos > neg, then neg discounts pos (and vice-versa), and then VSPatch only operates in relation to the PVpos values. So it isn't on the same terms as neg directly, and functionally, it is more about evaluating the positive side of the equation (whether it will show up or not) vs. the negative side, so it makes sense to treat it separately.

However, we can discount the positive PVpos as a function of the accumulated VSPatch activity, in comparison with the PVneg, to enable both factors to accumulate and work together to drive giving up. So, VSPatch reduces PVpos and thus accelerates the chance of PVneg driving a switch.

Stochasticity is also critical for this process: previously had some randomness in the effort max values -- much better to just compute a probability of giving up and roll the dice accordingly. Need a gain-tuned logistic function around the threshold value, to parameterize how noisy vs. deterministic this is.

Finally, the full "bicameral BOA" framework is likely necessary to enable integration and projection of the value of alternative options in light of accumulating costs on the dominant engaged one, but we can implement simpler heuristics.

Implementation

• First, if a positive US outcome is actually present, then there is no question of giving up, and the situation is evaluated as it stands, generating the relevant computed DA levels.

• Next, if any USneg > NegUSOutcomeThr, this drives a quick ACh salience driven VS gating with negative outcome with no positive US outcome, which also just drives standard computed DA values.

• Only if neither actual outcome is present, evaluate the accumulated PVneg vs. PVpos (optionally use decoded predicted USposP values to computed expected pos outcome), using separately accumulated VSpatch activities over time to discount PVpos (subtractively presumably -- it acts subtractively in the DA), and stochastically compute the give up bool.

[rcoreilly]

Expected PVpos for give up computation

Expected PVpos is tricky -- can leverage lots of actual network state predictions. Will iterate on those.

• Need to discount by current drives, which will automatically make novel US less robust to extensive exploration if drive is lower (replacing effort NovelMax param)

• Ultimately will want to use a network impl with competitive dynamics to directly leverage network activations -- again likely in bicameral model.