New notebook: causal inference with MMM's #1032
Conversation
|
Check out this pull request on See visual diffs & provide feedback on Jupyter Notebooks. Powered by ReviewNB |
drbenvincent
changed the title
|
@drbenvincent What if we do a train test split (before and after intervention) and generate predictions and counterfactuals as https://www.pymc-marketing.io/en/latest/notebooks/mmm/mmm_time_slice_cross_validation.html ? This is how we use it in practice. WDYT? I fear that if we train on the whole data set, we will leak information (for example, inferring the parameter of the trend component). |
|
I will think about it. There are at least 2 scenarios:
EDIT: I'm not sure I understand the issue with data leakage in this particular scenario. But it's been a long week. I'm also planning on taking about (but maybe not including a worked example) what you'd do in real work situations when you don't have counterfactual spends. That could be approached by using CausalPy & synthetic control to predict what the media spend would have been in the absence of the campaign. Though it might be the case that a company knows what they were going to spend in the absence of a campaign. |
For example, if you train in the whole data set, then you are already using information on the post-intervention period to infer parameters to generate counterfactuals. Now that I think about it, according to https://smithamilli.com/blog/causal-ladder/ (and similar references). Your approach is indeed at the last level: counterfactuals (and fitting on the whole data is ok I think). Whereas how I would use it (for scenario planing) is on level 2: interventions. This can actually be a nice discussion to have in the notebook (if we want to) |
|
I understand the point now - you could argue that training on the full data may overestimate the trend. But I think that any increase in the outcome should be accounted for by the media + transformations. Good point about counterfactual versus interventions. I think it is worth talking about that in the notebook. |
|
View / edit / reply to this conversation on ReviewNB cetagostini commented on 2024-09-14T00:09:53Z I like it, but I think that to be in line with the language usually used in causal inference, I would say that we add an intervention to the synthetic dataset.
Also, I would talk about the intervention, giving the example where we will increase our budget by 40% from period A to B. drbenvincent commented on 2024-09-16T14:59:45Z Have slightly modified this text. Let's see how it all reads once the larger scale structure of the notebook has settled down a bit. cetagostini commented on 2024-09-16T20:08:57Z 🚀 |
|
View / edit / reply to this conversation on ReviewNB cetagostini commented on 2024-09-14T00:09:53Z Just to be more descriptive I would perhaps call it something like drbenvincent commented on 2024-09-16T14:57:46Z Done in an upcoming commit |
|
View / edit / reply to this conversation on ReviewNB cetagostini commented on 2024-09-14T00:09:54Z Line #6. Based on the counterfactual dataset, apply the intervention on the specified date
You mentioned here |
|
View / edit / reply to this conversation on ReviewNB cetagostini commented on 2024-09-14T00:09:55Z Here we start talking about counterfactuals without any previous information, I would add a paragraph before to explain in detail this section.
Mentioning that we have a dataset with the real intervention, but because we know the strength of the real intervention, we can create another time series where it did not occur and estimate the true effect of the 40% increase in spending.
I think all the information is there, but I would make it clearer and with a little more storytelling. drbenvincent commented on 2024-09-16T15:01:29Z Yep - so this initial commit was very preliminary - I've expanded on the text and narrative of the notebook much more now cetagostini commented on 2024-09-16T20:09:48Z 🚀 |
|
View / edit / reply to this conversation on ReviewNB cetagostini commented on 2024-09-14T00:09:55Z Are these priors doing anything? We can leave the default ones we use. Otherwise, we could extend and show how changing priors changes estimated impact. (?) drbenvincent commented on 2024-09-16T15:01:46Z Good point. I've removed these and we now use default priors |
|
Just to say... this notebook is going to get substantial modification / TLC. But the input already is great - looks like this topic is interesting to people 🙂 |
|
Very excited about plans I have for this post. Though changes will have to wait until Monday. Watch this space... |
There was a problem hiding this comment.
Very cool.
Do you think we can do away with the large functions and duplication in the notebook? It seems the API should expose:
- parameter freezing via
pm.doinstead redefining theforward_pass- Someone can initialize model without data (that defines the structure) and call something like:
params = {"intercept: 1, ...}; y = mmm.generate_target(X, params=params, target_scaler=...). This opens door toX_counterfactual = some_modification(X); y_counterfactual = mmm.generate_target(X_counterfactual, params=params, ...)
- Someone can initialize model without data (that defines the structure) and call something like:
- The counterfactual comparision plot. We just need two targets, right? which we can store in xarray to have the date dim
plot_counterfactual(y, y_counterfactual, along="date", intervention_date="...")
WDYT?
|
Yep - so this is the first very crude implementation. I'm trying to move away from generating synthetic data from separate numpy code and doing it with the DAG, sampling for the prior. I didn't know about And I'll move towards getting counterfactual samples into the This notebook is going to change quite a lot before I flag as ready for review 👍🏻 |
That doesn't exist but just some ideas are API. I think we should make whatever is useful to us
The
👍 |
|
Done in an upcoming commit View entire conversation on ReviewNB |
|
Have slightly modified this text. Let's see how it all reads once the larger scale structure of the notebook has settled down a bit. View entire conversation on ReviewNB |
|
Yep - so this initial commit was very preliminary - I've expanded on the text and narrative of the notebook much more now View entire conversation on ReviewNB |
|
Good point. I've removed these and we now use default priors View entire conversation on ReviewNB |
|
Sorry, took me a moment to understand that this was a proposal @wd60622.
Yep - I will have a go at defending these functions (quickly and crudely) in the notebook. Then they can be built out either in this PR or a different one.
Yes, but I think we would need a way to set the I have to shift focus for a day or so, but I'll see what I can do code-wise in this notebook to inspire some new functionality. |
|
🚀 View entire conversation on ReviewNB |
|
🚀 View entire conversation on ReviewNB |
|
Feel free to make some issues if you don't touch in this PR
The |
Description
This PR adds a new docs page which demonstrates ability to do causal inference with MMM's. In particular we showcase an example where we run a campaign with elevated media spend for a period of time. Our goal is to estimate the causal impact of the campaign. This is illustrated with simulated data in order to run a parameter recovery. So we know the true causal impact of the campaign and can therefore evaluate if the estimated causal impact is close to the true causal impact.
Overall, this notebooks showcases the ability to conduct causal inference in MMM's in
pymc-marketing.Related Issue
Checklist
Modules affected
Type of change
📚 Documentation preview 📚: https://pymc-marketing--1032.org.readthedocs.build/en/1032/