Model Evaluation, Diagnostics and MLFlow Registry

[louismagowan]

Description

This PR is a recreation of this one which I was told to close and restart due to git issues on the pymc-marketing repo.

It could be nice to have some standardised model evaluation and diagnostic functions added to pymc-marketing. Ideally they'd be formulated in a way that makes them easy to log in MLFLow later on.
It would also be cool to build on top of the MLFlow module, to create a custom mlflow.pyfunc.PythonModel class to allow users to be able to register their models in the MLFlow registry. This would allow people to serve and maintain their MMMs more easily, and could help with MMM refreshes too.

Standard model metrics could include:

• Bayesian R2
• MAPE, RMSE, MAE
• Normalised RMSE and MAE (to allow comparisons across datasets and methodologies - in particular with Robyn models, who use NRMSE as one of their 2 key metrics)
  etc.

Diagnostic metrics could include:

• Step size, divergences
• LOOCV metrics e.g.

Model Registry Code:

A wrapper for an MMM model to make it conform to the MLFlow api, enabling registering and easier deployment
I'll open this as a Draft for now - since I'll need advice on how where best to put this code, as well as overall design etc.

Related Issue

• Closes Evaluation and Model Diagnostic Functions #911
• Closes Add Support for Model Registering to MLFlow Module #973
• Related to Expose serialized format #891 MLflow deployment example #901

Checklist

• [] Checked that the pre-commit linting/style checks pass
• Included tests that prove the fix is effective or that the new feature works
• Added necessary documentation (docstrings and/or example notebooks)
• If you are a pro: each commit corresponds to a relevant logical change

Modules affected

• MMM
• CLV

Type of change

• New feature / enhancement
• Bug fix
• Documentation
• Maintenance
• Other (please specify):

---

📚 Documentation preview 📚: https://pymc-marketing--1026.org.readthedocs.build/en/1026/

[juanitorduz]

@louismagowan, thanks for opening this one up again! Some tests (and the pre-commit) are failing. Do you need some help?

[louismagowan]

@juanitorduz - thanks but I think I'm alright for now (or at least I think I can try and fix it myself later). I think I probably need to recreate my conda env (as I think it's a bit old now).
Apologies for being slow on this, I have a deadline at work that I've been rushing for - so really not had much time to work on this recently.

@wd60622 - no need to review again yet, I'm still working on addressing your earlier feedback (I'll ping myself when I think I'm ready again)

Thanks!