Really good video by the Recast folks. Some choice quotes on marketing:
Experiments are “good” to the extent that they can provide support for or against a certain hypothesis
Various model metrics (R-Squared, MAPE, convergence) don’t tell you if your model is good, you need internal or external validity checks.
MMM’s should be assumed to be wrong until they’re proven to be correect
Continuously test hypotheses, and we want to falsify these hypotheses as quick as possible.
We don’t need truth, we just need to improve our marketing a little bit every day.
Michael made another good point in the video about how you’re certain of an experiment’s results during that time, and then after. But then the experiment continues to lose explanatory factor as time progresses.
MMM’s - main danger here is overfitting and not validating results.
Unidentifiability - basically, if you have solid colinearity, it’s hard to interpret these results
Need to figure out parameter recovery - most important internal validty check
External validity: Consistency with experiments and true forecast accuracy
Experimental results to constrain the prior during that the lift test was run