Summary: This study shows that the extremely popular traditional MMM way of sales on any other dependent variable decomposition is inconsistent with statistically correct decomposition used by AMModels. The case shows particular instability when applied to explain the differences between years or quarters due to factors in a model.
When a model is finished, the last step is to estimate the contribution of each marketing element to the dependent variable. Ultimately, this is what the model was built for – it tells us, let say, that sales are determined 3% by advertising, 12% by price change, and so on, which is needed for planning actions. Commonly, it has the form of a comparison of two years, e.g., in 2006 vs. 2005. It seems that this should not raise any questions and, indeed, it does not in MMM practice. However, there is a serious but usually unnoticed problem in this practice.
Regression maximizes the share of variance of the dependent variable, which is explained by the regression factors. Coefficient determination (R squared) is the measure of that approximation. Usually the modeler tries to have R squared high, say, 80% or 90%. There is a special formula, which allows decomposing R squared into components associated with each factor and say, is the size of each factor’s contribution to the whole variance explanation. Let’s call it variance decomposition of the model; this decomposition is absolutely valid from a statistical point of view and relevant to the very essence of the regression modeling (although it has found almost no use in MMM practice).
Another way to estimate the factor’s contribution is to multiply the regression coefficient by values of the factor and sum up the overall observations. It seems logical: if regression is a function Y=a0+a1*X1+a2*X2+…, the contributions are a1*X1, a2*X2, etc. In comparison analysis by years, the contributions are respectively a1*(X1(2006)-X1(2005)), a2*(X2(2006)-X2(2005)), etc. Let’s call it volume decomposition, the only method used in practice and the basis for all MMM conclusions. However, it is not as simple as it seems:
As a result, a paradoxical situation appears sometimes: the perfect model with R squared 95%, applied to comparison of two years, suddenly shows that the difference in sales in $10 M, if decomposed, is $3 M, i.e., the error is more than three times. The modeler may be puzzled by this fact and then “fix” the problem by different “adjustments”. The question that something is wrong in general does not even come up (we have heard the answer that “usually it is OK, the difference is not so big”). Here are the results of a small experiment demonstrating the described situation numerically.
An initial set of data was created with 100 observations (“weeks”), two factors X1 and X2 (both are uniformly distributed variables) and Y, which was equal to the sum of those variables factored by coefficients plus random noise, i.e., an usual regression setting. For this data two types of decomposition were determined. Also, the contribution of each factor into the difference between the first 20 weeks and the following 20 weeks, between weeks 2-21 and 22-41 and so on were calculated, i.e., 60 differences for one data set. Then the whole process with all these calculations was repeated on 30 random data sets of the original dimension. The averaged results for these 30 runs are provided in a table below.
- volume decomposition is not correct for all forms of the model, e.g., it cannot be applied to a dependent variable having negative values, like profit; it cannot be simply implied to non-linear models; and so on;
- variance decomposition has natural limits – from zero to 100%, while volume one does not;
- variance decomposition has a statistical nature, related to a model, but volume decomposition has just remote relevance to that;
- variance decomposition is calculated as one value for the entire model, while the volumetric one is calculated to any part of the data, including individual data points. It creates an illusion that it could be used in any flexible way, like to compare years or quarters, but in fact it is not true;
- two decompositions may give very different results in different situations, which are poorly analytically explained because of the uncertainties mentioned above. R squared may be 90%, but volume decomposition, say, 70% or 110%, etc.;
- difference (both in total and especially by marketing elements) is especially wild for comparisons between years or quarters.
||Variance decomposition, contributions
||Volume decomposition, contributions
||Average volume contribution for difference in 20 weeks
||Sum of contributions (R^2)
||Sum of contributions
|Coefficient of variation
Several conclusions could be made from this:
- data approximation is very high and very stable, with R squared 97% and just 1% of variation, i.e., one may say the model is very good no matter what;
- variance decomposition is very stable, around 18% for the first and 78% for the second factor with low variation (25% and 6%) between samples;
- volume decomposition is even more stable, but with results quite different from the first one: the sum of two contributions is almost 10% lower than for variance contribution (88% vs. 97%), and, most notably, if the ratio of contributions for variance (78/18) is more than 4, here it is less than 2–the two approaches give very different pictures;
- but volume decomposition is not usually used in that way at all; it is used only for comparison purposes, and here the situation is absolutely different. The ratio of average contributions for difference is just 24/32=0.75, i.e., the second factor is less significant than the first, in sharp contrast to both volume and variance contributions! However, it is in fact meaningless – the variation of those values is huge (more than 500%), i.e., one cannot trust averages.
The surprising (and somehow shocking for MMM modelers, in our opinion) general conclusion is that even very high determination of the model does not guarantee that decomposition of sales increments by years has any statistical or economic value. The common practice should be reconsidered. We have come up with many new studies and ideas about that, all of which are incorporated into Advanced Marketing Models (AMModels) practices.