Sunday, July 3, 2011

Conference Vs. Non-Conference

As a follow-on to this posting where I looked at how well the predictors worked over different periods of the season and noted some odd behaviors, I looked at conference versus non-conference games, to see if there was some difference in performance.  Here are the results:

  Predictor  MOV Error  % Correct
TrueSkill -- Conference Games10.68972.10%
TrueSkill -- Non-Conference Games11.53976.39%
Govan -- Conference Games10.58372.17%
Govan -- Non-Conference Games11.32576.83%

This shows the same sort of split we see in early-season versus late-season predictions:  conference predictions are closer to the actual Margin of Victory, but more often on the wrong side of the contest.  Of course, most non-conference games are also early-season games, there's some interdependence in these results.  We can try to factor this out by looking at late-season non-conference games and early-season conference games:

  Predictor  MOV Error  % Correct
TrueSkill -- Conference Games10.68972.10%
TrueSkill -- Early Conference Games11.48566.67%
TrueSkill -- Non-Conference Games11.53976.39%
TrueSkill -- Late Non-Conference Games11.14872.68%
Govan -- Conference Games10.58372.17%
Govan -- Early Conference Games10.98869.57%
Govan -- Non-Conference Games11.32576.83%
Govan -- Late Non-Conference Games11.08973.04%

This seems to show that it's really the early season versus late season that matters (and the predictors are particularly bad at early-season conference games).

One easy follow-up to this experiment is to train our predictor on only conference games (or only non-conference games) and see if this improves performance within the category.  I won't post the numbers, but there's no advantage in training on non-conference games when predicting non-conference games, etc.

None of this explains the oddity of MOV Error improving while % Correct simultaneously worsens, but at least there's some evidence that conference versus non-conference is probably not a big factor in prediction performance.

No comments:

Post a Comment