This is a reasonable expectation. But how often do you actually learn something or get truly insightful analysis from the color guy? For me, I actually learn insightful things from Tony Romo, Hubie Brown and Doris Burke in the NBA (although Hubie is ancient and has lost his fastball), sometimes Jay Bilas in college basketball, and some of the college football guys are actually pretty good. Most of the rest is not interesting or informative to me in the least, it's just background noise at best and annoying babble at worst.
Entertain me damnit, and Walton entertains me.
I really liked it when Bob Knight was a color guy years back, thought it was pretty cool. Not sure if ESPN didnt like it, or he got tired of it.