The size of the town doesn't generally matter because there's a number of schools in smaller towns who draw extremely well, and there's also a number of schools in larger cities that don't draw well at all. It's not a secret that Stanford's fanbase is weak even by west coast standards but in fairness that's not unusual among private schools. Here are the numbers from last season for all the P5 private schools (ND is the obvious exception so I left them off):
Stanford - 37,842
Vanderbilt - 28,045
Northwestern - 43,873
Baylor - 41,336
TCU - 42,868
Wake - 26,842
Duke - 26,646
Syracuse - 37,043
BC - 37,623