I don't think Kiffin was such a WTF hire when he was hired. He turned into one. Embree was praised in many corners. Petersen isn't a slam dunk, but I think it was solid.
I don't agree coaching hires are crapshoots. A good AD can find a good coach.
When they were initially hired as head coaches, I'm not sure there was much of a difference in resumes for these two pools of coaches:
Pool 1
Chip Kelly
Jimbo Fisher
Bob Stoops
Kevin Sumlin
Pat Fitzgerald
Charlie Strong
Pool 2:
Ron Zook
Mike Dubose
Randy Shannon
Frank Solich
Bob Davie
John Blake
The upper tier programs have better odds because they have the resources to poach proven coaches in the prime of their careers AND they always have talented players on their rosters. Even Alabama with nearly unlimited resources struck out on several coaches following the Gene Stallings era (who got old, but was a good coach).