Why do famous conductors rarely conduct ballet?

The conductor's movements work for both. However, it ballet, the principle performers are dancers, and they aren't looking to follow a conductor. A conductor of ballet takes on more the role of directing the accompaniment than directing the entire performance.