The Illusion of Robustness: Aggregate Accuracy Hides Prediction Flips under Task-Irrelevant Context