"Implying that, at some point in time, you were taking a beating like that, what, frequently?" He's not angry at Sarah; Rory keeps his tone mild. But he is annoyed at the world for making it so easy for people like Sarah, like Amy, like the Doctor to get themselves hurt. There's nothing wrong with being tough and head strong. There shouldn't be. But there was enough trouble in the world without people finding ways to get into it.