Harry could hardly believe that Dumbledore would not have intervened if such cruelty was happening inside his own house, and yet there was undoubtedly something odd about the story.
As valet, he kept Gerald's bedroom in order, and, as butler, he served the meals with dignity and style, but otherwise he pretty well let matters follow their own course.