I don't hold it against the me...
I don't hold it against the men who beat me because undoubtedly there are some ruffians of every nationality and the English are not exceptions.