hold against

to let your opinion of someone become worse because of a mistake they made or something they did in the past.