Lord Peter Mandelson should hand back the payoff he received after being sacked as ambassador to the US, or give it to charity, a Cabinet minister has said.