Prince Harry’s biggest regret about leaving the royal family has more to do with his grandmother than anyone else.