Lord Glasman told Sky News he flagged his concerns to Downing Street but received a "discreet suggestion to basically shut up ...