Have you matted the end of the pusher either by whacking an old file on it a few times or banging it into some coarse wet & dry on a bench block?