I have never had this happen, although to be fair I always start with bought in wire. It seems to indicate that the metal has not been sufficiently worked after pouring.
If you put it through your square rollers first, reducing the gap three or four times, and then anneal, it should work.