Ball up:

To collect a mass of sticky consolidated material, usually drill cuttings, on drill pipe, drill collars, bits, and so forth.

A bit with such material attached to it is called a balled-up bit.

Balling up is frequently the result of inadequate pump pressure or insufficient drilling fluid.

2.

In reference to an anchor, to fail to hold on a soft bottom, pulling out, instead, with a large ball of mud attached.