We've added a new 'Smooth' option to the Mask Tool Rough shape. With this option selected you will get smoothly rounded, randomized shapes for a natural looking coastline with much less effort. Give it a try!