So I just checked, and I did receive a reply from Maul. Here's what he had to say:
"There is no masking of WoW graphics in the sense of traditional graphics editing.
First, I go through pain-staking sizing and resizing until the icon is as large as it can be without peeking out the outside edges. I do this with the button at a scale of 3 (very large )
Second, once I get the icon to the size I want, I see how much of the "squareness" is exposed on the inside of the skin. Once I do that, I created an alpha layer to "blackout" those areas. Again, this can be a pain-staking process to get right.
You can also see this "blackout" alpha layer technique in the round skin of CyCircled/Button Facade, except they do it on the outside of the button. If you look closely, you can see the black ring on the skins. That black ring hides the corners that extend beyond the outside edge of the skin.
I prefer the technique I use (inner black mask/ring) as it gives the buttons a 'crisper' look, imho."
So I guess I've got some tweaking to do. Of course I pick a more difficult shape to work with for my very first button skin. But I've learned more by doing it this way
Thanks again to all that helped.