Heh,
I knew this sounded familiar.
I extracted the exact same blp file that the interface uses this function with ("Interface\\CharacterFrame\\TempPortraitAlphaMask"), and it does nothing but make my texture disappear when I try to call SetMask with it.
It works when you reference the texture that's inside the game files, but it doesn't appear to be possible to create your own masks.