An AlphaMask would be an additional image that multiplies the original image's alpha channel with the alpha channel of the AlphaMask.
As an example, lets say we want to display a headshot of the character's avatar inside of a circular display.
The problem here is that the headshot doesn't have a circular crop.
Now if we were to have an AlphaMask that looks something like this:
Then it would take the alpha channel of this image and then it would be applied to the original image, resulting in something that looks like our devforum avatars.
Another instance where this might be useful is with a map display. Perhaps you want the map to have a certain shape while it is dynamically cropped using ImageRectSize and ImageRectOffset. With an AlphaMask, you could accomplish this.
These are just a few examples I could think of, I have no doubt others would find really creative ways to utilize this.