Way to remove glossiness on meshparts?

Turning Brightness down to 0 is not really an option I want. I want certain blocks/ meshparts to not be so glossy, but they still become very glossy even when set to fabric and given a textureid.

The best option is the fabric material right now, but custom materials with specular properties are coming soon!