When changing the density of a part below 0.01, the mass will still be what it would be if the density were 0.01. For example:
A cube with a size of 10,10,10 or a volume of 1000 has a mass of 1000 when the density is 1. When the density is 0.02 the mass is 20 and when the density is 0.01 it is 10, which is what is expected. However when the density is 0.001, the mass remains at 10 when in reality it should be 1.
I am unsure if this is just an error when setting the property, however it is not simply an issue with displaying the mass as accessing it from a script will also return the wrong result. This means that calculating volume using mass over density returns a completely wrong result.
Part density is clamped when computing mass. Right now, the default behavior sets a lower bound of 0.01 on the density when computing mass. However, this is changing with a three phase rollout to 0.0001. You can activate the new lower bound on density by setting the workspace three phase rollout property workspace.DecreaseMinimumDensityMode to Enabled.
PhysicalProperties will still store densities between 0.0001 and 0.01 at the values you set them to, but mass will be computed using the old lower bound of 0.01 unless you opt into the new behavior. Soon, however, we will enter phase 2 of the rollout and density will be clamped to the new limit (0.0001) when mass is computed by default.