I have had quite a bit of experience in this field but I don’t main it so don’t take all of this as professional information. I am a modeler but enjoy doing this stuff on the side.
The first thing I would recommend is getting into a photo editing program like Adobe Photoshop, Paint.NET, GIMP, or my favorite, Adobe Illustrator.
The first three listen products (Photoshop, PaintNET, and GIMP) are all pixel based so you will be working with pixels. The last product (Illustrator) is a vector based program.
Illustrator works differently than pixel programs. This is what I found when trying to find an explanation for what they use:
Vector art is created using vector illustration software programs, such as Adobe Illustrator or Corel Draw. These programs use mathematic equations and geometric primitives (points, lines, and shapes) to create art that is clean, camera ready, and can be scaled infinitely, without any loss of quality or fidelity.
The reason I enjoy illustrator is because it provides so many features for smooth objects and you can theoretically scale the object infinitely and not see any shape deformation.
If you aren’t interested in any of these programs it is definitely harder to get a grip on raw UI editing. When I mean raw, I am meaning that you will only be using Roblox UI and nothing else.
With using an editing programs, you can create custom shapes and fit exactly what is desired for what you need and it can save a lot of time too.
Now, I suggest fitting a design that fits your game best. Make sure the style fits with your genre but don’t be afraid to take risks to see if it looks well with your game. I am currently working on an FPS/Battle Royale and am creating a dashboard look for my UI which hasn’t been widely used but I am taking that risk.
A lot of users enjoy a simpler design so they can navigate easier and faster so don’t go for too complex that a user doesn’t understand what to do and where to navigate.
As for colors, just use trial and error. Look at websites, programs, games, and advertisements to see what colors worked and how you could apply them to your UI.
Here are the programs with tutorial links and how each program looks:
GIMP
Adobe Photoshop
PaintNET
Adobe Illustrator
All of these programs have benefits that come with each so it can be very hard to choose. For free programs, I provided GIMP and Paint.NET. For purchasable programs, I provided Adobe Illustrator and Adobe Photoshop. It all depends on your budget and if you are even willing to spend money. I would suggest starting with a free program just to make sure you enjoy doing UI and then move into a paid program when you get into more advanced work.