How would I replicate Viewmodel movement to server?

Your best shot to make characters hand look up or down is to make character lean on client and replicate it.