Roblox Virtual Reality using just a webcam

Working on Roblox Virtual Reality using just a webcam :goggles::

  1. Computer guesses where your body parts are in a 3D space using a camera
  2. Data is sent to Roblox game
  3. Body parts are moved based off of this data

I have a dictionary called data filled with different joints of the human body, and their position in 3D space based off of this documentation:

POSE_WORLD_LANDMARKS

Another list of pose landmarks in world coordinates. Each landmark consists of the following:

  • x , y and z : Real-world 3D coordinates in meters with the origin at the center between hips.

As you can see in the image above, this actually works, but it’s inaccurate. For example, when I reach out in real life, it faces a different direction in my avatar. Here’s how I did it:

Can I use VRService to control all body like with VRService:RequestNavigation()? If not, is there another strategy I can use?

5 Likes

Wait, I’m confused. Are you saying that you did this or that your trying to do it?

As you can see in the image above, this actually works, but it’s inaccurate.

It works.

Awesome! How do I use it?

1 Like

It’s not ready yet. For example, when I reach out in real life, it faces a different direction in my avatar.

Ok, I don’t know how to help with that. I hope someone more experienced helps!

Thank you for your interest. I’ll let you know when it does! This is a fun project. I can high five my friends, but I have to twist my arms in real life.

1 Like

@xander5610 I’m finally working on it again. Almost figured it out. I’ll be adding a description soon. (:

4 Likes

This is awesome! Can’t wait until it’s done.

1 Like

Arms update:
image
Waving to an aircraft to turn left

Next up: Legs, knee joints and full three-dimensional head movements.

Because I want to move on to something new, I might end it at 3D head movement, but one-dimensional full body movement. However, the project is still open source so anyone can contribute and help too! Lua and Python programmers needed!

It is done:

Me pointing at my friend who came in first place in a race. His head looking towards me, while he stretches his legs, and poses towards the audience.

Summary: Arms, elbows, hips and knees can move in one dimension, while the neck can turn 180 degrees facing left to right.

I’m going to update the GitHub repository so that there’s instructions on how to use this. I’m not going to work on this anymore because I’m moving on.

Thoughts? Comments?

Alright. How can i set this up because im interested, And i domt habe vr but webcam?

I’ll let everyone know how soon because I need to create a manual on the GitHub page, including some performance improvements. Thank you for your interest!

1 Like

Good news! Turns out I was definitely not done, and added a lot more features like hips rotations, and access control. I’ll create setup instructions on the GitHub page soon. Here’s a teaser :grin::

2 Likes

That looks amazing. It’s great seeing some computer vision work being tethered to ROBLOX.

Thanks everyone for following up till now. I made a final post explaining setup and how to use for everyone: Virtual Reality Using Only a Webcam