Virtual Reality Using Only a Webcam

Resources Community Resources
, , , ,
#1

Ever watched videos of rich people in headsets hugging Roblox players but couldn’t join the fun?
Well now you can!

With Roblox 3D, you can control your Roblox character neck to knee like in VR, but using only a webcam and no headset.

Fun Ideas

  • Yoga and stretches
  • Emotes, waves and dances
  • Act-guessing game
  • Ground signals for planes

Features

  • Control your R15 character pose (neck, shoulders, elbows, hips, legs and knees)
  • One-dimensional flat pose (for now)
  • Authorization and access control
  • Sleak, ready-to-go in-game GUI

How it Works

A Python script running on your computer takes your webcam video feed, and estimates the location of the 3D points on your body, using an advanced artificial intelligence by Google/MediaPipe. This pose data is repeatedly sent several times a second to a third-party server, where a Roblox server constantly downloads it, and finally rotates your character’s body parts.

image
image960×720 139 KB

Continue reading: https://github.com/mmbaguette/Roblox-VR-with-Webcam#how-it-works
Note: This program does not upload your webcam image online, only the 3D data that was processed by the machine learning model.

Getting Started

All setup instructions are on the GitHub page below (NEVER download from anyone but mmbaguette/myself because they’re untrusted). At the time of posting, I was too lazy to finish setup instructions for games, so for now just test it at my place and hope you don’t crash the free server.

Concerns

I know that blindly downloading programs online is an unsafe idea, which is why this project is open-source, heavily documented, and simple to understand. Even if you only know a little scripting, I encourage you to read a few lines of Python code (similar to Lua) in main.py, but this is best viewed using an IDE like Visual Studio Code. Please ask questions or concerns if you have any!

Contributing

I could really use both talented Lua and Python scripters, and especially if you’re good at 3D physics (pls send help) to contribute. After all, it should be a community project. Some things I want to see:

  • Full 3D rotations (not just on the one, flat axis we have already)
  • 3D head rotation
  • Faster serve upload
  • Better solution to send pose data to Roblox game (important for accessibility)
  • Someone volunteers their server for everyone to use

GitHub page: https://github.com/mmbaguette/Roblox-VR-with-Webcam
My testing place: VR using a Webcam 📷 🥽 - Roblox
Roblox model: Virtual Reality and 3D Camera - Roblox

Update
It turns out setting this up was more difficult for the average user than I thought. I’ve been working closely with someone to figure this out. I might release a video soon demonstrating setup or someone else can before I do.

64 Likes
Roblox Virtual Reality using just a webcam
#2

The model is not for sale, you maybe want to fix that…

4 Likes
#3

Thanks, you can access it now! All I need is to upload the server script.

#4

Also I’m NOT the person in the video. I got it from here: 7 Upper Back Stretches For Pain Relief - YouTube

3 Likes
#5

do i have to put my username where it says username in the link becuase its notworking
https://mmbaguette.pythonanywhere.com/upload_pose/{why_zaid2}

#6

i need help i did everything it says but no window pop up or nothing happens it just runs and disappear

#7

I just updated the comments so it’s more clear:

# SERVER OWNERS (not players): move {username} to wherever the player's username should be inserted into the URL. 
# PLAYERS: this value should be given from the roblox game.
upload_url = f"https://www.example.com/upload_pose/{username}"

So basically change it back to https://mmbaguette.pythonanywhere.com/upload_pose/{username} like it says in my game. For other games you would change the upload_url value to whatever the 3D Roblox GUI tells you to.

1 Like
#8

i am sorry but i am still confused

#9

That’s okay, what do you need help with?

#10

i dont know i did everything it tells me to do its just not working. If you can make a video it will be much easier to understand

#11

Can you show me the program output on your computer including any errors in the console? What do you have for the "-----EDIT THESE VALUES-----" place?

Here’s what your python script would look like in your example if you joined my testing place:

"-----EDIT THESE VALUES-----"
# Roblox username of player
username = "jake_kajj2"
# AUTHORIZATION token (games will ask you for this)
authorization = "temporary access token given from Request Token button"
# SERVER OWNERS: move {username} to wherever the player's username should be inserted into the URL. 
# PLAYERS: this value should be given from the roblox game.
upload_url = f"https://www.mmbaguette.pythonanywhere.com/upload_pose/{username}"
1 Like
#12

SyntaxError: invalid syntax. Perhaps you forgot a comma? >>>

#13

Set this to your upload_url variable: https://mmbaguette.pythonanywhere.com/upload_pose/{username}

#14

Shouldn’t this just be renamed to Alternate Reality tech and not VR? You’re technically not interacting with a virtual world by being in it and instead is interacting with it in your own room which is basically AR.

Bit clickbaity ngl.

2 Likes
#15

agree with this one and I’m still not too sure how will it detect Controller while playing around with this

#16

Uh virtual reality is an alternate reality (also AR means augmented reality which isn’t very different but it’s something else), plus what’s stopping you from plugging in a cheap VR headset and using this like a cheap HTC vive? (other than the limitations) About the interaction thing, on every VR headset you’re interacting with your own room unless you got some fancy teleportation system that gets you in game. I don’t get how this is “click baity” as the topic title is very well self explanatory.

#17

1: No, no, we’re talking about the terms, not the philosophical version of it.
AR is virtual things being presented or projected into reality (or vice versa as seen in the OP)
VR is a virtual environment differentiating from our own reality.
(Also yes, I did just realize I made AR "alternate reality by accident, sorry about that.)

2: Headsets are expensive. Lowest price you can get in the US that’s actually reliable is $300.

3: It’s mainly clickbaity because it’s talking about “virtual reality without a VR device” which is the exact opposite of what the post is talking about.

#18

Isn’t this just motion tracking, not VR?

#19

No one knows what to expect if I said “motion tracking” or “AR” because it’s not supposed to exist in Roblox. Real Roblox VR also has this same feature, but not nearly as many limbs to control. The first few sentences are very clear as to what this does.

#20

I cant get the program to use my camera? Every other application on my pc works with the camera, but I only get a black screen on the webcam pose screen…