Teleoperation

A minimal implementation of a bi-manual remote robotic teleoperation system using VR hand tracking and camera streaming.

Setup

git clone https://github.com/kscalelabs/teleop.git && cd teleop
conda create -y -n teleop python=3.11 && conda activate teleop
make install-dependencies
pip install -r requirements.txt

Scripts

demo.py - POC for integrating PyBullet with Vuer for real-time robot control

collect.py - Data collection script that interfaces with demo.py for video and position collection

Usage

Start ngrok on the robot computer.

ngrok http 8012

Open the browser app on the HMD and go to the ngrok URL (there will not be any content until you start the server on the robot computer).

Teleop

Start the server on the robot computer.

python demo.py --gui --firmware

Data Collection

Start the server on the robot computer.

python collect.py --task_name CONFIG_TASK_NAME --use_firmware True

Replaying Trajectories

Simply run the following command to replay the recorded trajectories. This currently only works with episodes split into hdf5 and mp4 files.

python replay.py --dataset_dir PATH/TO/DATASET --episode_idx EPISODE_IDX

Common Issues

Large Errors in Robot State and Actions

Check the units, offsets, and any other factors that might be causing the numbers being recorded to be different than you expect. Also, check that the motor configs are correct (signs, offsets, etc). The Robot class has a handy method to test motors.

robot.test_motors()

Robot Not Moving

Check that the robot is connected to the firmware and that the firmware is running. Check the status of the can bus and restart if needed.

sudo ip link show can0

Jerky Robot Movement

Depending on your setup (power supply, battery, etc), check that current draw is not too high. An abnormally high current draw is also indicative of large errors in motor state and actions since we are using a closed loop PD torque controller.

Dependencies

• Vuer (opens in a new tab) is used for visualization
• PyBullet (opens in a new tab) is used for IK
• ngrok (opens in a new tab) is used for networking

Citation

@misc{teleop-2024,
  title={Bi-Manual Remote Robotic Teleoperation},
  author={KScale Labs},
  year={2024},
  url={https://github.com/kscalelabs/teleop}
}