A telepresence robot – Programming

In this article we are going to program the Telebot we have built in the previous article.

We will use WebRTC, which is the new standard for real-time communication in Web browsers, and take advantage of the necessary signaling channel to also transmit commands to move the robot.

General schematic of the whole control system
General schematic of the whole control system

Programming the robot actually consists of three different steps:

  • Writing Arduino-flavored C++ code for the Arduino-like controller to properly move and balance the robot
  • Building a specific Android application to handle a WebRTC session on the smartphone and relay commands to the controller via Bluetooth
  • Setting up a node.js server to serve an HTML5 control page over HTTPS allowing visioconference and remote control
The Telebot ready to be programmed
The Telebot ready to be programmed

Therefore, the project will be a mix of Arduino, Android Java, and Javascript (client-side and server-side). The source code is free software, licensed under BSD 2-clause license and GPLv3 (Arduino code). The complete source for the project is available on my repository on GitHub.

Arduino programming

First, motor control is achieved through an extremely simple text protocol over the Bluetooth serial, with one single-letter command and one value per line. Motors are driven with PWM using analogWrite. In parallel, battery voltage is measured thanks to the probe on pin A0, the value is scaled with the empirically-defined factor batteryProbeFactor.

If the battery voltage is too low, we trigger an emergency stop to prevent damaging it.

Before actually setting motor power, we have to add a correction in order to keep the robot balanced: the principle is to move in the direction the robot is leaning, like a Segway. To this end, we need to communicate with the MPU-6050 via the I2C bus to read the measured values. I tried first to use the Arduino Wire library, but it tends to hang after some time and does not seem to allow defining a timeout. It is quite a drawback since it can crash the robot and as a result make it fall down, so I prefer to use a software I2C library.

Then, to balance the robot, a double linear correction is added to motor control, taking into account the angular speed rotx and the angle angx, with two different empirical constants k1 and k2:
correction = rotx/k1 + angx/k2

Computation should be achieved with integers only, since the processor does not have a native floating point unit. The update loop runs at 100Hz, i.e., values are updated every 10ms.

Programming the robot with a USB to UART bridge
Programming the robot with a USB to UART bridge

Android application

The first job of the Android app is of course to connect to the base via Bluetooth.

We list paired devices called “Telebot”, which the HC-06 name has been previously changed to (You only need to connect it directly to the UART and send the AT command "AT+NAMETelebot"). Then we try to connect the serial port service on each of them.

I have attempted for some time to make use of OpenWebRTC to create a WebRTC-enabled native Android application, but I could not obtain something reliably interoperable with Firefox or Chromium, even after some debugging. For this reason, I’m creating an Android application handling the robot controls but relying on a Chrome instance to handle the actual WebRTC session. It is kind of a hack, but it works.

For the signaling channel, I simply use a modified version of OpenWebRTC simple signaling channel for Android. It is lightweight and fulfills the purpose entierly, so why recode my own thing?

In the app, the signaling channel is obvisously not used for WebRTC, since it will be handled in a separate browser, but it is of the utmost importance because it receives user commands to move the robot.

EDIT: I changed this behaviour in new versions. Now, user commands use an RTCDataChannel rather than the signaling channel, allowing to reduce latency.

Javascript frontend and backend

Since the robot uses a browser to handle the WebRTC session, the Javascript client must handle active mode (for the user) and passive mode (for the robot).

For its signaling channel, the Javascript client also use a modified version of OpenWebRTC simple signaling channel. Either the arrows on the page or the keyboard can be used for user input, and controls are sent over the channel just like WebRTC signaling. Please check the repository for the complete source code.

The node.js server is a slightly modified version of OpenWebRTC example server. It is deployed behind an Apache server with HTTPS configured with Let’s Encrypt.

I also installed coturn STUN/TURN server to help WebRTC session establishment when NAT is in use, i.e. in most of the usage cases. I can’t wait for ubiquitous IPv6 network access to bring the death of NAT.

Thanks to the HTML5 interface, you can use the Telebot from any device with a recent browser equipped for video and sound capture, even another smartphone. However, it won’t work out of the box on an iPhone, since iOS does not have any major browser available with WebRTC support as of today, even through this standard feature has been available on other platforms for years…

The interface in Firefox for Android
The interface in Firefox for Android

Eventually, it works wonderfully! I have only added a microphone to the smartphone to enhance sound capture.

The robot moving around
They see me rollin’, they hatin’

EDIT: You can read about subsequent enhancements here, in particular the use of an RTCDataChannel for controls.

5 thoughts on “A telepresence robot – Programming”

  1. This is almost exactly what I needed. I am am trying to mod an esolution i
    xisting telepresence robot, Mandy, that has been modified removing tablet and adding arduino to drine it via hand held controller. So I investigated ways to do this and came up with WebRTC and pubnub. But I think your solution might be easier. Although pubnub takes care of signaling.

  2. Very interesting project and a nice documentation!
    There is a downloadable OpenRTCBrowser available for iOS. Would this work together with an iPhone? Or is an Android device a must?

Leave a Reply

Your email address will not be published. Required fields are marked *