This page contains information about the robot monitoring and control using the ROSBridge suite and a HoloLens device. The solution presented in this article is similar to a certain extent to the Web based solution, that can be
found in the related articles section.
1. Why a monitoring and control tool ?
- To check if nodes are running properly
- To check if topics are published
- To check if parameters needed by nodes are being properly set up
- To have the possibility of checking whether services are alive
- To have the possibility of restarting nodes
- To have the possibility of stopping, restarting certain nodes
- To have the possibility of checking whether critical nodes are running
2. Why a HoloLens based tool ?
- Because operating nodes from a terminal requires a certain in-depth knowledge of the workflow
- Because connected industrial workers can use the functionalities without the in-depth ROS required knowledge
- Because the capabilities of a connected industrial worker can be enhanced using such a solution
- Because the industrial worker can be better integrated into the industrial environment by using such a solution
- Because the industrial worker can easily understand and control what other fellow robot workers are doing
- Because an industrial worker can better maintain and understand other robot colleagues
3. About HoloLens
4. Monitoring tool
The monitoring tool is developed using Unity and C#. It communicates with the Baxter robot or ROS environment using the WebSocket technology.
The functionalities exposed by this solution are the following:
- Connection to Baxter using ROSBridge
- Get a list of all running nodes
- Get a list of all published topics
- Get a list of the type of all running topics
- Get a list of all required/ compulsory nodes
- Possibility of stopping a node
- Possibility of starting a node
- Possibility of testing whether a node is alive or not (ping)
5. Components of the solution
From a top viewpoint, the solution has two components: a back end and a front end one.
- The back end is represented by baxter_monitoring_tool
- The front is represented by the actual HoloLens application
5.1 Back end - baxter_monitoring_tool
It is a ROS package that takes care of all the functionalities needed to start, stop, ping a node, acquire the list of all running nodes, all published topics including their types, as well as acquiring the list of
all nodes that are compulsory for the proper execution of a certain demo/process of Baxter.
5.1.2 Contents of the package
The package contains more folders, however the ones relevant for this context are the ones highlighted below:
- launch - contains a launch file that when executed automatically launches the monitoring tool back-end functionalities
- msg - contains the definition of specific messages that the tool uses, which will be presented below
- scripts - contains the additional back-end functionalities of the tool: status server, file parser developed in Python
- src - contains the part of the similar functionalities developed in C++
- srv - contains the definition of the specific service file, statusInfo.srv
The most important features are located in the scripts folder and are status_server.py and parser_help.py.
The status_server.py contains the implementation of ROS service server, that handles the requests from different clients returning the appropriate values. The server uses the functionalities comprised in the rosnode package to handle
the query of the list of nodes, topics, topic types and also to kill and ping nodes. A node starting functionality, however, is not exposed by this package thus a solution had to be come up with. In this case when a client issues a start request
for a certain node, what actually happens is that a background process is started for the respective node using functionalities from multiprocessing and os modules from Python. Thus, more precisely a rosrun command is issued but from code,
rather than from a terminal. From the tests carried out, there is no overhead and the nodes are starting as with the typical terminal based method. Do the nodes remain active as a background, daemon process is started? From carried out tests
the nodes do not remain active, due to the the fact that all the nodes are registered to the ROS Master and once the master is killed are nodes are implicitly killed as well.
The parser_help.py file contains a class named LaunchFileParser that has the role of parsing the contents of a ROS launch file. It extracts are the existing nodes in the launch file with their name, args, pkg and other information as well. The parser does not stop
only at its entry file, it parses all the included launch files as well, furthermore, replacing different argument variables defined in the files and used by nodes. An additional demo_domain argument is recommended to be also placed, at least at the entry launch file level,
for error tracking issues. For example, one of the nested launch files, from the entry launch file, gets a demo_domain="Perception" argument. In this manner all the nodes parsed from that launch file on(if not specified) are assumed to be from the respective demo_domain.
Thus if a problem occurs with one of the nodes parsed from that launch file, the error can easily be tracked to the specified demo_domain.
5.2 Front end - HoloLens application
The front end is represented by a HoloLens application, whom the industrial worker uses to query the state of the demo.
5.2.2 Contents and implementation
The application has been developed in Unity using C# and the HoloToolkit functionalities. The relevant code can be found in the Assets directory in the Scripts folder.
The scripts are organised accordingly:
- in the ConnectionUtils folder are the scripts that facilitate the communication between the HoloLens and the ROS environment.
- in the Utils folder are the JSON parsing classes, classes encapsulating the transmitted messages, classes grouping together all the transmitted values.
- in the Handlers folder there are all the scripts that facilitate all facilitate the events that occur during the interaction with the buttons, dropdowns, etc.
5.2.3 Communication with ROS
The application can be run in two modes, one is from the actual HoloLens device and the other is from the Unity player. In the latter mode, the communication is carried out using the functionalities from the WebSocketSharp package, unfortunately due to Unity-.NET, Mono
incompatibility issues those functionalities are not available on the actual device and Windows.Networking solutions have to be used in order to make the two entities communicate between them. That is the reason why in the ConnectionUtils folder there are two scripts one
ROSConnect for the actual device and ROSConnectS to be used in the Unity player. The functionality of the two could be placed into one script in that case preprocessor directives such as #!if UNITY_EDITOR #endif have to be used to differentiate between the two components
that facilitate the communication.
PS: UNITY_EDITOR is the flag when the code is run from the Unity using the play button.
6.1 Back end
The monitoring tool has been developed under Ubuntu 14.04 using ROS Indigo but it can be used with other distributions as well.
In order to be able to use the tool in its current state or to further develop it you need to make sure several packages are installed:
- Install the rosbridge_suite using the following command in a terminal sudo apt-get install ros-distro-rosbridge-suite
- Make sure Python and especially the multiprocessing module are installed. If not install multiprocessing with sudo pip install multiprocessing
If choose the option of downloading package from the repository then carry out step 1 and additionally make sure that step 2 is in accordance.
6.2 Front end
In order to be able to load the solution to a HoloLens device the build of the Unity project has to be carried out.
- Save your Unity project and go to File -> Build Settings ...
- From the menu opened up make sure to add the your desired scene from the Add Open Scenes
- Select from the Platform section the Universal Windows Platform
- Select from the Target device HoloLens and from Build Type D3D, SDK Latest installed, Build and Run on Local Machine
- From the Player Settings ... menu make sure to check the needed capabilities of the device, for example :Internet Client, InternetClientServer, PrivateNetworkClientServer, Microphone, SpatialPerception
- When all is done hit the Build button select the appropriate folder and then wait until the process finishes
- Go to the location where the project has been built and open the ****.sln file
- Plug in using a USB cable the HoloLens to your PC and select the from the green arrow Device and x86 and hit the green arrow
- Make sure the computer and HoloLens are paired, if all goes well the application will be started on the HoloLens
7. Starting the application
In order to use the application you have to make sure of the following things:
- The HoloLens is connected to the same network as Baxter is
- The rosbridge_server is started. If not, open a terminal, run ./baxter.sh and after type in roslaunch rosbridge_server rosbridge_websocket.launch
- The status server is running. If not, open a terminal, run ./baxter.sh and after type in rosrun baxter_monitorin_tool status_server.py
- The application is running on HoloLens. The start procedure has been presented in the previous section
The application can be started without the debugging part,simply search for holoConnection app on HoloLens and start it as an usual application.
8. UI of the application
8.1 UI version 1.0 of the application
The application consists of different panels that appear/disappear depending on user interaction.
When the application starts only the Connection panel is on. The user has to gaze to the Connect button and perform an air tap movement. In that moment a connection is tried to be established with ROS environment.
If the trial is successful the toggle above the Connected button will be ticked and the Advanced Panel will appear. The user can interact with the different buttons present such as Get Topics, Get Nodes, Get Required Nodes,
as well as with the Status Indicator. The indicator will be red when at least one of the required nodes is not active and green when all is working well. When gazed upon a text message will appear, which if ok it will display
All is good and when not it will list the inactive required nodes.
When interacting with the Get Topics, Get Nodes, Get Required Nodes buttons, the respective sub panels with their options will appear. If the requests get their appropriate responses and all is goes well, the dropdowns will be populated with their
respective information. The user can select a certain option from the dropdowns and perform the available options on them. For example after selecting a topic and gazing on the Topic Type button , the TopicType text will turn into the selected topic's type, which when
not gazed upon will disappear
The user can kill, ping start nodes as it desires.
8.3 UI version 2.0 of the application
The second version of the user interface extends the first one. The additional elements are the "Show Advanced Panel" and the three top indicating elements( AGV, Motion, Perception, buttons). At start, just the Connection Panel will be available. If the connection is successful the three top indicating elements and the Show Advanced Panel buttons will appear. When focused upon one of the top indicators, it will tell if the corresponding required nodes are running. If everything is running properly then the indicator will be colored green and the "AGV/Motion/Perception OK" message will appear, if not the indicator will be colored red and the "AGV/Motion/Perception Not OK" will appear. The indicator status will appear just when focused upon. The Advanced panel will appear/disappear when the user clicks on it. When disconnecting the Advanced panel will disappear.
8.3 UI version 3.0 of the application
The third version of the user interface extends the previous versions, by inserting an additional, fourth, indicator button, namely the Factory button. The activity of this indicator is similar to the previous three ones: AGV, Perception, Motion.
9. Issues and possible extensions
9.1. Updating the lists asynchronously
The Web based monitoring tool uses web browser related technologies. For example, it has been developed using AngularJS in which asynchronous requests are built in, thus the model and view are kept consistent by the backend.
In the case of HoloLens, the drawback is those functionalities have not been implemented, at least in this solution, thus the user has to update manually its list of nodes, topics, required nodes, by clicking on the required buttons and performing the
update of the lists manually.
9.2 Input of the Hostname and Port
In order to connect via the rosbridge a hostname and a port are necessary. Currently those are hard coded to the values that are shown on the two input fields. An important step would be to give the user the possibility of entering the values on its own
using air keyboards present on HoloLens.
9.3 Network traffic issues
The communication between HoloLens and ROS master relies heavily on the power of the network signal, if the data traffic is heavy an unusual workflow might appear while the application is running.
How the app works
How the application works