Lesson 2.2: Unity for Robotics

Introduction: Why Unity for Human-Robot Interaction (HRI)?

While Gazebo excels at physics-accurate simulations for robot control and navigation, Unity a popular real-time 3D development platform, offers unique advantages, particularly for Human-Robot Interaction (HRI) and high-fidelity visualization. Unity is renowned for its advanced graphics rendering capabilities, rich ecosystem of assets, and powerful user interface (UI) development tools, making it an ideal choice for creating visually compelling and interactive robot simulation environments.

Key Reasons Unity is Used for HRI:

1. High-Fidelity Visualization: Unity provides stunning visual fidelity, allowing for realistic rendering of robots, environments, and human avatars. This is crucial for HRI studies where visual cues and realistic appearance can significantly influence human perception and trust in robots.
2. Rich User Interface (UI) Development: With Unity's UI toolkit (UGUI or UI Toolkit), developers can create sophisticated and intuitive interfaces for humans to interact with robots. This includes dashboards, control panels, and augmented reality (AR) overlays that enhance the user experience.
3. Interactive Environments: Unity's game engine capabilities enable the creation of highly interactive and dynamic environments. This allows for simulating complex HRI scenarios, such as collaborative tasks, human-robot handover, and gesture recognition.
4. Hardware Integration: Unity can be integrated with various hardware devices, including VR/AR headsets, motion capture systems, and haptic feedback devices, to create immersive HRI experiences.
5. Multi-Platform Deployment: Simulations developed in Unity can be easily deployed across multiple platforms (PC, web, mobile, VR/AR), making it versatile for different HRI research and application contexts.

Comparison: Unity vs. Gazebo

Both Unity and Gazebo are powerful simulation tools for robotics, but they cater to different primary use cases.

Feature	Unity for Robotics	Gazebo
Primary Focus	High-fidelity visualization, HRI, user experience, realistic rendering	Physics accuracy, robot control, sensor simulation, rapid prototyping
Graphics	Excellent, advanced rendering pipelines	Functional, less emphasis on visual realism
Physics Engine	NVIDIA PhysX (highly configurable)	ODE (Open Dynamics Engine) or DART (customizable)
HRI Support	Strong (UI tools, AR/VR, interactive scenes)	Limited, primarily through external tools
ROS Integration	Excellent (Unity-ROS bridge, ROS-TCP-Endpoint)	Native and deep integration with ROS 1/2
Ecosystem	Vast asset store, game development community	Robotics-specific plugins and models
Complexity	Can be more complex for pure robotics physics	Focused on robotics, easier for physics tuning

Example: Unity-ROS Bridge (Conceptual)

Unity can connect to ROS 2 systems using packages like ROS-TCP-Endpoint (for direct TCP communication) or Unity-ROS-Bridge (which leverages standard ROS messaging). This allows Unity to act as a sophisticated visualization and HRI front-end for a ROS-based robot backend.

Conceptual Workflow:

1. ROS 2 Robot Stack: A ROS 2 system (e.g., controlling a simulated robot in Gazebo or a physical robot) publishes its state (joint states, sensor data, odometry) to ROS topics.
2. Unity-ROS Bridge: Unity runs a client that subscribes to these ROS topics.
3. Unity Visualization: Unity receives the robot's state and updates its 3D model in real-time, providing a high-fidelity visual representation.
4. Unity HRI: A human user interacts with the Unity environment (e.g., clicking on a target, speaking a command through a virtual microphone).
5. Unity-ROS Bridge (Publish): Unity publishes user commands or high-level goals back to ROS topics.
6. ROS 2 Robot Control: The ROS 2 robot stack receives these commands and executes them.

This bi-directional communication creates a powerful symbiotic relationship, combining Unity's visual prowess with ROS's robust robotics middleware.

Code Snippet (Illustrative - Unity C# script)

This is a conceptual C# script you might attach to a GameObject in Unity to subscribe to a ROS 2 topic. It requires the ROS-TCP-Endpoint or a similar Unity-ROS communication package to be set up.

using UnityEngine;
using Unity.Robotics.ROSTCPConnector;
using RosMessageTypes.Geometry; // Example ROS message type

public class RosPoseSubscriber : MonoBehaviour
{
    public string topicName = "/robot_pose"; // The ROS topic to subscribe to
    public GameObject robotModel; // The 3D model of your robot in Unity

    void Start()
    {
        ROSConnection.Get              // Get the ROSConnection singleton
            ().Subscribe<PoseStampedMsg>(topicName, ReceivePose); // Subscribe to the topic
    }

    void ReceivePose(PoseStampedMsg poseMessage)
    {
        // Update the position and rotation of your robot model in Unity
        if (robotModel != null)
        {
            // Convert ROS coordinates to Unity coordinates (adjust as needed)
            robotModel.transform.position = new Vector3(
                (float)poseMessage.pose.position.x,
                (float)poseMessage.pose.position.z, // ROS Z is often Unity Y
                (float)poseMessage.pose.position.y * -1 // ROS Y is often Unity -X or Z
            );

            robotModel.transform.rotation = new Quaternion(
                (float)poseMessage.pose.orientation.x * -1,
                (float)poseMessage.pose.orientation.z * -1,
                (float)poseMessage.pose.orientation.y,
                (float)poseMessage.pose.orientation.w
            );
        }
        Debug.Log($"Received pose: {poseMessage.pose.position.x}, {poseMessage.pose.position.y}, {poseMessage.pose.position.z}");
    }
}

Note: Coordinate system conversions between ROS and Unity are a common point of adjustment, as their conventions differ (e.g., Y-up vs. Z-up).