Reference

SPS architecture

High-level architecture

Each installation of Scalable Pixel Streaming includes a set of standard components for managing Pixel Streaming applications, along with a stack of customisable components that are deployed for each individual Pixel Streaming application. A high-level overview of these components is depicted in the diagram below:

An installation of Scalable Pixel Streaming includes one instance of each of the following global components for managing Pixel Streaming applications:

• Kubernetes Operator: the operational logic of Scalable Pixel Streaming is encapsulated in a controller component that follows the standard Kubernetes operator pattern. The operator is responsible for managing the Kubernetes custom resources used to represent Pixel Streaming applications and for deploying the stack of per-application components on behalf of each individual Pixel Streaming application.

• REST API Server: the REST API server exposes the public API that application deployment tools use to communicate with Scalable Pixel Streaming, and is responsible for abstracting away implementation differences between cloud platforms when accessing the underlying Kubernetes custom resources.

Every individual Pixel Streaming application deployed within a given installation of Scalable Pixel Streaming includes the following stack of per-application components for serving that application to users:

• Signalling Server: the signalling server is responsible for automatically scaling Pixel Streaming applications in response to user demand. At its core, it is a standard WebRTC signalling server and it’s responsible for establishing WebRTC connections between Pixel Streaming application instances and web browsers. The Scalable Pixel Streaming signalling server expands upon this core functionality with additional features to dynamically create and destroy application instances and assign them to users as needed, and it’s designed with efficient horizontal scaling of signalling server instances in mind. The signalling server is highly flexible and customisable, with aspects of its logic delegated to plugins.

• Plugins: Scalable Pixel Streaming is highly customisable and extendable through the use of a plugin-based architecture. The following plugins are included in the stack for each Pixel Streaming application:

  • Authentication Plugin: the signalling server defers authentication logic to a plugin, and developers can choose between no authorisation implementation for Pixel Streaming applications, an OAuth 2.0 plugin implementation for making use of external identity providers, or their own custom solution.

  • Instance Manager Plugin: when the signalling server creates or destroys a Pixel Streaming application instance, it defers the underlying implementation of these operations to a plugin. Scalable Pixel Streaming currently ships with a plugin implementation called the Kubernetes Instance Manager Plugin, which is responsible for the creation and deletion of Pixel Streaming application instances using Kubernetes objects. Alternative instance manager plugin implementations are not currently supported, and developers cannot deploy their own custom instance manager plugins. However, the Kubernetes Instance Manager Plugin is highly configurable, and also supports extension through a plugin mechanism of its own.

  • Instance Customisation Plugins: when the Kubernetes Instance Manager Plugin carries out the creation of a Pixel Streaming application instance on behalf of the signalling server, it supports invoking a list of custom plugins to modify the instance details. This provides a convenient mechanism for customising startup parameters based on information that cannot be known ahead of time (e.g. identifying an appropriate Unreal Engine dedicated server instance and instructing the application instance to connect to it) or triggering custom behaviour in response to the creation of an application instance without actually modifying its details.

• Application instances: these are the individual instances of the Pixel Streaming application itself that are created to serve user sessions. Each instance is either a GPU-accelerated Linux container or a Windows VM on cloud platforms where Scalable Pixel Streaming supports their use.

Automatic scaling

The signalling server is the component of Scalable Pixel Streaming that is responsible for automatically scaling Pixel Streaming applications in response to user demand. Scaling is triggered in response to two key events: when a user establishes a connection to request a streaming session (a “user connection” event) or when an existing user with an active streaming session disconnects (a “user disconnection” event.) The diagram below depicts the scaling operations that the signalling server triggers in response to these events:

The following operations are performed when a user connection event occurs:

1. The signalling server associates the incoming WebSocket connection with a new user session, so that it could terminate the session later when it detects that the user has disconnected.

2. The signalling server communicates with the web browser and the authentication plugin to authenticate the user.

3. The signalling server instructs the Kubernetes Instance Manager Plugin to allocate a new Pixel Streaming application instance for the user session.

4. The Kubernetes Instance Manager Plugin invokes each of its specified instance customisation plugins to modify the details of the allocation request (e.g. to modify application startup parameters).

5. The Kubernetes Instance Manager Plugin fulfills the allocation request by creating Kubernetes resources to spin up a container or VM.

6. The newly-allocated Pixel Streaming application instance connects to the signalling server, and a WebRTC connection is then established between the application instance and the web browser.

The following operations are performed when a user disconnection event occurs:

1. The signalling server identifies the user session associated with the WebSocket connection that was just closed.

2. The signalling server instructs the Kubernetes Instance Manager Plugin to deallocate the Pixel Streaming application instance associated with the user session.

3. The Kubernetes Instance Manager Plugin fulfills the deallocation request by deleting the Kubernetes resources that represent the container or VM.

4. The Pixel Streaming application instance is destroyed when the container or VM is deleted.

Components and APIs

The full architecture of Scalable Pixel Streaming encompasses numerous software components that communicate over a variety of APIs and protocols. The diagram below depicts all of the components and the APIs that they use to communicate:

Scalable Pixel Streaming encompasses the following software components:

• Kubernetes Operator: a standard Kubernetes operator that encapsulates the operational logic of Scalable Pixel Streaming. The operator manages the Kubernetes custom resources that represent the internal state of Scalable Pixel Streaming, and monitors resources for changes made by the REST API server on behalf of deployment tools. When changes are made to the custom resources that represent the configuration for a given Pixel Streaming application, the operator is responsible for deploying the appropriate per-application software components to reflect the configuration data.

• REST API Server: exposes the public REST API that application deployment tools use to communicate with Scalable Pixel Streaming, and is responsible for abstracting away implementation differences between cloud platforms. To achieve this, the REST API server translates between generalised API-level entities, such as Applications and Versions, and the underlying Kubernetes custom resources that represent their internal implementation details. These resources are then acted upon by the operator.

• Signalling Server: an enhanced version of a standard WebRTC signalling server with additional functionality and responsibilities. The signalling server is the component that accepts and monitors incoming user connections and manages the lifecycle of streaming sessions in response to connection status changes. As part of this lifecycle management process, the signalling server is responsible for authenticating users, scaling Pixel Streaming applications in response to demand, and establishing WebRTC connections between Pixel Streaming application instances and web browsers. To facilitate extensibility and customisation, the signalling server delegates the implementation details of authentication and application scaling operations to plugins.

• Authentication plugins:

  • No-op Authentication Plugin: disables authentication entirely and allows any user with your application endpoints to create instances of these applications.

  • OAuth 2.0 Authentication Plugin: an authentication plugin implementation that communicates with external identity providers via the standardised OAuth 2.0 protocol to authenticate users. Developers can provide a list of identity providers for use with the plugin, and if multiple identity providers are available then the frontend can allow users to select which provider to authenticate with.

  • Any number of custom authentication plugins provided by developers. These plugins are not managed by Scalable Pixel Streaming and must be deployed separately by the administrator of the Kubernetes cluster.

• Kubernetes Instance Manager Plugin: provides the implementation for the scaling operations that the signalling server performs in response to user demand. The Kubernetes Instance Manager Plugin creates and deletes Pixel Streaming application instances by manipulating Kubernetes objects, and also provides garbage collection functionality to ensure that resources are always cleaned up, even if the signalling server crashes or is restarted. The plugin also supports customisation and extension through the use of instance customisation plugins.

• Any number of instance customisation plugins provided by developers. These plugins can modify the details of a Pixel Streaming application instance at the time it is created, or simply trigger custom behaviour in response to the instance creation event without performing any modifications. Instance customisation plugins are not managed by Scalable Pixel Streaming and must be deployed separately by the administrator of the Kubernetes cluster.

• Application instances: individual instances of Pixel Streaming applications that are created to serve user streaming sessions. The Kubernetes Instance Manager Plugin manages the lifecycle of a given instance by creating and destroying Kubernetes objects that represent the underlying cloud resources used to run that instance. Instances are either GPU-accelerated Linux containers or Windows VMs on cloud platforms where Scalable Pixel Streaming supports their use.

• Deployment tools:

  • CLI tool: the official command-line client for Scalable Pixel Streaming. The CLI tool provides commands for managing REST API entities across multiple installations of Scalable Pixel Streaming, and supports the use of JSON files as input data to facilitate a Configuration as Code approach to managing Pixel Streaming applications. The client tool is suitable for both interactive use and for use in automated pipelines or scripts.

  • Frontend: the HTML, CSS and JavaScript code that runs in client web browsers to connect to Pixel Streaming applications and interact with them. The Scalable Pixel Streaming frontend is implemented as a modular and extensible TypeScript library that acts as a drop-in replacement for the frontend code that Epic Games ships with the reference implementation of Pixel Streaming, and ships with a number of example implementations that demonstrate the use of the library.

Software components communicate via the following APIs:

• REST API: the public REST API of Scalable Pixel Streaming, exposed by the REST API server. For details of this API, see the REST API Reference.

• Authentication Plugin API: the gRPC API that the signalling server uses to communicate with authentication plugins.

• Instance Manager Plugin API: the gRPC API that the signalling server uses to communicate with instance manager plugins.

• Instance Customisation Plugin API: the gRPC API that the Kubernetes Instance Manager Plugin uses to communicate with instance customisation plugins.

• Pixel Streaming Signalling API: the WebSocket API that web browsers and the Unreal Engine’s Pixel Streaming plugin use to communicate with the signalling server. The signalling API used by Scalable Pixel Streaming is a strict superset of the API used by the reference implementation, with extensions for authentication and instance management functionality.

• Pixel Streaming Control API: the WebRTC data channel API that web browsers use to communicate with the Unreal Engine’s Pixel Streaming plugin. This API is used for transmitting user input events from the browser to the Pixel Streaming application and for exchanging control messages between the Pixel Streaming plugin and the web browser. Scalable Pixel Streaming does not make any modifications to the control API from the reference implementation.

Licenses and availability

Components

Some of the software components encompassed by Scalable Pixel Streaming are freely available under open-source licenses, whereas other components are proprietary and are governed by the TensorWorks End User License Agreement (EULA) for cloud-based products. Use of the proprietary components is subject to the terms of the EULA.

APIs

Definitions for all of the communication protocols used by Scalable Pixel Streaming are freely available (or will soon be available) for developers to use when creating plugins or other software that integrates with Scalable Pixel Streaming. The API definitions are all licensed under the MIT License, except for the Pixel Streaming APIs, which are governed by the “Sharing Examples” clause of the Unreal Engine EULA.