The Pexip Distributed Architecture
A white paper by Håkon Dahle, CTO, Pexip
March 25, 2014

At Pexip, we believe that a properly designed highly scalable meeting platform is based on a distributed architecture consisting of a number of small, low-cost units of computing. This is similar to other architectures used to deliver services at global scale.

Conferences spanning multiple units must be handled as "business as usual”, not a special case, otherwise the maximum size of a conference will be restricted by the capacity of an individual unit.

This is in contrast to the traditional way of designing large videoconferencing systems, where the accepted approach is to build a large custom hardware box with as many “ports” as possible.

Using several small units of computing instead of a monolithic platform also has significant benefits in terms of reliability. In a monolithic system, the failure of a single node has catastrophic impact. In a distributed system consisting of smaller nodes, a single node failure has far less impact and is far easier to recover from.

Pexip Infinity is a software-based, virtualized and distributed platform for voice, video and data conferencing and gateway calls. This architecture is unique in the industry as it is the only distributed architecture that is widely interoperable with business video conferencing endpoints, with all versions of Microsoft® Lync®, as well as with new technologies such as WebRTC. This white paper discusses the Pexip Infinity distributed architecture, and how this architecture works in a global deployment of voice, video and data conferencing. Key aspects of the architecture are scalability, interoperability, reliability and reduced bandwidth usage.

Deployment

Pexip Infinity runs as two or more virtual machines (VMs) on top of a bare metal hypervisor such as VMware®  ESXi® or Microsoft Hyper-V®. Any off-the-shelf Intel-based server can be used, as long as it satisfies some basic CPU and memory requirements.

Pexip Infinity consists of two different VMs:

  1. The Management Node. A Pexip Infinity deployment, regardless of size, will have just one Management Node. When first installing Infinity, the administrator downloads an OVA file. After installing this onto the hypervisor, this VM becomes the Management Node. The purpose of the Management Node is to create and manage Conferencing Nodes. The Management Node is in neither the signaling nor the media path
  2. The Conferencing Nodes. A minimal Pexip Infinity deployment will have one Management Node and one Conferencing Node. However, most Pexip Infinity deployments will have multiple Conferencing Nodes

Pexip Node configurationFigure 1: Management and Conferencing nodes run as VMs on Hypervisors on standard Intel-based X86 servers.

Scaling Up, Scaling Out

After logging in to the Management Node, the administrator can create any number of Conferencing Nodes in one or more locations. This allows for easy scaling up and scaling out. No matter how many Conferencing Nodes are created, there is a unique benefit in managing everything from a single Management Node: Configuration and provisioning data is pushed out from the Management Node to all the Conferencing Nodes; diagnostics data and event information is sent from the Conferencing Nodes back to the Management Node. There is never a need for the administrator to manage a Conferencing Node directly. This centralized management ensures that the entire deployment is configured with a consistent data set – the entire deployment acts as a single application.

Scaling Up

An administrator can easily scale a deployment up by creating several Conferencing Nodes in the same location (i.e. the same data center). Capacity can even be added “on the fly” – Conferencing Nodes can be added in a couple of minutes if more capacity is needed.

Scaling out

A typical Pexip Infinity deployment will consist of two or more locations. If a customer has three main offices such as New York, London and Tokyo, and a concentration of users in those locations, it will make sense to deploy Conferencing Nodes in all three locations. There is no inherent limit on the number of locations in a Pexip Infinity deployment. When additional locations need to be added, this can be done “on the fly” while the system is operating, with no impact on service availability.

Distributed Conferences

After Conferencing Nodes have been created, and one or more VMRs (Virtual Meeting Rooms) have been created, the system is ready to host conferences. In order to understand the benefits of a distributed conference, consider first how a traditional monolithic MCU (Multipoint Conferencing Unit) deployment works. Here we assume an organization with offices in New York and London, and they want to have a six-party video conference with three participants in each location:

Six way traditional video conference

Figure 2: A video conference with six people between two locations, in this case New York and London.

This all works fine, with three HD (High Definition) video calls across the Atlantic at perhaps 1.5 mbps each. The total bandwidth used is about 4.5 mbps in each direction.

With a distributed conference the picture is quite different. Assuming the same setup with two offices, and with a Pexip Infinity Conferencing Node in each location, the bandwidth usage changes:Pexip Distributed Conference

Figure 3: A Pexip Infinity distributed conference dramatically reduces bandwidth usage.

  1. In each direction there are multiple separate video streams: Current Speaker in HD (and Previous Speaker in HD in the opposite direction). Note that this diagram shows the worst case bandwidth usage: if the Current Speaker and Previous Speaker happen to be in the same location, then the Previous Speaker’s HD stream will not be forwarded across the Atlantic
  2. In addition, the diagram shows two scaled down streams in each direction providing the live thumbnail view of the participants who are not currently speaking. The thumbnail streams are low resolution streams that each consume less than 1/10th of an HD stream

The result is that in this conference, the distributed solution uses about 1.2 mbps in each direction, compared to 4.5 mbps in each direction for the “old school” monolithic deployment.

For more complex conferences, bandwidth usage becomes more interesting. By adding a third location (Cape Town) to this diagram, we now see the following bandwidth usage:8-way Pexip distributed conference

Figure 4: Already at an eight-way multipoint conference there are huge bandwidth savings with the Pexip Infinity distributed architecture.

1.    Bandwidth usage between New York and London

a) One HD stream in each direction, assuming Current Speaker is in New York and Previous Speaker is in London
b) Two thumbnail streams in each direction

2.    Bandwidth usage between New York and Cape Town

a) One HD stream plus two thumbnail streams from New York
b) Two thumbnail streams from Cape Town

3.    Bandwidth usage between London and Cape Town

a) Two thumbnail streams in each direction

If we compare this with a traditional deployment with a single MCU in New York, we have reduced the bandwidth between New York and London by about 60%, we have reduced the bandwidth from New York to Cape Town by about 40%, and from Cape Town to New York with about 90%. We have added 0.2 HD streams in each direction between Cape Town and London, but this is a very small price to pay for the overall reduction in WAN (Wide Area Network) bandwidth usage. In fact, this addition improves the user experience, since latency between Cape Town and London is reduced.

Consistent User Experience

A very important aspect of Pexip Infinity is the consistency of user experience. The video layout on the display of the videoconferencing system is exactly the same, regardless of how many Conferencing Nodes or locations are involved in a conference:

  • All participants view the current speaker in high resolution, plus up to seven additional recent speakers as live thumbnail views
  • The current speaker is the only exception – this participant never views himself in high resolution, instead he views the previous speaker, plus up to seven additional recent speakers as live thumbnail views

This is a consistent, highly predictable and easy to understand user experience.

Application Level Resilience

Another benefit of the distributed architecture is the additional level of resilience built in. If there is a temporary network outage between two locations, the conference will experience a temporary conference split: what was one conference will continue as two conferences (one in each location) until network connectivity is re-established, at which point the two conferences will automatically re-join. In our example with the New York to London conference, the conference would temporarily split into a New York based conference and a London based conference until the two would again re-join.

No intervention is required from either administrators or users.

Appllication level resilience

Figure 5: Application Level Resilience greatly improves on a conference experience during for instance temporary network outages, as it automatically will re-establish the conference when the network connection is re-established.

This is quite different from what would happen in the case of a legacy monolithic MCU deployment: a network outage would mean that the videoconferencing endpoints would be disconnected.

Distributed Gateway

The Pexip Infinity scalable meeting platform was recently enhanced with gateway functionality. The gateway allows an endpoint to dial another endpoint and establish an HD video call, even if their media is mutually incompatible. An example is a Lync 2010 user dialing a traditional H.323 videoconferencing system: Lync 2010 uses the Microsoft RTVideo codec for HD video calls, whereas most H.323 systems use H.264. By using Pexip Infinity as a gateway, true HD connectivity can be established, as Infinity transcodes between RTVideo and H.264.  

It is possible to use hardware-based gateways for this purpose. However, these are often expensive to purchase and complex to manage. Hence, hardware gateways tend to be deployed in a centralized data center. There are some significant disadvantages to a centralized gateway deployment:

  • If the gateway is located in the New York location, and the gateway functionality is required between two endpoints in a Tokyo office (e.g. if a Lync user calls an H.323 system in the same building in Tokyo), media must be routed through the New York office in order to be transcoded. This adds perhaps half a second of latency, resulting in a poor user experience
  • This setup would also consume significant amounts of WAN bandwidth

 A distributed gateway deployment is the obvious answer:

  • All Pexip Infinity Conferencing Nodes are also gateway resources
  • Conferencing Nodes can easily be deployed remotely in any location
  • Hairpinning on media is avoided – local calls remain local
  • No added latency – the user experience is improved

Conclusions

Pexip Infinity is built on top of a distributed architecture which provides:

  • Centralized management of any number of Conferencing Nodes in any number of locations
  • Ability to add resources at any time, without service outage
  • Significant WAN bandwidth savings in conferences that span locations
  • Consistent user experience, independent of the number of Conferencing Nodes
  • Increased resilience to temporary network outages
  • Ability to use Conferencing Nodes as gateways for point-to-point calls, avoiding hairpinning of media back to a centralized data center

If voice, video and data conferencing is to become accepted as a mission-critical tool for business use, scalability, reliability, lower cost of ownership and a consistent user experience are key requirements. This is exactly what Pexip Infinity’s distributed architecture is designed to deliver.