News

Results: Free Software voice & video testing

Last weekend on Software Freedom Day the Manchester FSFE Fellowship group, assisted by additional participants in Britain and Germany, spent the afternoon testing Free Software alternatives to Skype.

Results

The 25 sets of results were recorded, and can be browsed, sorted, and searched below.

Six audio tests successfully passed (24%), as did five video tests (20%). Mumble was the most successful client, passing 100% of tests (audio only, video is not yet supported). XMPP passed four out of 14 audio tests, whereas SIP passed only one out of ten (both video and audio). Of nine apps tested, only Mumble, Pidgin, Jitsi, and Google Talk's web client achieved passes.

Conclusions

The clients tested performed more poorly than expected, probably due to network problems. One of the difficulties in testing was that generally there was little or no information about why the test had failed.

SIP clients couldn't connect successfully except when both testers used the same client, and had accounts on the same SIP server. This was surprising, especially considering that the accounts used for testing were is many cases paid for and commercially supported.

The only client and protocol which consistently did what it promised was Mumble, which had 100% test pass rate. Unlike all other clients, Mumble uses its own protocol, and also offers audio conferencing and text-to-speech by default. Mumble users are constrained to using the same server however, unlike SIP and XMPP users who should theoretically each be able to use a separate server of their choice.

It would have been useful to have a local SIP and XMPP server on the same network as the testers in order to better identify network related problems. This could have helped determine whether failures stemmed from the client, network, or server..

Examining STUN and ICE configurations was beyond the scope of our tests, but as these technologies seem critical to whether calls succeed or fail, they merit careful examination when choosing or configuring SIP and XMPP servers.

FSFE Fellows will be pleased to note that XMPP calls using FSFE's own XMPP server were often successful: several volunteers could see and hear each other when both were using their @jabber.fsfe.org accounts.

A Skype alternative?

Skype uses a variety of notorious methods to punch through network obstacles like firewalls, and to ensure it can locate the intended call recipient wherever they may be. It also benefits from being a centralised system where, if necessary, all roads lead to Rome (or a Skype managed server, at least).

Contrastingly, Free Software systems play by conventional rules of network traffic, and try and connect callers using established procedures. They also have to communicate with a variety of different servers, which may operate in ways they don't know or expect. In addition to this, some of the chat clients that we tested have only recently added support for more than text-based chatting (Gajim in Debian testing is too old for video, for example), and probably add a few problems of their own into the mix.

Based on our test results, those looking for a Free Software Skype alternative should use either Mumble, or SIP with all callers on the same server and using the same client.

Moving Forward

We had hoped to publish a conclusive compatibility table of chat clients, providing a reference for people needing to know "what works with what", and "what is easiest". Instead, we have table of subjective results, and a list of questions:

• How can we determine whether an XMPP server supports audio and video ('Jingle')?
• How can we gather more detailed information about why a call failed?
• How can we determine whether a SIP or XMPP server supports STUN (and should therefore be less prone to network issues)?

If you have answers to these questions, please share them in the comments of the blog edition of this article.

Table

More data is available in the table source code.