Free Software is key for fair competition. We advocate for a strict and effective enforcement of the Digital Markets Act in a developer friendly way. Help us with a donation to achieve that!

Donate Now

This page has not been translated yet. Please help us to translate this and other pages on fsfe.org, so people can read our message in their native language.

News

Results: Free Software voice & video testing

on:
Written by  on   (updated on  ) 

Last weekend on Software Freedom Day the Manchester FSFE Fellowship group, assisted by additional participants in Britain and Germany, spent the afternoon testing Free Software alternatives to Skype.

Results

The 25 sets of results were recorded, and can be browsed, sorted, and searched below.

Six audio tests successfully passed (24%), as did five video tests (20%). Mumble was the most successful client, passing 100% of tests (audio only, video is not yet supported). XMPP passed four out of 14 audio tests, whereas SIP passed only one out of ten (both video and audio). Of nine apps tested, only Mumble, Pidgin, Jitsi, and Google Talk's web client achieved passes.

Conclusions

The clients tested performed more poorly than expected, probably due to network problems. One of the difficulties in testing was that generally there was little or no information about why the test had failed.

SIP clients couldn't connect successfully except when both testers used the same client, and had accounts on the same SIP server. This was surprising, especially considering that the accounts used for testing were is many cases paid for and commercially supported.

The only client and protocol which consistently did what it promised was Mumble, which had 100% test pass rate. Unlike all other clients, Mumble uses its own protocol, and also offers audio conferencing and text-to-speech by default. Mumble users are constrained to using the same server however, unlike SIP and XMPP users who should theoretically each be able to use a separate server of their choice.

It would have been useful to have a local SIP and XMPP server on the same network as the testers in order to better identify network related problems. This could have helped determine whether failures stemmed from the client, network, or server..

Examining STUN and ICE configurations was beyond the scope of our tests, but as these technologies seem critical to whether calls succeed or fail, they merit careful examination when choosing or configuring SIP and XMPP servers.

FSFE Fellows will be pleased to note that XMPP calls using FSFE's own XMPP server were often successful: several volunteers could see and hear each other when both were using their @jabber.fsfe.org accounts.

A Skype alternative?

Skype uses a variety of notorious methods to punch through network obstacles like firewalls, and to ensure it can locate the intended call recipient wherever they may be. It also benefits from being a centralised system where, if necessary, all roads lead to Rome (or a Skype managed server, at least).

Contrastingly, Free Software systems play by conventional rules of network traffic, and try and connect callers using established procedures. They also have to communicate with a variety of different servers, which may operate in ways they don't know or expect. In addition to this, some of the chat clients that we tested have only recently added support for more than text-based chatting (Gajim in Debian testing is too old for video, for example), and probably add a few problems of their own into the mix.

Based on our test results, those looking for a Free Software Skype alternative should use either Mumble, or SIP with all callers on the same server and using the same client.

Moving Forward

We had hoped to publish a conclusive compatibility table of chat clients, providing a reference for people needing to know "what works with what", and "what is easiest". Instead, we have table of subjective results, and a list of questions:

If you have answers to these questions, please share them in the comments of the blog edition of this article.

Table

More data is available in the table source code.

From Client From OS From DE To Client To OS To DE Report Local type Audio Video Notes Account 1 Account 2
Pidgin 2.7.3 Crunchbang openbox Pidgin 2.1 Fedora 17 Gnome Anna true XMPP pass pass
Pidgin 2.7.3 Crunchbang openbox gajim 0.15 Fedora 17 Gnome Anna true XMPP fail pass
Jitsi 1.0-build.3967 Ubuntu 12.04 Gnome/Unity Gajim 0.15 Fedora 17 Gnome Jamie true XMPP fail fail Options were greyed out on Gajim
Pidgin 2.10.6 Ms Win XP na Pidgin 2.7.3 crunchbang openbox David false XMPP fail fail
Empathy 3.4.2.3 Fedora 17 Gnome Empathy 3.4.2.3 Ubuntu 12.04 Gnome/Unity Sam true XMPP fail fail only text chat worked - no indication of a call request was received for either video or voice calls
snom220 hardphone snom220-SIP 3.56z Snom Linux NA Wookey false SIP fail fail call connection failed: 'No such domain' - local DNS failure 3858102@SIPgate.co.uk wellybob@jabber.me
snom220 hardphone snom220-SIP 3.56z Snom Linux NA wookey false SIP fail fail call connection failed: 'No such domain' - local DNS failure 3858102@SIPgate.co.uk samtuke@SIP.linphone.org
snom220 hardphone snom220-SIP 3.56z Snom Linux NA n900 app Maemo 5 hildon wookey false SIP fail fail call connection failed: 'No such domain' - local DNS failure 3858102@SIPgate.co.uk 2542654@localphone.com
n900 21.2011.38-1 Maemo 5 Hildon linphone wookey false SIP fail fail call connection failed: 'unable to call, contact offline 9178850@SIPgate.co.uk samtuke@SIP.linphone.org
n900 21.2011.38-1 Maemo 5 Hildon n900 app Maemo 5 hildon wookey false SIP fail fail call connection failed: 'unable to call, contact offline 9178850@SIPgate.co.uk
n900 21.2011.38-1 Maemo 5 Hildon snom220 hardphone snom220-SIP 3.56z Snom Linux NA Sam false SIP fail fail call connection failed: no response" 1st time, call unauthorised 2nd time" samtuke@SIP.linphone.org | 2542654@localphone.com
n900 21.2011.38-1 Maemo 5 Hildon n900 21.2011.38-1 Maemo 5 Hildon Sam false SIP fail fail call connection failed samtuke@SIP.linphone.org | 2542654@localphone.com 3858102@SIPgate.co.uk
Jitsi 1.0-build.3967 Ubuntu 12.04 Gnome/Unity Jitsi 1 Fedora 17 Gnome Jamie true XMPP fail fail ICE failed 9178850@SIPgate.co.uk
Pidgin unknown Ubuntu 12.04 Gnome/Unity Pidgin 2.1 Fedora 17 Gnome Jamie true XMPP fail fail ICE failed
mumble 1.2.3 Fedora 17 Gnome Murmur murmur-static_x86-1.2.3-412 Debian Wheezy Sam false mumble pass na using bens server, it just works
Pidgin 2.7.3 Crunchbang openbox kopete 1.2.5 Fedora 17 Gnome Anna true XMPP fail fail kopete couldn't initiate call with pidigin, only pidgin could initiate with kopete
Kopete 1.0.0 Crunchbang openbox Pidgin 2.1 Fedora 17 Gnome Anna true XMPP fail fail
Jitsi 1.0-build.3967 OSX 10.6.8 NA Pidgin? ? Ubuntu KDE Robin false XMPP pass pass notes
Jitsi 1.0-build.3967 OSX 10.6.8 NA Jitsi 1.0-build.3967 Fedora Gnome Robin true SIP pass pass Audio one one was pretty garbled, both using SIP2SIP
Jitsi 1.0-build.3967 OSX 10.6.8 NA Jitsi 1.0-build.3967 Fedora Gnome Robin true SIP fail fail failed using different servers: SIP2SIP and localphone
linphone 3.5.2 Fedora 17 Gnome Jitsi 1.0-build.3967 OSX 10.6.8 NA Sam true SIP fail fail failed using different servers: SIP2SIP and linphone
Pidgin 2.10.3 Kubuntu 12.04 KDE Google Talk, webclient ??? OSX, Android NA Steffi true XMPP pass pass
Pidgin 2.10.3 Kubuntu 12.04 KDE Google Talk, webclient ??? ??? NA Steffi false XMPP pass fail Video was fine, audio worked but the quality was very low
Psi 0.14 Fedora 17 Gnome psi+ 0.15.5337 OSX NA Sam true XMPP fail fail
Psi/Psi+ v0.15.5339-webkit Fedora 17 Gnome psi+ Psi+ v0.15.3910 KKubuntu 12.04 KDE Sam true XMPP fail fail auth requests never appeared, voice chat enabled via pssimedia client