Homer - A Better Way to Debug SIP
If you work in IT support or infrastructure, you may already be familiar with a wire capture tool like Wireshark for diagnosing the many problems that can occur. With VoIP integrated into your operating infrastructure, like anything else problems can and will occur. While Wireshark can be used for SIP packet capture, it can be a bit archaic plugging in increasingly complex filters on what could be a rather large capture. This is especially true as SIP fundamentally has properties that make correlating requests and responses without needing complex filtering statements beyond common filters used in Wireshark such as source and destination IP. Enter Homer and Sipgrep.
Homer does largely for SIP packet capture via Sipgrep (and other like applications) what Kibana does for Elasticsearch and Logstash - it provides a place to:
- Access singularly to retrieve SIP captures via Web UI
- Centrally store SIP capture data across many hosts
- More intuitively filter SIP capture data and correlate the data to the dialog/transactions each request/response is part of (this is immensely useful!)
- Gracefully age dated capture data you don’t want to persist for very long (though persisting longer term can be configured very easily)
- Charts! Oh glorious charts!
Core Components
The whole of Homer will depend on the following components, some of which may require some manual configuration depending on your operating system and environment. I will not cover the configuration for each component, but the quick install script in the section below will get a base working configuration going for you:
- Kamailio
- Apache (httpd)
- Elasticsearch
- mySQL (or similar)
Deployment
I’ll be frank here: Homer is not easy to deploy manually, and it really gave me a whipping when I attempted it roughly half a year ago on a CentOS 6.8 host. That said, if you happen to be using a host that isn’t Debian 8 or CentOS 7 you can take a whirl at it based on the instructions found here.
If you happened to gander at that page, the obvious path of least resistance is to stand up a fresh CentOS 7 or Debian 8 host and simply use their installer script. Do be aware you will want this host to be absolutely fresh - don’t even bother running updates on it until after you’ve successfully installed and configured Homer (to be even safer, you might want to back this host up either as a snapshot or proper backup in case updates after installation break something).
The installation script instructions are fairly straightforward, but some of the nuanced points I didn’t immediately pick up were:
-rotation.ini is a very important file and therein contains your DB credentials for Homer (which is actually used by Kamailio - the credentials in /etc/kamailio.cfg need to match those in rotation.ini) as well as connection information (such as using a socket for DB connection) and rotation threshold settings.
-sipcapture.crontab - you’ll probably want to just adapt this to a job in /etc/crontab rather than have it hanging out as an isolated job.
-configuration.php, which will usually be under /var/www/html/ or wherever you placed the Web UI for Homer. This also needs to be in agreement with the contents of rotation.ini and where you control whether Homer only attempts to and exposes the localhost for SIP capture querying or if it can attempt to query other nodes that contain SIP capture data. This is where you’d turn to if you happened to forget the DB root credentials (and also a very good reason to limit permissions and ownership of this file in particular). More on this a bit later.
Aside from the above and what the install script handles, the only additional legwork required is to allow UDP 9060 or 9061 - as a best practice limit this to an internal IP range or as a series of specific addresses you know will be forwarding SIP capture data to Homer.
A Quick Word on Distributed Topologies
If you peered into configuration.php you’ll have noticed a define(‘SINGLE_NODE’, 1) statement. Flipping this to 0 and restarting httpd will allow a Homer instance you connect to via web UI to query other Homer instances the connecting instance is made aware of. Do note that this is not meshing nor is it replication - each Homer instance can only be queried for SIP capture data it locally contains. This becomes very obvious given the web UI requires you to manually “register” other nodes (or you can insert the instances into the DB directly).
Should you choose to do something like this, you will need to either acknowledge or accept there is no replication provided. It’s also recommended that you choose only one or two Homer instances to be accessed via web UI as a security precaution.
Getting Capture Data to Homer
Homer with no data doesn’t yield us a very useful tool! So we need a tool that can capture SIP requests/responses - in NocTel’s case, VoIP servers or SIP Proxies are the ideal hosts for the capture tool; of which we prefer Sipgrep. Depending on your operating system variant, additional packages may need to be installed as prerequisite of Sipgrep.
Once Sipgrep is installed, it’s very easy to get running and shipping capture data off to Homer: sipgrep -H udp:homer_ip:homer_port
Where homer_ip is the IP of the Homer host and homer_port the UDP port Homer is set to use (this is found in the /etc/kamailio/kamailio.cfg file as #!substdef “!HOMER_LISTEN_PORT!9060!g”). Aside from this, the obvious bit of configuration is ensuring the host Homer is on allows UDP 9060 (9061 can also be used, but this requires changes elsewhere like kamailio.cfg).
If not given the flag for a discrete duration to run, Sipgrep will run indefinitely and must be terminated with a ctrl + c sequence. When run as above, Sipgrep will not save any capture data locally, so do be aware you will need to provide the proper flag and argument to save locally. However, when actually digging through dialogs in troubleshooting/debugging, the dialog chart view includes several export options to save the dialog off. Given this, you may find there is hardly any need to locally store Sipgrep captures on the local host.
What capturing application you use isn’t terribly constricted, but it should be able to forward off what it captures to a remote host.
Filtering Captured Data
Homer has a very obvious Search functionality where you can build in a semi-graphical manner the equivalent to a Wireshark filter. For basic terminology, the To and From “User” fields correspond typically to the phone number and Source/Destination Host are roughly equivalent to IP Address. Terminology may vary slightly between tools relating to SIP, such as calling the Source/Destination Host the Domain, which is what Kamailio uses when creating routing configurations.
Like Wireshark, your initial search results may be...quite numerous. However, SIP is nifty in that every packet has a CallID value which ties it into a specific dialog between source and destination. You might notice the CallID value in the table has the appearance of a hyperlink - clicking it will bring up a dialog chart view showing the exact flow of requests and responses for that particular CallID dialog. With this in mind, you can select any single packet’s CallID from the Search result table and it will construct the dialog for you. Further, from the dialog chart view you can select any of the request/responses to see the contents of the packet. This altogether makes sleuthing through what’s happening quite easy.
Viewing and Filtering
Off the bat, the most immediately useful view within Homer is the grouping of dialogs and transactions such that you can see everything that occurred in a given call by selecting the CallID column for returned activity which switches to a flow chart view of the entire dialog for that CallID. For example, a dialog from one of our SIP Proxies would show an incoming call request received by the SIP proxy who then passes that call off to the VoIP server that particular customer’s extension is set to and all the responses occurring for that dialog from trying - ringing to answer to hanging up. Additionally, depending on if it’s included or not by the parties involved in the dialog you can even view quality of service related properties like what audio codec was used for a given call as well as MOS min/max.
Related, Homer also gives some insight to how things are performing - this is more conducive to running Sipgrep for extended periods to gauge network stability and quality of service. This can be found under Panels - stats:VoIP Traffic and includes charts like answer seizure ratio and network efficiency. The former is useful in determining how many calls are successfully completing.
Finally, don’t be alarmed if you see nothing showing up for UAC Registrations - this is very common as your hosts will more than likely not allow Registration from UACs and this is handled beyond your scope.
Closing
While we certainly wish we could all have perfect conditions all the time, problems do arise and investigation becomes very necessary to remediate or assist in remediating. Tools like Homer and Sipgrep are among the toolbox we keep handy to immediately start getting to the bottom of issues experienced and reported by you, our customers.
As a company of geeks and nerds, we love to learn and teach. While the information and practices we share through articles such as this work for us for our purposes, we’re not averse to hearing from readers on what we could do better. Keep your eyes peeled for future articles on IT and networking related topics and tooling!
Addendum 2018-05-23
After playing around with Homer and Sipgrep for a while, a recurring issue was popping with Sipgrep where it would be unable to run for extended periods of time and wasn't easy to daemonize resulting in the perpetual uncertainty as to whether anything was being captured on account of Sipgrep dropping out. As an alternative to Sipgrep if you need long term captures, I recommend using Captagent instead as it can rigged up fairly easily in the context of systemd. You'll need to build it from source and will need to install libuv and libuv-devel prior to configuring and installing. The configuration is a bit more involved than Sipgrep and can be somewhat confusing initially, but if your primary use is to perform SIP capture to ship to Homer the documentation is sufficient at guiding. We've been able to successfully run Captagent for 3 months and counting on numerous busy servers - Captagent is very lightweight and generally does its job without fuss.