Naxxfish

Lately I’ve been busy trying to implement a SMS interface for a small community radio station.

The way it works is this:  we’ve got a provider who recieves the SMSes on our behalf through a geographical telephone number (i.e. it starts 01227).  Once they recieve the SMS, they will use HTTP POST to send it to our servers.  It gives us the message, a unique ID for the message, timestamp and other such things.  Once we’ve got the SMS, we have to do something with it.

Problem:  How do you handle incoming SMSes in a live broadcast situation?

When in the middle of a live broadcast, every second counts.  If a presenter asks the audience a question at the end of a link, and then there is a 3-5 minute song until the next link at which point they will want to read the answers out on air. The faster we respond to listeners, the more likely they are to interact with us again.  So the aim of the game is to get the SMS from the phone to the presenter as promptly as possible.

Approach 1: Pull PHP/HTML interface

Lets say we have a script which takes the HTTP headers and stores them all into a table in a database.  We then have another script which p0lls the database, and generates a HTML web page with all of the latest SMSes to be entered into the DB.  This is fine, but we’d need to refresh the page frequently.  We could do this with AJAX or similar.  But, unless you are refreshing the feed very often (say, once every second),  you’re going to induce an amount of latency.  Also the constant polling of the XML file will generate load on the client, and the server, and cause lots of data to flow over the network even when there are no new messages arriving.

This method I would call “pull”.  Whilst the message is being “pushed” to us by our gateway provide, we are then buffering it and waiting for the client to pull it back out of storage.  Let’s not get all starry eyed about it – HTTP XML feeds are pull.  Pull methods will always have a timliness vs. resource utilisation tradeoff.

However, this approach will work OK on any standard web server with no special requirements.

Approach 2: Ajax Push

So perhaps a better way to go would be to push the message all the way from the SMS gateway to the end user.  We change our script that it will not only store the message in a database, but also notify the clients that a new message has arrived such that they can display it to the end user immediately.

After much googling around,  I came across the APE Project (Ajax Push Engine).

APE is a full-featured OpenSource solution designed for Ajax Push. It includes a comet server and a Javascript Framework. APE allows to implement any kind of real-time data streaming to a web browser, without having to install anything on the client-side.

The interesting thing to note here is that whilst HTTP feeds are normally pull, the APE server is a custom HTTP server which is written to allow the HTTP server to push data to the clients using some clever method (what I’m guessing is the Ajax with long polling method going by what I’m seeing in Firebug).

I spent about 20 minutes setting up the APE server on a Ubuntu Server VM.  Took  a little tweaking (the docs aren’t so clear here), but works excellently.

Then I found EsenAPE, on Paul’s Blog, which is almost exactly what I need.  The only trouble is that he’s using Esendex, who is not who I’m using.  So I’ll have to rewrite the inbound part.  Also, there’s currently no way of having the historic messages shown when you first log in, which is a must for our problem.  I’ll have to do some modifications to get that working. Apparently you can write SSJS modules to interface with MySQL databases, so that is likely the route I’m going to be taking.

How exactly we end up implementing this is yet to be seen, but I’d like to try out the APE server and see what it can do for us.  The ability to push data around out network,  and from outside of our network,  in real-time is quite exciting.

We did it! High fives all round.

After a lot of preparation, sweat and blood we managed to broadcast live, for 11 hours a day for 3 days, from a pigshed in the middle of nowhere. Here’s a little tour of our makeshift station:

This is the Edit Suite and Tech Corner, fully set up. As I mentioned in Part II, we have the encoder on the back wall which encoded audio from the studio (coming up next), and sent it over the point to point wireless link. The edit machines saw a lot of action, as we (the tech team) dashed about the site transporting Marantz recorders from the various stages back to base to be edited for broadcast. They were also used to edit interviews and other bits of audio for broadcast. Once the editing was complete, the data was transferred to the Playout machine in the Studio, where they were added to the database and played out on air.

Here’s the studio. Presenter goes behind the desk, guests and co-hosts in front. We did a lot of mic swapping, and patching over the course of the weekend to accomodate different requirements. The studio ended up being used not only for live interviews and shows, but also for live musical performances! Below we see the rather excellent Harp for Hangovers performing a live set in the studio. We also had Lucy Kitt perform two tracks for us. The quality was excellent, and far better than we expected.

All in all, it was a very successful OB. The editing team have got many hours of live recordings to work with, which will hopefully start being played out on FM in the near future. As one of the OB’s I’ve helped with, I was extremely impressed at how smoothly it went. There were a few hicups, but nothing that caused any significant problems for the running of the TX. High fives indeed!

You can see the rest of my photos for the event on my flickr tag “lotf”.

• Part I
• Part II

CSR is doing an outside broadcast at the music festival called Lounge on the Farm, and broadcasting live (right now as of this writing) on the Internet and on FM 97.4MHz in Canterbury. I am a volunteer for Engineering, and the team had the challenge of making all this possible.

On Thursday morning at about 1030 we rocked up to the “venue” in The Kitchen van. We found a pig/horse/cattle shed, complete with straw and cobwebs, and various forms of Hanging Death.

We got the farmer to remove the hanging guttering you see in the above picture, and we removed most of the dangling bits of wood in the roof and put duct tape over all the sharp bits on the walls.

After a large amount of sweeping and clearing, we manage to get it into a semi-decent state so we can start putting all of the kit into the “rooms”. Meanwhile, we were setting up the 5GHz link between the pigshed and the site office where there is an Internet connection.

Here we have a wind up stand, with a crossbar holding an point-to-point wireless link we got from Solwise, and a FM antenna for our radio tuner in the studio.

Once we had the client end of the wireless link in position, we were ready to install the other end. Fortunately, we had found a place to mount the other side that was in perfect line of sight to the receiver. However, it was a little bit scary.

Here we see a terrified looking Rich on the top of a bent lighting mast, and a bit of CAT5 flying down to the portacabins below. We then ran the CAT5 through a window in the site office and into the kitchen (!!) where they have their ADSL router. We installed the Power over Ethernet Injector (which we were praying would work over that length of cable), and plugged it in. And what do you know, it worked perfectly first time.

Once the Internet connection was up, we set about building the transmission equipment.

We’ve got a little AMD Semperon machine which has a DI coming from the studio on the other side of the shed, which is then encoded into AAC+ by SAM Broadcaster and sent over the network connection to the station on the Canterbury campus. We chose AAC+ due to its ability to encode to high quality over very low bitrates. This also means that we don’t need very much bandwidth to provide the stream, which is very good when you’re out in the sticks! It’s quite resilient to packet loss too, as we found.

Most of the networking kit is running off a UPS – mainly so that if the power fluctuates (which is does, quite a lot) then we don’t break the power supply too much, and the link won’t go down (causing a lengthy rebooting process, and potential dead air).

Next to Tech Corner (also known as Rat corner, due to there being a dead rat lying there when we arrived), we have 2 edit stations, where the producers can download content from the recorder units we are using on the stage, and edit them for playout using Audacity.

And finally, the studio.

The studio consists of 4 mics, 2 mixers, a playout system, 2 CD decks and an outboard with compressors and a Saphire 90 sound card. We ended up using SAM Broadcaster for playout. There’s an area for guests in front of the desk, and 2 presenters behind it.

So far, the broadcast has been going beautifully – there was a power outage this morning (we think the main stage is using the same power feed as us, and the breakers tripped). After our Engineers moved the power over to a genny feed, we were back up and running.

I wasn’t actually on site today, however I will be there tomorrow and I hope to post an update from the pigshed!

• Part I
• Part III

Since April I’ve been volunteering some of my free time to work with the Tech team at Canterbury Student Radio CSR FM (broadcasting on 97.4 FM in the Canterbury area, as well as streaming on the Internets). It’s great fun, and gives me lots of freedom to work on interesting projects.

So far, I’ve been involved in building Linux servers, helping maintain Windows Servers, writing perl scripts, doing analog circuit design, building an Asterisk PBX, customising Cisco 7960G IP Phones, as well as lots of other odds and ends.

CSR is going to be broadcasting from the music festival in Canterbury called Lounge on the Farm. This is quite a technical challenge, as you might imagine. Broadcasting from a farm in the middle of nowhere is quite tricky. I’ve been helping out with the preparation for the Outside Broadcast, which will be going live on air this coming weekend (11th -13th July 2009).

We built 4 Windows XP computers to use on Lounge on the Farm to do an outside broadcast for CSR

One for encoding, one for playout (using JAZZLER , apparently) and two with Audacity for audio editing.

300g disks are in the playout and edit machines. And shared with peer to peer file sharing. All set up so all machines can see all drives.

Job is what one might call a good ‘un.

• Part II
• Part III