PayPal and IPNs

As you may well know we have a website called WalkLakes which is about walking in the Lake District and the main income stream from that is that we sell PDF versions of our walks and also GPX route files for use with GPSs and mapping software.

We sell them for £1 a time so this only makes economic sense if it's completely automated, so it is and we typically only have to manually intervene in any way about once every few hundred sales.

How this works technically is that once the user has selected the walk or walks our server transfers them to PayPal's web site where PayPal take the money off them, either from the PayPal account or, mainly, their debit/credit card. PayPal then bounces them back to us and to this page which says:

Thank you for your purchase. As soon as PayPal tell us we've received payment (it usually only takes a few seconds) we will email confirmation of your purchase to you so the email should be in your inbox at [email address] shortly (and if it's not then please check your "Spam" folder). Once you have that you can then download the items from your rucksack.

Meanwhile two more things happen. Firstly PayPal send us an email to tell us they've taken payment including who the purchaser is and what they've purchased from us. That mail normally goes straight to a folder unread.

Secondly PayPal triggers an Instant Payment Notification or IPN. And this is where the fun begins.

I won't go into the technicalities but essentially an IPN is PayPal's server contacting our server with details of the payment and it contains enough information that our server can then put the walk in the user's "rucksack" and email them to say we've done that, as described in the quoted text above.

This works very well ... except when it doesn't, as was the case last Friday when some IPNs started arriving late or not at all. We were alerted to this by disgruntled users. So we started processing them by hand: so watching for incoming emails, manually adding the walks to the user's rucksack, and then emailing them to tell them we'd done that. Tedious, but the users were happy.

Then it stopped being a problem ... until Sunday when it started again in earnest and all IPNs simply stopped arriving and we were back into the cycle of processing purchases by hand.

And this is what happened after that (this is based on a comment I made to the ticket once this was resolved):

1. IPNs stop arriving
2. PayPal's status page says all is well
3. We raise a ticket
4. PayPal's status page says all is well
5. PayPal claim it's our fault and ask us to check we're not blocking IP addresses
6. We check. We're not. We tell them that
7. PayPal's status page says all is well
8. We post on the developer forum and find a significant number of people experiencing similar issues
9. PayPal's status page says all is well
10. PayPal's server sends us an automatic message saying there's a problem our end
11. Finally PayPal begrudgingly admits that there may be a problem
12. PayPal's server sends us an automatic message saying there's a problem our end
13. The problem is resolved after two days of stress

So in summary it's clear that PayPal have no watchdog process monitoring IPNs, that the status page lies, and that they assume we're at fault even when we have never been over multiple instances.

This is about the third or fourth time this has happened. It doesn't happen often thankfully but the path to resolution I describe above seldom varies: it's never their fault, right up to the point where they finally admit that it is.

It's always very stressful for us as we have to watch for mail from when we get up until 11pm or later. This time that went on for two days and Beth and I had to cope with this alongside other issues like a long meeting of Highland Council on Tuesday so it was very stressful.

Sadly we can't find another card processor offering such a competitive micropayment tariff as PayPal (we retain 90p of every £1 sale, with most card processors it would be more like 65p) but if you know of anyone do let me know.