Operations

Yesterday in the afternoon at 4 pm CET, we acknowledged a failure in a central component in our data infrastructure which caused Debitoor to become unavailable for about an hour.

This meant that if you tried to log on to Debitoor within this timeframe, you probably just saw the page loading in vain.

After service was reinstored, we discovered an issue in our monitoring settings which caused the app to restart periodically.

As a result, you may have experienced issues logging in on an on and off basis for about half an hour after that.

As a user of online services, chances are you already know of the recently discovered Heartbleed vulnerability, and you’ve probably already changed passwords on your most used programs to be on the safe side.

Like 2/3 of all online services, Debitoor uses an OpenSSL security framework, and two days ago, we learned about this security vulnerability, which also affects Debitoor.

As you may have experienced, we had an unplanned suspension of service on Wednesday last week when the Debitoor application was unavailable for 2,5 hours with Quotes being unavailable for another 4,5 hours.

As we aim to have Debitoor available for you around the clock, this is not something we are proud of and we deeply apologise for any inconvenience related to this downtime.

So in the following, I’ll detail what happened, what we did to address this issue and what actions we’re taking to avoid it from happening again.

It's this time every year when Debitoor and e-conomic get together with teams from all our country offices in one big celebration of our achievements. But don't mistake this for any other corporate party. It's more of a brainstorming session where we pick our brightest minds and come up with new solutions for you.

On Saturday 29 June, we're running scheduled system maintenance on the Debitoor media.

If you're in Europe or South Africa, your work in Debitoor might be affected in the very early morning hours...

Yesterday a serious bug was introduced into Debitoor that unfortunately affected all users on the site and prevented creating and sending invoices and pretty much everything else you can do.

During the day several of you reported about the issue on e-mail and in the UserVoice forums (thank you!). Around 10.30PM CET we became aware of the problem and started to actively look for a solution. At 2.55AM CET a final fix was deployed into our production environment restoring full functionality to all users.

This morning we have completed analysis of the incident and can conclude that the bug was introduced by UserVoice (3rd party feedback platform embedded on all our pages) when they pushed out a a new version of their script that conflicted with our code.

Going forward we will either remove UserVoice as a feedback platform (which we would hate to do because we need your feedback) or move the conversation somewhere else.

With Debitoor, we have adopted a fully cloud-based model, or rather cloud-on-cloud model.

First off, Debitoor is a web application that runs in the browser - so in that sense it is a cloud product itself. Additionally, we have built the application so that it runs on top of cloud products such as Amazon Web Services (AWS), Loggly, Github, Google Analytics etc.

This doubling of cloud applications - one on top of the other - is why we refer to Debitoor as a cloud on cloud product.