A few years ago, following an overnight deployment, I managed to delete a large proportion of the content on a client’s website.
There are all kinds of excuses I could make, but the fact is that I did something stupid because I wasn’t concentrating properly.
The deployment had introduced some odd bugs, and the client were making a lot of noise about them. In my sleep-deprived state, I made a bad choice about the right way to fix a problem, and my choice caused a bigger problem.
Luckily the site wasn’t actually live to the public yet, and there was a recent backup, and we were able to restore it fairly quickly, so no content was permanently lost. It could have been a lot worse. Certainly, in the moments after I realised my mistake, I thought that things were a lot worse. After talking to my colleagues, and getting help from them to sort the problem out, I was able to put the whole thing down as a learning experience.
So what did I learn?
- Think carefully before doing anything drastic
- Test any changes before you make them
- Always have a backup, and make sure you’ve tested the process for restoring backups
- Don’t try to do difficult things if you haven’t had enough sleep
If, after all of that, you do make a mistake, the most important thing is to stay calm. You may feel hot flushes of embarrassment at your own stupidity, but if you’re like most web developers, it’s not the end of the world. What’s the worst that could happen? For most people, nobody will die, and there won’t be millions of pounds lost because of your mistake, no matter how much shouting gets done. If you’re working on the kind of project where those sort of bad things could happen, you should spend a lot of time and energy on putting safeguards in place to make sure that bad things won’t happen from a single mistake.
Chances are that most of us will make a mistake sooner or later. If and when you do, here’s my advice:
- Don’t panic, and think through things logically.
- Don’t try to cover things up to save face
- Tell the right people about it, at the right time, in the right way, so that they can help you fix the problem
- Afterwards, buy drinks for the people who help you fix it.