Today was a sad day. I found out my streaming mirror/microblog server had been cracked. They exploited webmin and set up shop. Alas, there is no one to blame but myself. This is how I believe it went down.
About a month ago I tried to ssh into the streaming server but could not get in. Since this is a co-lo vms I contacted support informing them of the problem and asking them to kick the server. I suspected it was from a sour upgrade. They replied back that they found a problem in the sshd_config file and that things were back to normal.
This server had been running some three or more years now, it was the same debian install just upgraded. The company, tektonic, had gone through some changes with their systems and I never kept up with their administrative interface changes as I always used ssh, not their tools. Thus, having found myself in this predicament a few times in the past I figured I should find another way to remotely kick the system.
I looked around at their docs and tools but could not get access to my system. I figured it was just because I never bothered to sign up when they changed tools those few times. Thus, I thought I would try out webmin. I slapped that on and then got pulled away with something else and eventually forgot about it.
Cut to today. Pat was having trouble getting gwibber working properly and it’s largely because I don’t have apache2 ssl running. So I figured it was high time I rectified that. I went in to get the system up to date and configure ssl. That’s when I noticed problems. Apt error-ed out complaining that ufc and openssh server could not be upgraded. I kept getting bad hashes on ufc files. After some frantic searching around the web I set to some other tasks which led me to learn that I would get the weird errors using common command like cp. In particular, bad address when preserving permissions. That’s when I knew something was amiss.
Whoever cracked the machine did not do a good job cleaning up after themselves or hiding their work. I found a suspect process running and tracked it down to a hidden directory in /tmp. There I learned they had exploited a hole in webmin and were using my system to find other targets running webmin. Well needless to say I shut that down, tried to preserve what I could from the system and wiped it clean.
It’s a good thing I don’t use the same passwords everywhere. Even more, most of the data on there was not critical. The big downside though was that this was the laconica server. So we pretty much lost that, the posts, subscriptions and users on there. While I preserved what I could, it was little use.
Long story short, after some fussing with Laconica for a fifth time, I finally got it up and running again successfully. I am at the latest version and it’s running pretty smooth. So if you had an account on the microblog, you will have to re-create it. If you were subscribed to someone on there you will probably have to re-subscribe. For that I am ashamed and apologize.
It really stinks though, three+ years running and they finally get in because of my oversight. That’s how it happens, one little slip up and you leave yourself wide open. But, events like this help one learn and you can bet I won’t make that same mistake again, at least not for a long time.
So remember: 1) Do your backups, 2) Keep your systems up to date, and 3) Never run any services you do not absolutely need and those you do; make sure you lock them down as tight as possible.
Have a happy, I go to hang my head in shame.