What if?


Then one morning shit hits the fan, and it's all over the place. Something goes terribly wrong and production makes a distinct halt. It takes ages to file a new order. The Intranet page just won't load. Local shares won't mount. Remote Desktop Connections time out. SSH does not work… OK, this is not good, by any measures. And yes, this is an exaggeration, you probably won't see the above happening all at once unless someone blew up your data centers or you've had a very, very malicious attack where someone erased the lot.

In one way or another you have to access your devices, be it through ILO, KVM or good old RS-232, but some serious digging into your logs (that are hopefully collected) has to be done. Following your Incident response plan (you have one, right?) you call in experts in several fields to help you chase traces of faults and translate various logs from all your systems. These experts might be in-house specialists, or you could buy this expertise from consultants. The point being, you just won't cope with logging in to all of your boxes in reasonable time and look for suspicious lines of logged events, even if the amount of devices are counted in tens. So get some help.

By now you are at the investigation level. The likelihood of missing logs, or logs not containing enough data, is probably closer to 1 than 0. You decide that logging would have been a nice thing to have, and start configuring devices and applications to log, at least locally, and you are slowly starting to embrace the thought of collecting the logs at some kind of central repository. Let's be straight and honest - log management is a must have. It doesn't matter how you do it, but you need to configure logging on your systems. If you can save them centrally investigations will be so much easier.

With that in mind, and a serious incident that cost the company a lot of money which gives you a very good incentive, you go ahead and ask for money to invest in some kind of log collecting gadget, which also has rudimentary capabilities to parse the log data automatically. Of course there is a possibility that the incident wasn't serious enough for your investment board to consider investments, or your boss is reluctant to buy something that will jeopardize his budget (which of course will affect his bonus). If that is the case there are options that are free to download. Some are Open Source, and some vendors offer free versions of their commercial programs with some limitations in volume, functionality is usually the same as the commercial product. Welcome to level four, reporting!

Comments

Post a Comment

Popular posts from this blog

Overhead lines

Image
One would think that in a climate such as this in southern Australia someone would come up with a better idea than keeping all lines (cable, electricity, telephone etc) up in the air. The wind can be pretty furious at times and city planners insist on planting trees which are known to grow over time. No doubt arborists are having fun coming up with ideas how to keep city planners and power utility companies happy. Once upon a time these cables were drawn in a similar way in Stockholm, where I have lived most of my life. At some point the phone companies decided this wasn't going to work in the long run. IMHO it would make sense to dig ditches into the ground and place all cables there (at least in the cities). Of course it will cost, but I also believe there is an algorithm that will give a number that will turn black a few years time after investment (ROI). Until then I will walk around this country with a smile on my face looking at the funny trees growing along city ...
Read more

Links for August 19th 2009

So the first signs of fall appeared today. First day in weeks I couldn't tak the bike to work. Schools stated today and hence I had to take my older boy to school in the car. Ah well, all good things must come to an end. Maybe I will have the time to take apart the bike and customize it in the ways I always planned to. Or maybe it'll just come to taking out the battery for witner care, as usual. But I'll keep on dreaming. Hacks: Social skills are as important as programming knowledge when it comes to penteration testing [or it could be used in malicious ways of course, but we don't, ringht?] and the creativity amongst people obtaining knowledge never ends to stun me. Hacks: Ethical hackning seems to be popular title to hide behind. Here's a security primer on how things can be done, written in proper English and not the hacker jibberish nerds speak. Hacks: More on social skills , or sort of. How do you use your mail? Do you have different mail accounts for diff...
Read more

R.I.P. Google Reader

For years my preferred application for reading RSS-feeds has been Google Reader. It has proven reliable, fast and most of all - clean in its interface. No clutter, pix or ads. Just plain information from sources of my choice. Now Google has decided to call it quits. Sad day. Call me an information junkie, and I'm fine with it. But I am a bit baffled by the comments on this decision to lay Reader to rest. An amazing majority of comments seem to be asking the question: is RSS an outdated technology? Really? What else is out there in terms of gathering information/news/updates/searches in one application, keeping this in one application, in one unified format? Facebook sucks with its censorship. Tumbler... oh well... no. I am terribly sorry, I just can't think of anything else to use. Here are some thoughts on Reader from Gigaom interview with Chris Wetherell. So I switched to Vienna . It seems to do the job fast and relatively clutter free. I'll go with that for a whil...
Read more