Penthouse level

Somewhere along the way to this level you realize that all of this data has to reside somewhere. The raw data can be zipped and stored "somewhere else", you rarely need those logs but you probably want to save them for 90 days or so, if regulatory requirements say differently. For instance banking information might be required to keep up to 10 years in some countries. So you need to start thinking about your retention policies regarding different types of data. And you don't have to keep everything on-line either. Store it on backups of some kind, it is cheaper than the terabytes of disk you would need to keep everything on-line. You still have the normalized data to work with, and that is in most cases all you need, if done correctly.

The products you will be working with at this level are very, very powerful and can help you with advanced reporting tools and alerting schemes. They connect to ticketing systems an monitoring suites. They also are able to correlate data from a vast range of devices and software. Usually they come with a bunch of agents/collectors/connectors/daemons and whatnot for various sources of logs with the soul purpose of translating data from a specific device into the normalized format, which can be proprietary or an open format with open API:s so you can write your own log collector agents. Often these products gives you a choice of which approach you can take towards collecting logs. These can be, but not limited to, installing daemons or agents onto a device, triggers in a database, wrappers or listeners, they could be connectors on an appliance or software collectors on a virtual server. The sharp minds in the development departments come up with many interesting solutions, and since you often can write your own agents, you will probably come up with a few new ideas!

Not every log is as easy to read as syslog, where every event is one line. Some fine vendors have decided that if the logs are multiline events they will be easier to read. Yes, if you only use that product, and don't have anything else in your data center. Some log in XML. Remember, these ideas are the best on the market, just ask their developers. With these kinds of logs your normalization agents needs to do magic things. Be sure that you know all your log sources and their formats when shopping for a SIEM/SIM/SEM solution. They need to be able to do this magic or you will lose the correlation part in monitoring. Correlation is only effective if all pieces of the puzzle are available.

Since you've done your homework on the levels below you also have filtered out all the noise by now. So the result in your SOC/NOC monitors are only a few alerts per day, if any. Monthly/weekly/daily reports are mailed out to personnel and management. The bosses have their own webpage which shows red, amber and green lights for systems that are important for their focus area. All logs, from entrance systems to CCTV, Unix boxes to Windows7 clients, routers, switches and WiFi AP's, from New York to Birmingham, through Cape Town, Singapore and Tokyo… OK, a big shop that is. A bit of an exaggeration but you get the drift. You are on top of things, and that's why you earned the penthouse flat.