RE: Offline audit trail analysis

Karen, I'm most impressed. I've been rewriting the audit trails as English text, so I can shove them into syslog and a detailed review could have more context. I REALLY, REALLY like your --karen as root-- and --karen as dan-- lines. Very impressive. I've also toyed in my head with strings that might look like --karen as dan-- via ssh(a)badhost.hacker.com, but so much state information has to be kept that might not be available when things roll off the archive. You've obviously put some thought into usability. I added a keyword such as AUDIT_SRO to every record where I was testing access to security-relevant objects which makes for an easy grep later. I also include the raw event ID for more details if I want them later. My focus is the investigator looking at my audit trails four years from now trying to see the kinds of things an individual did while on my network. Presumably, they've been identified as an insider threat and an investigation is underway. I'm certain to have changed IP address resolution and user names around. OS upgrades will certainly have happened. Thanks for the reponse and continued ideas. My idea for getting "the boss" to let me share is to go to my security officer and say "if I release this code, it will help increase uniformity in the community will make the higher-echelon security folks feel like we know what we're doing in our shop." Once, they're on board with sharing, then your management chain might have fewer arguments. (we can take that discussion offline if you want) Charlie Todd Ball Aerospace & Technologies Corp. ________________________________ From: Wieprecht, Karen M. [mailto:Karen.Wieprecht@jhuapl.edu] Sent: Tuesday, September 11, 2007 4:59 PM To: Todd, Charles; linux-audit(a)redhat.com Subject: RE: Offline audit trail analysis This isn't probably as savvy as some of you would like, but I started writing it before there were any auparse libraries to help with this kind of stuff. I've done similar stuff on Solaris, Mac OSX, irix, etc. My linux version scripting isn't finished (not enough hours in the day), but it's still useful for my log reviews. At some point I hope to get out internal people to bless it enough to share with whoever might be interested in the open source world, but I'm not there yet. here's the general thing I've setting up: I use scripting to do the following on each machine : - rotate the log - (I want to try a script segment to make sure all related audit event records are not interleaved) - run an ausearch on it with some flags (interpret, all messages) and some shell commands to get all related records for a particular event onto a single line - save this processed file (with user IDs, etc. particular to that system) as something like audit.log.hostname.2006-09-10.ausearched - Store the original and "ausearched" files in a central place so I have them all gathered for my weekly log review Then weekly, I use a "reviewlogs" script to post process the "ausearched" "single line" data into something more readable. It also filters some noise I don't care about. The Chronological results look like this from various test sessions: karen FAILED TO mv /etc/nsswitch.conf /etc/nsswitch.conf.mine on patton, Permission denied on 01/15/2007 14:05:16.582 dan FAILED TO console login into patton on 01/17/2007 13:21:45.133 dan successful console login into patton on 01/17/2007 13:21:59.152 dan successful console logout from patton on 01/17/2007 13:22:26.059 karen FAILED TO ssh login into patton from ? on 01/17/2007 13:34:26.517 karen successful ssh login into patton from patton.jhuapl.edu on 01/17/2007 13:34:31.387 karen FAILED TO touch /etc/nsswitch.conf on patton, Permission denied on 01/17/2007 13:34:57.759 karen FAILED TO touch /etc/nsswitch.conf on patton, Permission denied on 01/17/2007 13:34:57.759 karen successful SU LOGIN (PAM authentication) as root on patton on 01/17/2007 13:35:15.140 karen successful SU LOGIN (PAM accounting) as root on patton on 01/17/2007 13:35:15.141 karen successful SU LOGIN (PAM session open) as root on patton on 01/17/2007 13:35:15.141 -- karen as root -- successful SU LOGIN (PAM authentication) as dan on patton on 01/17/2007 13:35:31.364 -- karen as root -- successful SU LOGIN (PAM accounting) as dan on patton on 01/17/2007 13:35:31.366 -- karen as root -- successful SU LOGIN (PAM session open) as dan on patton on 01/17/2007 13:35:31.367 -- karen as dan -- FAILED TO rm /etc/hosts on patton, Permission denied on 01/17/2007 13:35:39.129 -- karen as dan -- FAILED TO rm /etc/hosts on patton, Permission denied on 01/17/2007 13:35:40.680 -- karen as root -- successful SU LOGIN (PAM setcred) as dan on patton on 01/17/2007 13:35:44.177 -- karen as root -- successful CLOSED SU as dan on patton on 01/17/2007 13:35:44.178 karen successful SU LOGIN (PAM setcred) as root on patton on 01/17/2007 13:35:46.089 karen successful CLOSED SU as root on patton on 01/17/2007 13:35:46.089 Note that it's easy to spot users trying to make it look like other users tried to do stuff. When I review the logs, I usually sort the reformatted lines so that similar entries are all grouped (which makes it really easy to let chunks of stuff I don't care about scroll by quickly). If I see something suspicious, I'll go back and look at the stuff in chronological order to see what was going on. Here's what the above sample looks like sorted (and if I had a lot of data, the chunks of similar stuff would be more apparent): -- karen as dan -- FAILED TO rm /etc/hosts on patton, Permission denied on 01/17/2007 13:35:39.129 -- karen as dan -- FAILED TO rm /etc/hosts on patton, Permission denied on 01/17/2007 13:35:40.680 -- karen as root -- successful CLOSED SU as dan on patton on 01/17/2007 13:35:44.178 -- karen as root -- successful SU LOGIN (PAM accounting) as dan on patton on 01/17/2007 13:35:31.366 -- karen as root -- successful SU LOGIN (PAM authentication) as dan on patton on 01/17/2007 13:35:31.364 -- karen as root -- successful SU LOGIN (PAM session open) as dan on patton on 01/17/2007 13:35:31.367 -- karen as root -- successful SU LOGIN (PAM setcred) as dan on patton on 01/17/2007 13:35:44.177 dan FAILED TO console login into patton on 01/17/2007 13:21:45.133 dan successful console login into patton on 01/17/2007 13:21:59.152 dan successful console logout from patton on 01/17/2007 13:22:26.059 karen FAILED TO mv /etc/nsswitch.conf /etc/nsswitch.conf.mine on patton, Permission denied on 01/15/2007 14:05:16.582 karen FAILED TO ssh login into patton from ? on 01/17/2007 13:34:26.517 karen FAILED TO touch /etc/nsswitch.conf on patton, Permission denied on 01/17/2007 13:34:57.759 karen FAILED TO touch /etc/nsswitch.conf on patton, Permission denied on 01/17/2007 13:34:57.759 karen successful CLOSED SU as root on patton on 01/17/2007 13:35:46.089 karen successful SU LOGIN (PAM accounting) as root on patton on 01/17/2007 13:35:15.141 karen successful SU LOGIN (PAM authentication) as root on patton on 01/17/2007 13:35:15.140 karen successful SU LOGIN (PAM session open) as root on patton on 01/17/2007 13:35:15.141 karen successful SU LOGIN (PAM setcred) as root on patton on 01/17/2007 13:35:46.089 karen successful ssh login into patton from patton.jhuapl.edu on 01/17/2007 13:34:31.387 We don't have a ton of Linux systems (yet), so this strategy works well for reviewing a handful of systems from a central location. The scripts save me a lot of time interpretting any one given log entry as well as keeping me from having to log into N systems one at a time and running M commands on each one each to decide if someone was messing around on my machines or not. Karen Wieprecht ________________________________ From: linux-audit-bounces(a)redhat.com [mailto:linux-audit-bounces@redhat.com] On Behalf Of Todd, Charles Sent: Tuesday, September 11, 2007 3:32 PM To: linux-audit(a)redhat.com Subject: Offline audit trail analysis Thanks to Steve for being our biggest target for questions on this list! Has anyone talked about sane ways to do offline analysis of Linux audit logs? Presumably, this would be on another Linux system, but maybe not the same host, and probably not on the same release or with the same username/IP address access. Conceptually, ausearch would save and optionally read a system's "configuration" to be saved for interpretation later. My goal is central logging, but doing the reporting/analysis on the central host. That way, I can see a user across the Enterprise (or at least in the Linux hosts), but with all the power of ausearch for refining the report. Ideally, I would do an ausearch -ts <date> -te <date> --raw --config-to=<hostname.ausearch.config> and it would do things like saving the syscall lookup table, lookup users referenced in the reported audit trail, and resolve IP addresses references in the reported audit trail. Maybe one config file could be written for each data type in an existing format (e.g. users in /etc/passwd format, hosts in /etc/hosts format, etc.). I'm mainly after whether or not anyone has considered extending ausearch for this kind of processing? This way, an archive of raw logs could be kept along with the exact system configuration which allows offloading the audit trail analysis to a trusted location, rather than risk side effects from a rootkit. Charlie Todd Ball Aerospace & Technologies Corp. This message and any enclosures are intended only for the addressee. Please notify the sender by email if you are not the intended recipient. If you are not the intended recipient, you may not use, copy, disclose, or distribute this message or its contents or enclosures to any other person and any such actions may be unlawful. Ball reserves the right to monitor and review all messages and enclosures sent to or from this email address. This message and any enclosures are intended only for the addressee. Please notify the sender by email if you are not the intended recipient. If you are not the intended recipient, you may not use, copy, disclose, or distribute this message or its contents or enclosures to any other person and any such actions may be unlawful. Ball reserves the right to monitor and review all messages and enclosures sent to or from this email address.