Karen,

I'm most impressed. I've been rewriting the audit trails as English text, so I can shove them into syslog and a detailed review could have more context. I REALLY, REALLY like your --karen as root-- and --karen as dan-- lines. Very impressive. I've also toyed in my head with strings that might look like --karen as dan-- via ssh@badhost.hacker.com, but so much state information has to be kept that might not be available when things roll off the archive. You've obviously put some thought into usability. I added a keyword such as AUDIT_SRO to every record where I was testing access to security-relevant objects which makes for an easy grep later. I also include the raw event ID for more details if I want them later.

My focus is the investigator looking at my audit trails four years from now trying to see the kinds of things an individual did while on my network. Presumably, they've been identified as an insider threat and an investigation is underway. I'm certain to have changed IP address resolution and user names around. OS upgrades will certainly have happened.

Thanks for the reponse and continued ideas. My idea for getting "the boss" to let me share is to go to my security officer and say "if I release this code, it will help increase uniformity in the community will make the higher-echelon security folks feel like we know what we're doing in our shop." Once, they're on board with sharing, then your management chain might have fewer arguments. (we can take that discussion offline if you want)

Charlie Todd
Ball Aerospace & Technologies Corp.

From: Wieprecht, Karen M. [mailto:Karen.Wieprecht@jhuapl.edu]
Sent: Tuesday, September 11, 2007 4:59 PM
To: Todd, Charles; linux-audit@redhat.com
Subject: RE: Offline audit trail analysis

This isn't probably as savvy as some of you would like, but I started writing it before there were any auparse libraries to help with this kind of stuff.    I've done similar stuff on Solaris, Mac OSX, irix, etc.   My linux version scripting isn't finished (not enough hours in the day), but it's still useful for my log reviews. At some point I hope to get out internal people to bless it enough to share with whoever might be interested in the open source world, but I'm not there yet. here's the general thing I've setting up:

I use scripting to do the following on each machine :

- rotate the log
- (I want to try a script segment to make sure all related audit event records are not interleaved)
- run an ausearch on it with some flags (interpret, all messages) and some shell commands to get all related records for a particular event onto a single line
- save this processed file (with user IDs, etc. particular to that system) as something like audit.log.hostname.2006-09-10.ausearched
- Store the original and "ausearched" files in a central place so I have them all gathered for my weekly log review

Then weekly, I use a "reviewlogs" script to post process the "ausearched" "single line" data into something more readable.   It also filters some noise I don't care about. The Chronological results look like this from various test sessions:

karen    FAILED TO     mv /etc/nsswitch.conf /etc/nsswitch.conf.mine   on patton,      Permission denied       on 01/15/2007 14:05:16.582

dan      FAILED TO     console login   into patton on 01/17/2007 13:21:45.133
dan      successful      console login   into patton on 01/17/2007 13:21:59.152
dan      successful      console logout          from patton on 01/17/2007 13:22:26.059

karen    FAILED TO     ssh login       into patton from ? on 01/17/2007 13:34:26.517
karen    successful      ssh login       into patton from patton.jhuapl.edu on 01/17/2007 13:34:31.387
karen    FAILED TO     touch /etc/nsswitch.conf        on patton,      Permission denied       on 01/17/2007 13:34:57.759
karen    FAILED TO     touch /etc/nsswitch.conf        on patton,      Permission denied       on 01/17/2007 13:34:57.759
karen    successful      SU LOGIN (PAM authentication)   as root         on patton on 01/17/2007 13:35:15.140
karen    successful      SU LOGIN (PAM accounting)       as root         on patton on 01/17/2007 13:35:15.141
karen    successful      SU LOGIN (PAM session open)     as root         on patton on 01/17/2007 13:35:15.141
-- karen as root --     successful      SU LOGIN (PAM authentication)   as dan          on patton on 01/17/2007 13:35:31.364
-- karen as root --     successful      SU LOGIN (PAM accounting)       as dan          on patton on 01/17/2007 13:35:31.366
-- karen as root --     successful      SU LOGIN (PAM session open)     as dan          on patton on 01/17/2007 13:35:31.367
-- karen as dan --      FAILED TO       rm /etc/hosts           on patton,      Permission denied       on 01/17/2007 13:35:39.129
-- karen as dan --      FAILED TO       rm /etc/hosts           on patton,      Permission denied       on 01/17/2007 13:35:40.680
-- karen as root --     successful      SU LOGIN (PAM setcred)          as dan          on patton on 01/17/2007 13:35:44.177
-- karen as root --     successful      CLOSED SU as dan        on patton on 01/17/2007 13:35:44.178
karen    successful      SU LOGIN (PAM setcred)          as root         on patton on 01/17/2007 13:35:46.089
karen    successful      CLOSED SU as root       on patton on 01/17/2007 13:35:46.089

Note that it's easy to spot users trying to make it look like other users tried to do stuff.    When I review the logs, I usually sort the reformatted lines so that similar entries are all grouped (which makes it really easy to let chunks of stuff I don't care about scroll by quickly). If I see something suspicious, I'll go back and look at the stuff in chronological order to see what was going on.   Here's what the above sample looks like sorted (and if I had a lot of data, the chunks of similar stuff would be more apparent):

-- karen as dan --      FAILED TO       rm /etc/hosts           on patton,      Permission denied       on 01/17/2007 13:35:39.129
-- karen as dan --      FAILED TO       rm /etc/hosts           on patton,      Permission denied       on 01/17/2007 13:35:40.680
-- karen as root --     successful      CLOSED SU as dan        on patton on 01/17/2007 13:35:44.178
-- karen as root --     successful      SU LOGIN (PAM accounting)       as dan          on patton on 01/17/2007 13:35:31.366
-- karen as root --     successful      SU LOGIN (PAM authentication)   as dan          on patton on 01/17/2007 13:35:31.364
-- karen as root --     successful      SU LOGIN (PAM session open)     as dan          on patton on 01/17/2007 13:35:31.367
-- karen as root --     successful      SU LOGIN (PAM setcred)          as dan          on patton on 01/17/2007 13:35:44.177
dan      FAILED TO     console login   into patton on 01/17/2007 13:21:45.133
dan      successful      console login   into patton on 01/17/2007 13:21:59.152
dan      successful      console logout          from patton on 01/17/2007 13:22:26.059
karen    FAILED TO     mv /etc/nsswitch.conf /etc/nsswitch.conf.mine   on patton,      Permission denied       on 01/15/2007 14:05:16.582
karen    FAILED TO     ssh login       into patton from ? on 01/17/2007 13:34:26.517
karen    FAILED TO     touch /etc/nsswitch.conf        on patton,      Permission denied       on 01/17/2007 13:34:57.759
karen    FAILED TO     touch /etc/nsswitch.conf        on patton,      Permission denied       on 01/17/2007 13:34:57.759
karen    successful      CLOSED SU as root       on patton on 01/17/2007 13:35:46.089
karen    successful      SU LOGIN (PAM accounting)       as root         on patton on 01/17/2007 13:35:15.141
karen    successful      SU LOGIN (PAM authentication)   as root         on patton on 01/17/2007 13:35:15.140
karen    successful      SU LOGIN (PAM session open)     as root         on patton on 01/17/2007 13:35:15.141
karen    successful      SU LOGIN (PAM setcred)          as root         on patton on 01/17/2007 13:35:46.089
karen    successful      ssh login       into patton from patton.jhuapl.edu on 01/17/2007 13:34:31.387

We don't have a ton of Linux systems (yet), so this strategy works well for reviewing a handful of systems from a central location.    The scripts save me a lot of time interpretting any one given log entry as well as keeping me from having to log into N systems one at a time and running M commands on each one each to decide if someone was messing around on my machines or not.

Karen Wieprecht

From: linux-audit-bounces@redhat.com [mailto:linux-audit-bounces@redhat.com] On Behalf Of Todd, Charles
Sent: Tuesday, September 11, 2007 3:32 PM
To: linux-audit@redhat.com
Subject: Offline audit trail analysis

Thanks to Steve for being our biggest target for questions on this list!

Has anyone talked about sane ways to do offline analysis of Linux audit logs? Presumably, this would be on another Linux system, but maybe not the same host, and probably not on the same release or with the same username/IP address access. Conceptually, ausearch would save and optionally read a system's "configuration" to be saved for interpretation later.

My goal is central logging, but doing the reporting/analysis on the central host. That way, I can see a user across the Enterprise (or at least in the Linux hosts), but with all the power of ausearch for refining the report. Ideally, I would do an ausearch -ts <date> -te <date> --raw --config-to=<hostname.ausearch.config> and it would do things like saving the syscall lookup table, lookup users referenced in the reported audit trail, and resolve IP addresses references in the reported audit trail. Maybe one config file could be written for each data type in an existing format (e.g. users in /etc/passwd format, hosts in /etc/hosts format, etc.). I'm mainly after whether or not anyone has considered extending ausearch for this kind of processing?

This way, an archive of raw logs could be kept along with the exact system configuration which allows offloading the audit trail analysis to a trusted location, rather than risk side effects from a rootkit.

Charlie Todd
Ball Aerospace & Technologies Corp.
This message and any enclosures are intended only for the addressee.  Please  
notify the sender by email if you are not the intended recipient.  If you are  
not the intended recipient, you may not use, copy, disclose, or distribute this  
message or its contents or enclosures to any other person and any such actions  
may be unlawful.  Ball reserves the right to monitor and review all messages  
and enclosures sent to or from this email address.


This message and any enclosures are intended only for the addressee.  Please  
notify the sender by email if you are not the intended recipient.  If you are  
not the intended recipient, you may not use, copy, disclose, or distribute this  
message or its contents or enclosures to any other person and any such actions  
may be unlawful.  Ball reserves the right to monitor and review all messages  
and enclosures sent to or from this email address.