7:35 a.m.
There have been a few requests on this list for a
single-line-per-event output format.
From what I understand, supporting this feature in the kernel is a
little challenging due to the potential memory/cache requirements -
having to save off the 'pieces' of an event from initiation to exit,
could be quite expensive when we're talking in-kernel resources.
Hence, we're left with an audit output that could:
* Have an arbitrary number of lines per unique event,
* Be chronologically distributed in some cases (event lines may be
spread over several seconds),
* Be out of sequence (numerically - two lines from event A, might be
followed by 1 line from event B, then another line from event A,
followed by the rest of event B).
* Have multiple 'items' with the same name (eg: source/dest UID's for
a CHMOD have the same 'name/key', though they are on different lines).
(Please let me know if these assumptions are incorrect!)
However, the dispatcher infrastructure offers the potential to
implement this sort of functionality in user-space.
As such, I've been playing with some perl that should translate
SYSCALL events to something that should be reasonably parse-able by
follow-on processing applications that expect events in a single-line
form (eg: logwatch, a SQL-based data injector, Snare or Snare Server).
Here's a sample line. Note that events are in header/data format (tab
delimited between components, comma delimited within a component).
rhel4 LinuxKAudit event,11,20060509 06:36:25
user,0(root),0(root),0(root),0(root) process,25068,id
path,/usr/bin/id return,0,yes a0,9f96ba8 a1,9f96b28
a2,9f84b40 a3,9f96b28 arch,40000003 auid,4294967295
exe,/usr/bin/id fsgid,0 fsuid,0 items,2 sgid,0 suid,0
The program works roughly as follows:
while read line
break line up into key/value pairs
pop the key/value data into an associative array (with a key of the
event number)
if we have an items=x key/value pair saved off for this event
number, and we have 'x' PATH-related lines now, then we must have a
complete event. Push it out.
ALSO push out any events that we haven't had any new lines for, for
more than 15 seconds.
..
We also getpwname/getgrname (with an internal cache to avoid recursive
audit events), and an internal simple realpath() to turn
/path/to/blah/../../to/somewhere into /path/to/somewhere.
The raw perl is attached. Does anyone have any comments, or
suggestions? (I don't care about structure at this point - it's early
days yet - I'm sure perl aficionados could do the entire program in a
single line).
In particular, comments would be welcome on how to absolutely,
programatically determine when an 'event' is complete, and it is safe
to push out our final 'line'.
Regards,
Leigh.
9:22 a.m.
On Monday 22 May 2006 08:35, Leigh Purdie wrote:
From what I understand, supporting this feature in the kernel is a
little challenging due to the potential memory/cache requirements -
having to save off the 'pieces' of an event from initiation to exit,
could be quite expensive when we're talking in-kernel resources.
This is pretty much what the audit system does in the kernel. It saves
everything in the context until it decides to output an event. Then it dumps
all records associated with the event into the netlink queue.
Hence, we're left with an audit output that could:
* Have an arbitrary number of lines per unique event,
* Be chronologically distributed in some cases (event lines may be
spread over several seconds),
Doubtful.
* Be out of sequence (numerically - two lines from event A, might be
followed by 1 line from event B, then another line from event A,
followed by the rest of event B).
I haven't seen this except when we introduced a bug in the kernel.
* Have multiple 'items' with the same name (eg: source/dest
UID's for
a CHMOD have the same 'name/key', though they are on different lines).
(Please let me know if these assumptions are incorrect!)
The record type has something to do with the meaning of any items.
However, the dispatcher infrastructure offers the potential to
implement this sort of functionality in user-space.
Yes, I'm glad to see more people use this feature. Please note that there is
an API version number that dictates the format of the records passed through
the event dispatcher mechanism. Right now, the version is 0. It is the 1st 32
bit word sent in the dispatcher header. I should probably add a specification
to my people page so that everyone knows the rules.
Also, the audit daemon can be told not to log anything locally so that its all
done across the network. Its in the auditd.conf man page. The dispatcher is
responsible for adding the machine or host's name if that's needed in the
record.
We also getpwname/getgrname (with an internal cache to avoid
recursive
audit events), and an internal simple realpath() to turn
/path/to/blah/../../to/somewhere into /path/to/somewhere.
Not looking at your code...but you probably are referring to munging CWD &
PATH records to create complete path? If not, that's what you need to do.
The raw perl is attached. Does anyone have any comments, or
suggestions?
Just one, at some point I'm going to pick up work again on the audispd
program. It will have a plugin architecture that will allow multiple programs
like yours to share the event data. Its likely to be a couple months before I
start back into it...but I just wanted you to be aware of the long range
plans.
But until then, its fine to grab the event dispatcher hook and use it
exclusively.
In particular, comments would be welcome on how to absolutely,
programatically determine when an 'event' is complete, and it is safe
to push out our final 'line'.
When the serial number changes is what I've been using in ausearch/aureport.
The events should all be out within a second. When it decides to dump the
records for a single event, it all happens quickly.
-Steve
7:45 p.m.
> * Be chronologically distributed in some cases (event lines may
be
> spread over several seconds),
Doubtful.
Excellent - thanks for that. I thought I spotted this issue very early
on in development, but can't find the problem now. That makes it much
easier if I'm going to rely on timeouts to push events out - I can
reduce the threshold from 10 seconds to closer to a second.
> * Be out of sequence (numerically - two lines from event A,
might be
> followed by 1 line from event B, then another line from event A,
> followed by the rest of event B).
I haven't seen this except when we introduced a bug in the kernel.
Here's a sequence from the current auditd on RHEL4 U3:
type=PATH msg=audit(1146852361.936:228389): name="/etc/ld.so.cache"
flags=101 inode=32717 dev=fd:00 mode=0100644 ouid=0 ogid=0 rdev=00:00
type=SYSCALL msg=audit(1146852361.936:228390): arch=40000003 syscall=5
success=yes exit=3 a0=b7fbc918 a1=0 a2=0 a3=b7c4f8 items=1 pid=28998
auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0
comm="cat" exe="/bin/cat"
type=CWD msg=audit(1146852361.936:228390): cwd="/"
type=PATH msg=audit(1146852361.936:228390): name="/lib/tls/libc.so.6"
flags=101 inode=148688 dev=fd:00 mode=0100755 ouid=0 ogid=0
rdev=00:00
type=CWD msg=audit(1146852361.931:228387): cwd="/"
type=PATH msg=audit(1146852361.931:228387):
name="/lib/libselinux.so.1" flags=101 inode=148695 dev=fd:00
mode=0100755 ouid=0 ogid=0 rdev=00:00
type=SYSCALL msg=audit(1146852361.937:228391): arch=40000003 syscall=5
success=yes exit=3 a0=b7f11918 a1=0 a2=b7f08000 a3=b7c4f8 items=1
pid=28997 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0
fsgid=0 comm="init" exe="/sbin/init"
type=CWD msg=audit(1146852361.937:228391): cwd="/"
Note the 228389, 228390, 228390, 228387, 228387, 228391, 228391 sequence.
Here's a quick method to identify them, based on eventnumber:
cat audit.log | awk '{print $2}' | cut -d: -f2 | sed 's/)//' | perl -e
'while($i=<STDIN>){chomp($i); if($i<$c){print "Reversal! (expected $c
or more, got $i)\n";} $c=$i; }'
Reversal! (expected 228390 or more, got 228387)
Reversal! (expected 228898 or more, got 228896)
Reversal! (expected 231981 or more, got 231973)
Reversal! (expected 232595 or more, got 232582)
Reversal! (expected 232610 or more, got 232604)
Reversal! (expected 232623 or more, got 232610)
Reversal! (expected 232713 or more, got 232689)
Reversal! (expected 232723 or more, got 232718)
...
> However, the dispatcher infrastructure offers the potential to
> implement this sort of functionality in user-space.
Yes, I'm glad to see more people use this feature. Please note that there is
an API version number that dictates the format of the records passed through
the event dispatcher mechanism. Right now, the version is 0. It is the 1st 32
bit word sent in the dispatcher header. I should probably add a specification
to my people page so that everyone knows the rules.
It would actually be very helpful, if the dispatcher could receive
normal, newline/null terminated strings, rather than having to
interpret C structures (with the record type expanded as ASCII text).
Working with C structures in languages that are suited to text
processing (perl, python, php, etc), is challenging at best, and it's
these sort of languages that are most likely to do post-processing of
audit log data.
Any thoughts on this?
Also, the audit daemon can be told not to log anything locally so
that its all
done across the network. Its in the auditd.conf man page. The dispatcher is
responsible for adding the machine or host's name if that's needed in the
record.
Yep. That's the ultimate goal. During the testing process, I'll just
avoid auditing write()s and related calls for the moment.
We also getpwname/getgrname (with an internal cache to avoid
recursive
> audit events), and an internal simple realpath() to turn
> /path/to/blah/../../to/somewhere into /path/to/somewhere.
Not looking at your code...but you probably are referring to munging CWD &
PATH records to create complete path? If not, that's what you need to do.
Yes. However, can the dispatcher guarantee that the paths in CWD and
PATH are fully resolved? (ie: we'll never see a
"/tmp/.././etc/./passwd" in PATH ?)
Regards,
Leigh.
9:22 a.m.
On Monday 22 May 2006 20:45, Leigh Purdie wrote:
> > * Be chronologically distributed in some cases (event lines
may be
> > spread over several seconds),
>
> Doubtful.
Excellent - thanks for that. I can reduce the threshold from 10 seconds to
closer to a second.
Yes. 2 seconds ought to get it if you only use time to the second. If you
handle milliseconds, maybe .5 second would do it. Generally the only delay
would be from the audit daemon doing its processing before handing you the
next event.
> > * Be out of sequence (numerically - two lines from event A,
might be
> > followed by 1 line from event B, then another line from event A,
> > followed by the rest of event B).
>
> I haven't seen this except when we introduced a bug in the kernel.
<snip>
Note the 228389, 228390, 228390, 228387, 228387, 228391, 228391 sequence.
Right, but they are not interlaced. You can protect against this just to be
safe.
> Yes, I'm glad to see more people use this feature. Please
note that there
> is an API version number that dictates the format of the records passed
> through the event dispatcher mechanism. Right now, the version is 0. It
> is the 1st 32 bit word sent in the dispatcher header. I should probably
> add a specification to my people page so that everyone knows the rules.
It would actually be very helpful, if the dispatcher could receive
normal, newline/null terminated strings, rather than having to
interpret C structures (with the record type expanded as ASCII text).
The audit daemon is just handing the same real-time information it received to
the dispatcher interface. I hate to make just this one change to the
protocol, though. (To make this change, I need to bump the protocol version
number. I hate to waste them.) If there were other changes that need to be
made, I'd feel better about doing the change now.
I also put an interface specification here:
http://people.redhat.com/sgrubb/audit/audit-rt-events.txt
If you see anything that need clarification, please let me know.
Yes. However, can the dispatcher guarantee that the paths in CWD and
PATH are fully resolved? (ie: we'll never see a
"/tmp/.././etc/./passwd" in PATH ?)
You could very well see that. I just tested what you wrote and it shows up
relative path and all. Also...please note that if the file name has a space
in it, you get a ascii hex representation of the file name.
-Steve
8:26 p.m.
Thanks for that Steve,
Note the 228389, 228390, 228390, 228387, 228387, 228391, 228391
sequence.
Right, but they are not interlaced. You can protect against this just to be
safe.
Actually, they are. Hang on, and I'll give you a bigger example (event numbers):
$ cat audit.log | awk '{print $2}' | cut -d: -f2 | sed 's/)//' | grep
"2283[89][0-9]"
...
228386
228386
228386
228387
228388
228388
228388
228388
228389
228389
228389
228390
228390
228390
228387
228387
228391
228391
228391
Note the 228387 four from the top, and also after the 228390's.
Is this something that should be considered a bug? If so (and it's
likely to be fixed in RHEL4 in the near future), then I'll be able to
greatly simplify the dispatcher code. :)
The audit daemon is just handing the same real-time information it
received to
the dispatcher interface. I hate to make just this one change to the
protocol, though. (To make this change, I need to bump the protocol version
number. I hate to waste them.) If there were other changes that need to be
made, I'd feel better about doing the change now.
No problems. I'm in no hurry for it - I can definitely work around
this issue for our Snare-related requirements. :)
However, it's probably something that others would appreciate.
The protocol may not need to change explicitly, two separate paths in
auditd may be viable - ie:
dispatcher = /path/to/something-that-receives-a-raw-feed or
asciidispatcher = /path/to/something-that-receives-nullterminated-ascii-to-stdin
.. if 'asciidispatcher' is active, auditd does a little extra work to
push the line out in a 'nicer' format.
> However, can the dispatcher guarantee that the paths in CWD and
> PATH are fully resolved? (ie: we'll never see a
> "/tmp/.././etc/./passwd" in PATH ?)
You could very well see that. I just tested what you wrote and it shows up
relative path and all.
No problems. The internal path resolver will cope with this (and the
CWD/PATH amalgamation).
Also...please note that if the file name has a space
in it, you get a ascii hex representation of the file name.
No worries - easy to cope with. Thanks for the hint.
Just spaces? How about inverted-commas, embedded newlines (or
Carriage-Returns), or other non-displaying characters that may be
valid on linux (or other) filesystems?
Also, anyone have any thoughts on how to translate "eventid 11" to a
more human-readable "execve" in perl without writing my own C Header
translator?
Regards,
Leigh.
7:41 a.m.
On Tuesday 23 May 2006 21:26, Leigh Purdie wrote:
> Right, but they are not interlaced. You can protect against this
just to
> be safe.
Actually, they are. Hang on, and I'll give you a bigger example
<snip>
Note the 228387 four from the top, and also after the 228390's.
Is this something that should be considered a bug? If so (and it's
likely to be fixed in RHEL4 in the near future), then I'll be able to
greatly simplify the dispatcher code.
It might be, but its not likely to get fixed soon (bigger fish to fry...). I'd
work around it for now.
However, it's probably something that others would appreciate.
The protocol may not need to change explicitly, two separate paths in
auditd may be viable - ie:
dispatcher = /path/to/something-that-receives-a-raw-feed or
asciidispatcher =
/path/to/something-that-receives-nullterminated-ascii-to-stdin
.. if 'asciidispatcher' is active, auditd does a little extra work to
push the line out in a 'nicer' format.
The way I had expected this to work is for people to use the audit parsing
library. I haven't been able to dedicate any time to it for a while, but as
kernel work winds down, I think I'll have more time for it.
> Also...please note that if the file name has a space
> in it, you get a ascii hex representation of the file name.
No worries - easy to cope with. Thanks for the hint.
Just spaces? How about inverted-commas, embedded newlines (or
Carriage-Returns), or other non-displaying characters that may be
valid on linux (or other) filesystems?
This is the function that does it:
http://sosdg.org/~coywolf/lxr/source/kernel/audit.c#L819
if (*p == '"' || *p < 0x21 || *p > 0x7f)
Also, anyone have any thoughts on how to translate "eventid
11" to a
more human-readable "execve" in perl without writing my own C Header
translator?
Are you meaning how to translate the raw time stamp/serial number ? If so, I'd
point to the ausearch source code. I don't program in perl so I may not be
the best source of info. The long term plan is to have an audit event parsing
library so that external apps do not have to have secret audit format
knowledge. The audit parsing library could probably be turned into a perl
module as could libaudit. If you wanted to send a patch for that, I'd be
happy to integrate it. (Right now we only have python bindings.)
-Steve
7:22 p.m.
On 5/24/06, Steve Grubb <sgrubb(a)redhat.com> wrote:
On Tuesday 23 May 2006 21:26, Leigh Purdie wrote:
> Note the 228387 four from the top, and also after the 228390's.
>
> Is this something that should be considered a bug?
> <snip>
It might be, but its not likely to get fixed soon (bigger fish to fry...). I'd
work around it for now.
No worries. Easy done, at the expense of a bit of memory, and by
sacrificing near-real-time for 'within a few seconds of generation
time'.
> Just spaces? How about inverted-commas, embedded newlines (or
> Carriage-Returns), or other non-displaying characters that may be
> valid on linux (or other) filesystems?
This is the function that does it:
http://sosdg.org/~coywolf/lxr/source/kernel/audit.c#L819
if (*p == '"' || *p < 0x21 || *p > 0x7f)
Thanks. :)
> Also, anyone have any thoughts on how to translate "eventid
11" to a
> more human-readable "execve" in perl without writing my own C Header
> translator?
Are you meaning how to translate the raw time stamp/serial number ? If so, I'd
point to the ausearch source code. I don't program in perl so I may not be
the best source of info. The long term plan is to have an audit event parsing
library so that external apps do not have to have secret audit format
knowledge. The audit parsing library could probably be turned into a perl
module as could libaudit. If you wanted to send a patch for that, I'd be
happy to integrate it. (Right now we only have python bindings.)
Nah, timestamps are fine. I was thinking of 'syscall=11'.
I can translate the value of '11' to 'execve' manually by grepping
through /usr/include/asm/unistd.h. However, it's a little harder
programatically - particularly when:
* The system call numbers are subject to change (though, rarely), and
* The numbers might be slightly different for different architectures, and
* The header file might include things like:
#define __NR_mq_notify (__NR_mq_open+4)
.. which makes a quick 'hack' in perl to scan in unistd.h, not worthwhile.
So, to rephrase my question slightly - is there a programmatic way to
turn syscall=5 into syscall=execve that anyone can suggest?
WRT perl, I'm language agnostic. If there's better support for audit
in python, I'll switch the code over. (Perl has, historically, been
more likely to be installed on server systems that are likely to be
running audit though).
Regards,
Leigh.
7:30 a.m.
On Wednesday 24 May 2006 20:22, Leigh Purdie wrote:
So, to rephrase my question slightly - is there a programmatic way
to
turn syscall=5 into syscall=execve that anyone can suggest?
OK, then libaudit has that function, audit_syscall_to_name(). There are
several factors that have to be considered to correctly interpret a syscall
name.
WRT perl, I'm language agnostic. If there's better support
for audit
in python, I'll switch the code over.
Yes, there is better support for python right now. We've also written a
dispatcher used for real-time SE Linux event analysis using python. It grabs
the events as a dictionary and passes them on for analysis. I should be
releasing audit-1.2.3 today which improves python support a little bit more.
-Steve
8:52 a.m.
On Wednesday 24 May 2006 20:22, Leigh Purdie wrote:
If there's better support for audit in python, I'll switch
the code over.
Dan just posted his python code. Its aimed at filtering for AVC's, so you'd
want to make some changes. But its a starting point that gets you past the
terminating NUL issue.
-Steve
8:32 p.m.
All,
Some of you may remember the post a month or two back about an audit
dispatcher that converts the output of auditd, into
one-line-per-event/one-event-per-line.
I've reached an alpha state for this program, and would appreciate
comments/suggestions etc.
Features:
* Takes the output from auditd, and migrates the data into something
that is suitable for applications that expect an event to be self
contained on a single line.
* Tries to extrapolate usernames from userids (using an internal cache
if it can, to cut down on the getpw* calls) so that a centralised
audit collection system doesn't have to keep a UID->username mappings
for all systems.
* Turns eventID numbers into event names (multi-arch compatible).
* Filters audit log data based on administrator-configurable objectives.
* Automatically turns on events as appropriate, based on the
administrators defined objectives.
* Internal/Embedded web server for remote control of the audit
configuration, and (to a certain extent) review of the most recently
received audit events. Fully contained within the code - no external
files accessed to build the web pages (except the config file). The
http server can be password protected, and has a basic IP-based access
control capability.
* Sends audit data to a specified IP address/port combination (snare
format, or syslog format), or local file (though this isn't supported
in the web-gui).
Installation:
$ tar xzf SnareLinux-1.0.tar.gz
$ make
# cp /etc/audit.rules /etc/audit.rules-`date "+%Y%m%d"`
# cp /etc/auditd.conf /etc/auditd.conf-`date "+%Y%m%d"`
# make install
# vi /etc/snare.conf
.. uncomment:
# allow=1
# /etc/init.d/auditd restart
(make uninstall will revert).
Point a browser at port 6161 of the target machine, and
configure/manage appropriately.
If you don't want to fire up a syslog server, or snare micro server to
receive events, feel free to run something like this for testing:
$ socat udp4-listen:6161,reuseaddr,fork OPEN:/tmp/snare.log,creat,append
Alternatively, manually add the following into the [Output] section of
the config file:
file=/tmp/snare.log
Developed on RHEL4U2/Centos4U2. Only very basic testing/qa has been
performed so far. I'd be very interested to know if it works 'out of
the box' on any other distros, or if people have any problems with
installation/use.
BTW: Assume the code is fully GPL - I haven't plastered the
notification through the source yet though. :)
Regards,
Leigh.
10:14 a.m.
On Wed, 24 May 2006 11:26:08 +1000, Leigh Purdie said:
Note the 228387 four from the top, and also after the 228390's.
Is this something that should be considered a bug? If so (and it's
likely to be fixed in RHEL4 in the near future), then I'll be able to
greatly simplify the dispatcher code. :)
This smells a lot like a missing lock on an SMP or preemtable kernel. I'm not
thrilled about having to grab a lock before the first call to audit_log_start
and leaving it locked till the final audit_log_end for the event, since it
could easily be 4 or 5 start/end pairs to get all of it out.
7:26 p.m.
On 5/25/06, Valdis.Kletnieks(a)vt.edu <Valdis.Kletnieks(a)vt.edu> wrote:
On Wed, 24 May 2006 11:26:08 +1000, Leigh Purdie said:
> Note the 228387 four from the top, and also after the 228390's.
>
> Is this something that should be considered a bug? If so (and it's
> likely to be fixed in RHEL4 in the near future), then I'll be able to
> greatly simplify the dispatcher code. :)
This smells a lot like a missing lock on an SMP or preemtable kernel. I'm not
thrilled about having to grab a lock before the first call to audit_log_start
and leaving it locked till the final audit_log_end for the event, since it
could easily be 4 or 5 start/end pairs to get all of it out.
Should be safe to rule out SMP. I'm running on a uni-processor, AMD64
(no hyperthreading) with an i686 kernel:
$ uname -a
Linux rhel4 2.6.9-34_snare.EL #1 Sat Apr 8 23:19:35 EST 2006 i686
athlon i386 GNU/Linux
$ cat /proc/cpuinfo
processor : 0
vendor_id : AuthenticAMD
cpu family : 15
model : 31
model name : AMD Athlon(tm) 64 Processor 3200+
stepping : 0
cpu MHz : 2001.031
cache size : 512 KB
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 1
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca
cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx mmxext lm 3dnowext
3dnow
bogomips : 4012.44
L.
6710
days inactive
6788
days old
linux-audit@lists.linux-audit.osci.io
11 comments
3 participants
participants (3)
-
Leigh Purdie -
Steve Grubb -
Valdis.Kletnieks＠vt.edu