Kernel audit output is inconsistent, hard to parse
by John Dennis
The format of audit messages from the kernel is a mess. The
bottom line is one cannot parse the audit messages without special
case knowledge of each audit message because the data formatting does
not follow any regular rules.
I don't know how it got this way, but it really needs to be fixed.
The primary offense is string formatting, specifically the use or
non-use of the functions audit_log_hex() audit_log_untrustedstring(),
and audit_log_n_untrustedstring(); depending on circumstances.
The net result is a field value might be one of the following cases:
1) a string without quotes (maybe a string, maybe an int, etc.)
2) a string enclosed in quotes (implies a string with no escaped chars)
3) a string which is represented as a sequence of hex values (not
enclosed in quotes, but how do you distinguish this from case 1?)
Given the name=value formatting it is absolutely impossible to
correctly interpret the value component unless you know how
audit_log_format was invoked to generate the name=value pair. This
will be dependent on the kernel version, the field name, and the audit
record type. To be specific, during parsing only case 2 is
unambiguous. You cannot determine between case 1 and case
3. Heuristics based on each character being in the hexadecimal
character set fail for a significant subset of data, thus you don't
know if the value is a string encoded in hexadecimal which needs to be
decoded or a string which happens to be composed of hexadecimal
characters but is not encoded.
Thus we have the situation where to correctly parse the name=value
pair one must know the audit record type, the field name, and the
kernel version. That is just plain CRAZY and UNNECESSARY. Trying hide
this logic in auparse is just a band-aid over the problem compounded
by the fact auparse does not always get it right either. This is in
conjunction with the fact auparse has no way to know the kernel
version of the audit data it is attempting to parse (nor should it
even have tables based on kernel version).
The answer is to make the output parsable without special case
knowledge. It would appear many of these problems were introduced with
the functions audit_log_hex() audit_log_untrustedstring(), and
audit_log_n_untrustedstring() which attempt to correct for a double
quote, white space, or non-printable character in the output
string. However these are not used uniformly nor do they follow any
common approach for string representations in user land (why not?).
All field values without exception need to be enclosed in quotes to
delimit the value. Special characters inside the quotes need to
be escaped, following some standard convention. Please, lets not
invent a new encoding, this problem has already been solved
elsewhere many times before!
Also note the function audit_log_n_untrustedstring() in audit.c has a
bug and ignores the len parameter (it iterates till it finds a NULL
terminator even though it's supposed to stop after n chars).
Suggested Fix:
--------------
Most of these problems can easily be fixed if there is exactly one
central place to format an audit field value. The function
audit_log_vformat() could very easily ensure consistent formatting via
% format specifiers in the format string, e.g.:
audit_log_format("n=%d path=%s", n, path);
Building audit output piecewise would be deprecated, e.g. these types
of sequences would be eliminated.
audit_log_format(ab, " n=%d", n);
audit_log_format(ab, " name=");
audit_log_foo();
and replaced with:
audit_log_format(ab, " n=%d name=%s", foo_to_string(foo));
Whenever audit_log_vformat() encounters a % format specifier it
formats to a string, then it converts the string to an escaped quoted
string, and then inserts the escaped quoted string into the buffer
(e.g. n="123" name="foo bar\n" )
This way the formatting is consistent, easy to apply, and is never
special cased by the caller.
There are no performance penalties of any note, calling a routine to
escape only needs to be done when the format specifier is %s. Currently
this is already done for a subset of output strings, so all we're doing
is removing the responsibility for escaping from the caller and doing
it consistently instead of in a subset of cases.
I don't really care what the encoding is. I only care that it is an
encoding with wide support. Backslash quoting is very popular,
familiar and has many implementations. The MIME quoted-printable
transfer encoding would be another option but might pose some problems
with line endings. I think backslash quoting would be a good choice. I
suspect everyone reading this message already knows exactly how to
interpret a string with backslash escapes.
Auparse is not the answer:
--------------------------
Auparse is not the answer to irregular kernel audit message
formatting. First of all it forces auparse to have special case logic
which is not 100% robust and is tied to the kernel source code
version.
Second, in it's current implementation auparse confuses transfer
decoding and substitution, two entirely different concepts needing to
be applied in entirely different circumstances, but which have been
conflated.
auparse_get_field_str() returns the field value in it's encoded form,
this is almost never of value to the caller. The caller wants the
field value to be unencoded so it can operate on it. If you want the
field value to be unencoded you have to call
auparse_interpret_field(). But auparse_interpret_field() performs two
distinctly different operations, it both decodes AND performs
contextual substitution. Contextual substitution only has meaning when
applied on the same host and at approximately the same time as when
the audit record was generated. Contextual substitution is mainly of
value for human readable output, it is difficult to utilize with
automated machine processing. At the moment it is not possible to get
a decoded value from auparse without it also performing undesired
substitution.
While we're at it:
------------------
If we do fix the format of audit messages we might as well fix some
other inconsistencies at the same time.
1) The initial part of AVC messages do not follow the standard
name=value formatting used everywhere else in audit.
a) It includes the string "avc:" which is redundant with the audit
record type (e.g. type=AVC), the string "avc:" should be removed,
it serves no purpose and only makes parsing much harder because of
the inconsistency.
b) denied|granted are bare words without a field name, it should be
seresult="denied", once again to avoid special case parsing.
c) The list of operations are enclosed in curly braces {} without a
field name, this should be seperms=xxx, where xxx is a list. The
use of curly braces to encode a list in audit data is unique. We
should define how any audit message should encode a list of values
and use that consistently for all audit data. While one could
define a syntax such as "[value1, value2]" or some such, it might
be informative to look at how other transfer mechanisms such as
structured markup and ldap handle this case. They both utilize the
concept of multi-valued attributes. Thus there is no list
structure, but an attribute is allowed to repeat itself and in the
process implicitly creates a list of values for the attribute. Thus
{read write} might be represented as seperms="read"
seperms="write". This regularity makes parsing much easier, it
avoids special case syntax.
2) (Note, this is not a kernel issue) The host data is currently
prepended to the audit record with the format host=xxx. Is this an
encoded string or not? It should be encoded and it should be
encoded in exactly the same format as the name/value pairs in the
audit records. The same holds true for the record type, it should
follow the same syntax as every other name/value pair.
3) The string "audit(ssssss.mmmm:iiii):" is a critical delimiter, it
separates record properties (e.g. host, type, timestamp) from
record data, which must be a sequence of name="value" pairs. But
the time stamp should really follow the name/value pair encoding
used elsewhere.
Desired syntax:
---------------
Records consist of a sequence of name="value" pairs.
Ordering of name/value pairs is significant for multi-valued
attributes (i.e. where name appears more than once), insignificant
otherwise.
The value MUST be enclosed in double quotes with interior characters
properly escaped.
White space between name, '=', and "value" is insignificant and
ignored.
The audit record is partitioned into two parts
a) record properties (i.e. host, record type, timestamp)
b) record data
The partition of properties and data occurs at a colon delimiter,
i.e. properties : data
The current formatting of the record timestamp
(e.g. audit(ssss.mmm:iii) is inconsistent with
all other name/value pairs. It should be "seconds="sss"
milliseconds="mmm" serial="iii", this allows parsing to be regular and
consistent.
Thus an audit record with consistent syntax would look like this,
where brackets [] indicate optional components:
[host=""] type="" seconds="" milliseconds="" serial="" : name="" [name=""]
What has to change and what's optional:
---------------------------------------
The formatting of name/value pairs in the kernel must be fixed, it is
simply impossible to correctly parse in it's current state.
The rest of the suggested changes are syntactic sugar which would make
parsing easier because of regular syntax, but they are not
critical. We could retain the existing formats if backwards
compatibility is felt to trump syntactic cleanliness and ease in
parsing. It's a judgment call over when and how to introduce change
and the anticipated impact.
--
John Dennis <jdennis(a)redhat.com>
17 years
audit 1.6.7 released
by Steve Grubb
Hi,
I've just released a new version of the audit daemon. It can be downloaded
from http://people.redhat.com/sgrubb/audit It will also be in rawhide
soon. The Changelog is:
- In ausearch/report, prefer -if to stdin
- In ausearch/report, add new command line option --input-logs (#428860)
- Updated audisp-prelude based on feedback from prelude-devel
- Added prelude alert for promiscuous socket being opened
- Added prelude alert for SE Linux policy enforcement changes
- Added prelude alerts for Forbidden Login Locations and Time
- Applied patch to auparse fixing error handling of searching by
interpreted value (Miloslav Trmac)
Based on feedback from the prelude-devel list, I changed the analyzer name of
the prelude plugin to auditd. This means that you will have to re-register
the audisp-prelude plugin and use the new name. I have put instructions in
the prelude HOWTO that you can find here:
http://people.redhat.com/sgrubb/audit/prelude.txt
This release completes the initial development for the audisp-prelude plugin.
It adds alerts for logins from forbidden locations and times, promiscuous
socket open/close, and changes to SE Linux policy enforcement. This release
also fills in more fields to better meet IDMEF standards. The SE Linix AVC
alert now has the actual AVC in it. I will work more on this plugin later
this spring, I need to spend some time on remote logging.
Please let me know if you run across any problems with this release.
-Steve
17 years
audit aggregation
by LC Bruzenak
Just a thought from someone who is following this list closely b/c I'm
tasked with setting up a multi-host system auditing capability - one
thing Steve G. mentioned was:
>
> > it both decodes AND performs contextual substitution. Contextual
> > substitution only has meaning when applied on the same host and at
> > approximately the same time as when the audit record was generated.
>
> Correct. You are talking about something the library does not handle
> today. The reason is because there is no designed method to aggregate
> logs. So, when that work is done, auparse will be fixed up to handle
> the situation.
I have been thinking about how to solve this also; I bet I'm not alone.
So if/when changes are made I'd be grateful if it is included. I'll be
willing to participate as required.
LCB.
ps: Steve the prelude plugins are excellent!
--
LC (Lenny) Bruzenak
lenny(a)magitekltd.com
17 years
gui based search and report tools
by Abhishek Gupta
I have few questions :():
1)What exactly is expected in gui based search tool and on what basis
search will take place ? will it be same as ausearch utility?
2)What exactly is expected in gui based reporting tool ? will it be same
as aureport utility?
.
17 years
What does each audit record field mean?
by Marius.bao
Hi,
I'm a newbie, I'm sorry for my question if anyone has already asked.
I use auditctl -a exit,always -S open -F success=0 to audit all
successful open syscalls
But in the audit.log file I found the following audit records:
type=SYSCALL msg=audit(1201421673.445:1508): arch=40000003
syscall=5 success=no exit=-2 a0=bfec1e40 a1=0 a2=b7ee6548 a3=bfec1e40
items=1 ppid=9571 pid=96 95 auid=0 uid=0 gid=0 euid=0 suid=0
fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts1 comm="vim" exe="/usr/bin/vim"
key=(null)
The "success" fields of the record is no, what does it mean? Does
it represent the syscall is failed?
And what does "exit" field mean? Does it represent the syscall's exit code?
I'm also confused with the meaning of the fields of "a0" "a1" "a2" and "a3".
17 years
[PATCH] ratelimit printk messages from the audit system
by Eric Paris
Some printk messages from the audit system can become excessive. This
patch ratelimits those messages. It was found that messages, such as
the audit backlog lost printk message could flood the logs to the point
that a machine could take an nmi watchdog hit or otherwise become
unresponsive.
Signed-off-by: Eric Paris <eparis(a)redhat.com>
---
kernel/audit.c | 28 ++++++++++++++++++----------
1 files changed, 18 insertions(+), 10 deletions(-)
diff --git a/kernel/audit.c b/kernel/audit.c
index f93c271..a3d828b 100644
--- a/kernel/audit.c
+++ b/kernel/audit.c
@@ -163,7 +163,8 @@ void audit_panic(const char *message)
case AUDIT_FAIL_SILENT:
break;
case AUDIT_FAIL_PRINTK:
- printk(KERN_ERR "audit: %s\n", message);
+ if (printk_ratelimit())
+ printk(KERN_ERR "audit: %s\n", message);
break;
case AUDIT_FAIL_PANIC:
panic("audit: %s\n", message);
@@ -231,11 +232,13 @@ void audit_log_lost(const char *message)
}
if (print) {
- printk(KERN_WARNING
- "audit: audit_lost=%d audit_rate_limit=%d audit_backlog_limit=%d\n",
- atomic_read(&audit_lost),
- audit_rate_limit,
- audit_backlog_limit);
+ if (printk_ratelimit())
+ printk(KERN_WARNING
+ "audit: audit_lost=%d audit_rate_limit=%d "
+ "audit_backlog_limit=%d\n",
+ atomic_read(&audit_lost),
+ audit_rate_limit,
+ audit_backlog_limit);
audit_panic(message);
}
}
@@ -405,7 +408,11 @@ static int kauditd_thread(void *dummy)
audit_pid = 0;
}
} else {
- printk(KERN_NOTICE "%s\n", skb->data + NLMSG_SPACE(0));
+ if (printk_ratelimit())
+ printk(KERN_NOTICE "%s\n", skb->data +
+ NLMSG_SPACE(0));
+ else
+ audit_log_lost("printk limit exceeded\n");
kfree_skb(skb);
}
} else {
@@ -1164,7 +1171,7 @@ struct audit_buffer *audit_log_start(struct audit_context *ctx, gfp_t gfp_mask,
remove_wait_queue(&audit_backlog_wait, &wait);
continue;
}
- if (audit_rate_check())
+ if (audit_rate_check() && printk_ratelimit())
printk(KERN_WARNING
"audit: audit_backlog=%d > "
"audit_backlog_limit=%d\n",
@@ -1433,9 +1440,10 @@ void audit_log_end(struct audit_buffer *ab)
skb_queue_tail(&audit_skb_queue, ab->skb);
ab->skb = NULL;
wake_up_interruptible(&kauditd_wait);
- } else {
+ } else if (printk_ratelimit())
printk(KERN_NOTICE "%s\n", ab->skb->data + NLMSG_SPACE(0));
- }
+ else
+ audit_log_lost("printk limit exceeded\n");
}
audit_buffer_free(ab);
}
17 years
Matt Weale/UK/CSC is out of the office.
by Matt Weale
I will be out of the office starting 21/01/2008 and will not return until
28/01/2008.
In my absence please contact either Les Klein or David Stapeleton.
17 years
prelude HOWTO
by Steve Grubb
Hi,
I have a prelude HOWTO started here:
http://people.redhat.com/sgrubb/audit/prelude.txt
I'm just missing the final apache configuration, which I should get to in the
next day or so. This should be good enough for anyone to find out what
prelude can do since you can run prewikka by hand to test with.
If you are testing with rawhide right now, you have to put selinux in
permissive mode since this is new behaviors of the audit daemon,
-Steve
17 years
