[PATCH 1/8] audit: avoid soft lockup due to audit_log_start() incorrect loop termination
by Richard Guy Briggs
Commit 82919919 caused the wait for auditd timeout condition to loop endlessly
rather than fall through to the error recovery code.
BUG: soft lockup - CPU#0 stuck for 22s! [preload:785]
RIP: 0010:[<ffffffff810fb240>] [<ffffffff810fb240>] audit_log_start+0xf0/0x460
Call Trace:
[<ffffffff810aca40>] ? try_to_wake_up+0x310/0x310
[<ffffffff81100fd1>] audit_log_exit+0x51/0xcb0
[<ffffffff811020b5>] __audit_syscall_exit+0x275/0x2d0
[<ffffffff816ec540>] sysret_audit+0x17/0x21
If the timeout condition goes negative, the loop continues endlessly instead of
breaking out and going to the failure code and allow other processes to run
when auditd is unable to drain the queue fast enough.
This can easily be triggered by readahead-collector, in particular during
installations. The readahead-collector (ab)uses the audit interface and
sometimes gets stuck in a 'stopped' state.
To trigger:
readahead-collector -f &
pkill -SIGSTOP readahead-collector
top
See:
https://lkml.org/lkml/2013/8/28/626
https://lkml.org/lkml/2013/9/2/471
https://lkml.org/lkml/2013/9/3/4
Signed-off-by: Luiz Capitulino <lcapitulino(a)redhat.com>
Signed-off-by: Konstantin Khlebnikov <khlebnikov(a)openvz.org>
Signed-off-by: Dan Duval <dan.duval(a)oracle.com>
Signed-off-by: Chuck Anderson <chuck.anderson(a)oracle.com>
Signed-off-by: Richard Guy Briggs <rgb(a)redhat.com>
---
kernel/audit.c | 5 +++--
1 files changed, 3 insertions(+), 2 deletions(-)
diff --git a/kernel/audit.c b/kernel/audit.c
index 91e53d0..7b0e23a 100644
--- a/kernel/audit.c
+++ b/kernel/audit.c
@@ -1117,9 +1117,10 @@ struct audit_buffer *audit_log_start(struct audit_context *ctx, gfp_t gfp_mask,
sleep_time = timeout_start + audit_backlog_wait_time -
jiffies;
- if ((long)sleep_time > 0)
+ if ((long)sleep_time > 0) {
wait_for_auditd(sleep_time);
- continue;
+ continue;
+ }
}
if (audit_rate_check() && printk_ratelimit())
printk(KERN_WARNING
--
1.7.1
11 years, 3 months
[PATCH 0/8] Audit backlog queue fixes related to soft lockup
by Richard Guy Briggs
This patchset covers a number of issues surrounding the audit backlog queue.
The original trigger was a bug in a patch to fix code that produced negative
sleep times (commit 82919919). This bug caused soft lockups. There were a
number of other issues raised surrounding this bug which have been attempted
to be addressed in this patchset.
The first patch fixes this obvious bug.
The 2nd fixes the lack of reset to the original wait time that was set to
zero under error conditions should the system recover from the conditions that
tickled the first bug.
The 3rd makes use of the remaining timeout to avoid re-checking the loop
conditions on error.
The 4th and 5th are efficiency fixes for waking up waiting processes only when
necessary.
The 6th adds a boot parameter to temporarily override the backlog queue length
in case more buffers are needed before auditd is available.
The 7th and 8th are to add a config option to make the backlog wait time
configurable from the hard-coded default.
Richard Guy Briggs (8):
audit: avoid soft lockup due to audit_log_start() incorrect loop
termination
audit: reset audit backlog wait time after error recovery
audit: make use of remaining sleep time from wait_for_auditd
audit: efficiency fix 1: only wake up if queue shorter than backlog
limit
audit: efficiency fix 2: request exclusive wait since all need same
resource
audit: add boot option to override default backlog limit
audit: clean up AUDIT_GET/SET local variables and future-proof API
audit: add audit_backlog_wait_time configuration option
include/uapi/linux/audit.h | 2 +
kernel/audit.c | 111 +++++++++++++++++++++++++++++++-------------
2 files changed, 80 insertions(+), 33 deletions(-)
11 years, 3 months
user message limits
by LC Bruzenak
I know I can go look at the code, however I figured I'd ask here first
about the limits on the user message in both audit_log_user_message and
ausearch.
With audit_log_user_message the maximum length allowed appears to be
around MAX_AUDIT_MESSAGE_LENGTH-100. I think it may depend on the
executable name length (and other stuff auto-pushed into the string)
which is why I say "around".
Even when I get a successful return value (from audit_log_user_message),
I don't get my string back out in "ausearch" unless it is WAY smaller -
~1K or less I think.
Any ideas/thoughts?
This is the latest (1.7.11-2) audit package.
Thx,
LCB.
--
LC (Lenny) Bruzenak
lenny(a)magitekltd.com
11 years, 3 months
[PATCH] audit: format user messages to size of MAX_AUDIT_MESSAGE_LENGTH
by Richard Guy Briggs
Messages of type AUDIT_USER_TTY were being formatted to 1024 octets,
truncating messages approaching MAX_AUDIT_MESSAGE_LENGTH (8970 octets).
Set the formatting to 8560 characters, given maximum estimates for prefix and
suffix budgets.
See the problem discussion:
https://www.redhat.com/archives/linux-audit/2009-January/msg00030.html
And the new size rationale:
https://www.redhat.com/archives/linux-audit/2013-September/msg00016.html
Test ~8k messages with:
auditctl -m "$(for i in $(seq -w 001 820);do echo -n "${i}0______";done)"
Reported-by: LC Bruzenak <lenny(a)magitekltd.com>
Reported-by: Justin Stephenson <jstephen(a)redhat.com>
Signed-off-by: Richard Guy Briggs <rgb(a)redhat.com>
---
kernel/audit.c | 2 +-
1 files changed, 1 insertions(+), 1 deletions(-)
diff --git a/kernel/audit.c b/kernel/audit.c
index 91e53d0..939cff1 100644
--- a/kernel/audit.c
+++ b/kernel/audit.c
@@ -715,7 +715,7 @@ static int audit_receive_msg(struct sk_buff *skb, struct nlmsghdr *nlh)
}
audit_log_common_recv_msg(&ab, msg_type);
if (msg_type != AUDIT_USER_TTY)
- audit_log_format(ab, " msg='%.1024s'",
+ audit_log_format(ab, " msg='%.8560s'",
(char *)data);
else {
int size;
--
1.7.1
11 years, 3 months
Can't get syslog built-in plugin to post messages to syslog
by Peter Butler
I can't get syslog built-in plugin to post messages to syslog.
My syslog.conf plugin file is:
active = yes
direction = out
path = builtin_syslog
type = builtin
args = LOG_INFO LOG_LOCAL3
format = string
Presumably the file is indeed being parsed by the user-space audit daemon, as after having changed 'active' to 'yes' (and restarted the system), I see the audit daemon has started up the child process audispd, as required.
The audit daemon is indeed logging the audit logs to /var/log/audit/audit.log, but is not also sending them to syslog as configured.
For what it's worth I am using rsyslog rather than syslog but I assume this makes no difference (?).
The rsyslog configuration for the audit logs is straightforward - the line in question being:
local3.* /var/log/audit_trail
I assume my rsyslog is configured properly as I can send a message to LOG_LOCAL3 from the command-line using 'logger' and the message appears in /var/log/audit_trail. But the audit logs never do.
I have the following packages installed:
audit-2.2.1-1
audispd-plugins-2.2.1-1
audit-libs-2.2.1-1
11 years, 3 months
[PATCH] audit: do not reject all AUDIT_INODE filter types
by Eric Paris
commit ab61d38ed8cf670946d12dc46b9198b521c790ea tried to merge the
invalid filter checking into a single function. However AUDIT_INODE
filters were not verified in the new generic checker. Thus such rules
were being denied even though they were perfectly valid.
Ex:
$ auditctl -a exit,always -F arch=b64 -S open -F key=/foo -F inode=6955 -F devmajor=9 -F devminor=1
Error sending add rule data request (Invalid argument)
Signed-off-by: Eric Paris <eparis(a)redhat.com>
---
kernel/auditfilter.c | 1 +
1 file changed, 1 insertion(+)
diff --git a/kernel/auditfilter.c b/kernel/auditfilter.c
index f7aee8b..a87b08b 100644
--- a/kernel/auditfilter.c
+++ b/kernel/auditfilter.c
@@ -343,6 +343,7 @@ static int audit_field_valid(struct audit_entry *entry, struct audit_field *f)
case AUDIT_DEVMINOR:
case AUDIT_EXIT:
case AUDIT_SUCCESS:
+ case AUDIT_INODE:
/* bit ops are only useful on syscall args */
if (f->op == Audit_bitmask || f->op == Audit_bittest)
return -EINVAL;
--
1.8.3.1
11 years, 3 months
Audit of physical users
by Maupertuis Philippe
I have a requirement to trace the activity of the physical users on Redhat 5/6 systems.
I spent the last week sifting through the archive to find that the question was asked time and again.
The basic rule is easy but the hitch is when an administrator restarts a service.
Unfortunately, it seems there is no solution until systemd is used to start daemon instead of service.
The only useful thing I found was in this old post from 2007 http://www.redhat.com/archives/linux-audit/2007-February/msg00071.html to reset the auid.
I would like to know if it can be used with the current version of auditd.
If yes, I will probably give it a try with a fixed dedicated auid to clearly state that the auid was changed.
Do I need to install something besides audit and audit-libs ?
Is there any special need, for compiling this program ?
Thanks
Philippe
________________________________
Ce message et les pi?ces jointes sont confidentiels et r?serv?s ? l'usage exclusif de ses destinataires. Il peut ?galement ?tre prot?g? par le secret professionnel. Si vous recevez ce message par erreur, merci d'en avertir imm?diatement l'exp?diteur et de le d?truire. L'int?grit? du message ne pouvant ?tre assur?e sur Internet, la responsabilit? de Worldline ne pourra ?tre recherch?e quant au contenu de ce message. Bien que les meilleurs efforts soient faits pour maintenir cette transmission exempte de tout virus, l'exp?diteur ne donne aucune garantie ? cet ?gard et sa responsabilit? ne saurait ?tre recherch?e pour tout dommage r?sultant d'un virus transmis.
This e-mail and the documents attached are confidential and intended solely for the addressee; it may also be privileged. If you receive this e-mail in error, please notify the sender immediately and destroy it. As its integrity cannot be secured on the Internet, the Worldline liability cannot be triggered for the message content. Although the sender endeavours to maintain a computer virus-free network, the sender does not warrant that this transmission is virus-free and will not be liable for any damages resulting from any virus transmitted.
11 years, 3 months
Re: Benchmarking the performance impact of auditd
by zhu xiuming
So detailed explanation.
Thanks a lot.
Really appreciate it.
On Fri, Aug 30, 2013 at 11:34 AM, Steve Grubb <sgrubb(a)redhat.com> wrote:
> On Thursday, August 29, 2013 02:14:32 PM zhu xiuming wrote:
> > Thanks a lot.
> > May I ask for more information?
> >
> > I am a little confused by the code. What is the reason behind that?
>
> To measure the impact, you have to repeat the test over and over. So, what
> you
> want to do is create a test that does a syscall and loops on it. You test
> with
> audit disabled and then reboot with audit enabled and run the same test.
> You
> also have to shutdown as many dameons as possible because they alter the OS
> timing.
>
> > It seems that the test.c only access /usr/include directory.But "access"
> > does not trigger any rules in auditd.
>
> Sure it does. If you have a syscall rule on open, for example, the audit
> system does not know if the syscall is of interest. So it walks through the
> list of rules to see if any match the current syscall. If it does, then it
> starts walking through each field of that rule.
>
> So, if you want to do a thorough study, then you want to test both misses
> and
> hits to see how much time is spent in both cases and then compare that when
> the audit system is not enabled and no rules loaded to when the audit
> daemon
> is enabled and rules loaded. You might also compare to when the audit
> daemon
> is enabled but there are no rules loaded.
>
> > Or you mean the test.c only stress the cpu?
>
> Nope. Any syscall affects the audit system. But you want to do micro
> benchmarks
> to see the cumulative effect. Its easier to see a 5% performance hit on a
> microbenchmark that runs for 30 seconds than one that takes milliseconds to
> complete. You can also change the syscall to see if different syscalls have
> more impact than others. For example, the rename syscall probably gets hit
> hard because it has to create auxiliary records when there is a match. You
> might test networking as filesystem watches, too.
>
> -Steve
>
>
> > On Thu, Aug 29, 2013 at 1:24 PM, Steve Grubb <sgrubb(a)redhat.com> wrote:
> > > On Thursday, August 29, 2013 12:59:33 PM zhu xiuming wrote:
> > > > Has someone done some work related to the performance impact of
> enabling
> > > > auditd on syscalls watching?
> > >
> > > Yes, long ago.
> > > http://people.redhat.com/sgrubb/files/lspp-perf.tar.gz
> > >
> > > Short story is watches were undistinguishable from cache hit/misses and
> > > syscall auditing gets more impact as more rules get added and based on
> how
> > > complicated the rule is. CPU's have changed so much since I did the
> > > benchmarking that I won't even hazard a guess as to what the
> performance
> > > hit
> > > is on current hardware with current kernel.
> > >
> > > -Steve
>
11 years, 3 months
