5:14 p.m.
Hello,
I am testing the .72 kernel and seem to have run into a new problem. I wrote a
test script that inserted 150 syscall rules & deleted them over & over, ran
another script that listed them over and over, the enabler script, and the
trusty old fs-torture script.
Anything related to netlink hung. Everything else seemed to be fine. I was
able to open new terminals, but su would fail. auditctl -l would hang. There
was no disk activity, so auditd must be hung.
There was no messages in syslog indicating a problem. Eventually everything
stopped responding (including caps lock key) and I had to hit the reset
button.
-Steve
3:07 p.m.
On Wednesday 29 June 2005 03:03, David Woodhouse wrote:
Show SysRq-T, SysRq-P output.
I don't have serial cable or spare computer with serial port.
However, I did track the problem down somewhat. It turns out that:
auditctl -a entry,always -S socketcall
is what triggers the hang. I remember auditing socket calls when working on
ausearch interpretation code. So I booted into old kernels.
.61 runs, but starts heavy disk activity.
.62 runs, but reports backlog overflow to console
.63 hangs.
The changelog for .63 says:
- Wait for backlog to drain where possible
So, my guess is that there is something in the draining of the backlog that
causes this problem. The audit logs from .62 shows that recvfrom() for auditd
is being audited over and over.
I also wonder if this is related to the problem Debbie & Denise were
reporting? It seems similar - system hung, no visible indicators that it hit
the backlog draining.
-Steve
3:17 p.m.
On Wed, 2005-06-29 at 16:07 -0400, Steve Grubb wrote:
auditctl -a entry,always -S socketcall
is what triggers the hang. I remember auditing socket calls when
working on ausearch interpretation code. So I booted into old kernels.
.61 runs, but starts heavy disk activity.
.62 runs, but reports backlog overflow to console
.63 hangs.
Hm. I cannot reproduce this on my i686 SMP test box. I'll try harder in
the morning. Can you reproduce it without auditd running? Can you show
the audit messages which were being logged at the time?
--
dwmw2
12:08 p.m.
On Wed, 2005-06-29 at 16:07 -0400, Steve Grubb wrote:
So, my guess is that there is something in the draining of the
backlog that
causes this problem. The audit logs from .62 shows that recvfrom() for auditd
is being audited over and over.
I also wonder if this is related to the problem Debbie & Denise were
reporting? It seems similar - system hung, no visible indicators that it hit
the backlog draining.
I've cleaned up the backlog waiting, and made sure that it will _stop_
waiting once it's called audit_panic() once. That'll avoid the illusion
of a hung system if auditd isn't making progress. We should probably
allow auditctl to set both the audit_backlog_wait_time and
audit_backlog_wait_overflow variables, but you weren't about on IRC to
discuss the necessary changes to the AUDIT_SET layout, so this'll do for
now.
--- linux-2.6.9/kernel/audit.c~ 2005-06-24 15:13:54.000000000 +0100
+++ linux-2.6.9/kernel/audit.c 2005-06-30 17:45:22.000000000 +0100
@@ -79,6 +79,8 @@ static int audit_rate_limit;
/* Number of outstanding audit_buffers allowed. */
static int audit_backlog_limit = 64;
+static int audit_backlog_wait_time = 60 * HZ;
+static int audit_backlog_wait_overflow = 0;
/* The identity of the user shutting down the audit system. */
uid_t audit_sig_uid = -1;
@@ -723,6 +725,7 @@ struct audit_buffer *audit_log_start(str
struct timespec t;
unsigned int serial;
int reserve;
+ unsigned long timeout_start = jiffies;
if (!audit_initialized)
return NULL;
@@ -735,8 +738,9 @@ struct audit_buffer *audit_log_start(str
while (audit_backlog_limit
&& skb_queue_len(&audit_skb_queue) > audit_backlog_limit +
reserve) {
- if (gfp_mask & __GFP_WAIT) {
- int ret = 1;
+ if (gfp_mask & __GFP_WAIT && audit_backlog_wait_time
+ && time_before(jiffies, timeout_start + audit_backlog_wait_time)) {
+
/* Wait for auditd to drain the queue a little */
DECLARE_WAITQUEUE(wait, current);
set_current_state(TASK_INTERRUPTIBLE);
@@ -744,12 +748,11 @@ struct audit_buffer *audit_log_start(str
if (audit_backlog_limit &&
skb_queue_len(&audit_skb_queue) > audit_backlog_limit)
- ret = schedule_timeout(HZ * 60);
+ schedule_timeout(timeout_start + audit_backlog_wait_time - jiffies);
__set_current_state(TASK_RUNNING);
remove_wait_queue(&audit_backlog_wait, &wait);
- if (ret)
- continue;
+ continue;
}
if (audit_rate_check())
printk(KERN_WARNING
@@ -758,6 +761,8 @@ struct audit_buffer *audit_log_start(str
skb_queue_len(&audit_skb_queue),
audit_backlog_limit);
audit_log_lost("backlog limit exceeded");
+ audit_backlog_wait_time = audit_backlog_wait_overflow;
+ wake_up(&audit_backlog_wait);
return NULL;
}
--
dwmw2
7114
days inactive
7116
days old
linux-audit@lists.linux-audit.osci.io
4 comments
2 participants
participants (2)
-
David Woodhouse -
Steve Grubb