4:11 p.m.
I had a problem with the system hanging while running a test case that
exercises the boundary limits on path name and file name while inserting
watches. This seemed to only occur on SMP machines. I also tried the
test case on audit (0.9.4 -> 0.9.9) and it seemed to break on all those
versions regardless of the kernel version running.
The system hung when attempting to stop audit after trying to insert a
watch on a long filename (> NAME_MAX).
With audit0.9.10 (and the latest kernel.65) the problem seemed to just
go away.. I am not sure what changed in the code to fix it ... but I
thought it would be good to report it in case someone encounters a
similar problem again ... also it would be nice if we know what happened
to fix it .. or break it in the first place ...
Thanks for Klaus ... helping narrow the problem down.
To reproduce :
# /sbin/auditctl -w /tmp/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
-k good-key
> The base name of the path is too big
#/etc/init.d/auditd restart
> Stopping auditd: [ OK
]
> Starting auditd: [ OK ]
# /sbin/auditctl -w /tmp/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx -k
good-key
> The base name of the path is too big
# /etc/init.d/auditd restart
>Stopping auditd: [ OK
]
------ IT hangs here -------
-loulwa
4:23 p.m.
On Tuesday 21 June 2005 17:11, Loulwa Salem wrote:
# /etc/init.d/auditd restart
>>Stopping auditd: [ OK ]
------ IT hangs here -------
My guess is that auditctl was deleting the rules/watches. Did the whole
machine freeze up or just the init script? There was a spinlock problem fixed
yesterday - not sure if you fell into that hole. It would hang the whole
machine.
-Steve
4:30 p.m.
Steve Grubb wrote:
My guess is that auditctl was deleting the rules/watches. Did the
whole
machine freeze up or just the init script? There was a spinlock problem fixed
yesterday - not sure if you fell into that hole. It would hang the whole
machine.
Maybe that's what it was ... because the system entirely hangs ... even
going to the console didn't help .. it had to be force rebooted every time
- loulwa
11:20 a.m.
David,
I've upgrade to the 0.9.11 audit tools and have the same problem.
Its very repeatable. I have SysRq-T output if you're still interested,
although I'm not sure how useful it is. There's no stack information
for the auditctl process or the audit_list_rules thread but I will
send it along if you'd like.
-- ljk
David Woodhouse wrote:
On Tue, 2005-06-21 at 16:30 -0500, Loulwa Salem wrote:
>Maybe that's what it was ... because the system entirely hangs ...
>even going to the console didn't help .. it had to be force rebooted
>every time
Please show SysRq-T output if SysRq works.
11:22 a.m.
On Wed, 2005-06-22 at 12:20 -0400, Linda Knippers wrote:
I've upgrade to the 0.9.11 audit tools and have the same
problem.
Its very repeatable. I have SysRq-T output if you're still
interested, although I'm not sure how useful it is. There's no stack
information for the auditctl process or the audit_list_rules thread
but I will send it along if you'd like.
Please. If you could also strace the auditctl process (from the
beginning), that would be useful.
--
dwmw2
12:39 p.m.
On Wednesday 22 June 2005 12:20, Linda Knippers wrote:
I've upgrade to the 0.9.11 audit tools and have the same
problem.
Its very repeatable.
I see one obvious thing. The kernel code has changed the way that it handles
strings from user space regarding watches. User space does not NUL terminate
strings, it passes the length. The kernel needs to ensure that NUL terminated
strings are used for operations like strcmp.
We should go back to the way that it used to be done. User space hasn't
changed how it passes strings since we agreed on the interface. The kernel
changed.
-Steve
12:46 p.m.
On Wednesday 22 June 2005 13:39, Steve Grubb wrote:
We should go back to the way that it used to be done. User space
hasn't
changed how it passes strings since we agreed on the interface. The kernel
changed.
Nevermind - I'm wrong. I missed the call to audit_to_watch() in
audit_receive_watch(). Sorry for the noise...
-Steve
5:13 p.m.
I just tried the same thing on my 2 CPU ia64 box (.65 kernel and 0.9.10
audit tools) and on my system, the system doesn't hang but it doesn't
work either.
Where your system hung, just my window hangs. I can see that the
auditd script is trying to do an auditctl -D and the auditctl process is
running constantly and can't be killed.
The first time I did this I forgot that I had some rules in the rules
file that would cause all syscalls with my uid to be audited. When
that happened, not only was auditctl running constantly but all my
processes seemed to go wild. Touching any window caused shell prompts
to start scrolling off the screen, etc. I could ssh in and that seemed
to work ok but the messages file (because there was no auditd) was
filling up. There were lots of select, read, sigprocmask, ... syscalls
being audited, like signals gone wild?.
Anyone else tried this?
-- ljk
PS I'd experiment some more tonight but I have to leave soon (my SO's
birthday). I can try more tomorrow.
Loulwa Salem wrote:
I had a problem with the system hanging while running a test case
that
exercises the boundary limits on path name and file name while inserting
watches. This seemed to only occur on SMP machines. I also tried the
test case on audit (0.9.4 -> 0.9.9) and it seemed to break on all those
versions regardless of the kernel version running.
The system hung when attempting to stop audit after trying to insert a
watch on a long filename (> NAME_MAX).
With audit0.9.10 (and the latest kernel.65) the problem seemed to just
go away.. I am not sure what changed in the code to fix it ... but I
thought it would be good to report it in case someone encounters a
similar problem again ... also it would be nice if we know what happened
to fix it .. or break it in the first place ...
Thanks for Klaus ... helping narrow the problem down.
To reproduce :
# /sbin/auditctl -w /tmp/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
-k good-key
>> The base name of the path is too big
#/etc/init.d/auditd restart
>> Stopping auditd: [ OK ]
>> Starting auditd: [ OK ]
# /sbin/auditctl -w /tmp/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx -k
good-key
>> The base name of the path is too big
# /etc/init.d/auditd restart
>>Stopping auditd: [ OK ]
------ IT hangs here -------
-loulwa
--
Linux-audit mailing list
Linux-audit(a)redhat.com
http://www.redhat.com/mailman/listinfo/linux-audit
5:27 p.m.
On Tuesday 21 June 2005 18:13, Linda Knippers wrote:
I just tried the same thing on my 2 CPU ia64 box (.65 kernel and
0.9.10
audit tools) and on my system, the system doesn't hang but it doesn't
work either.
You might try 0.9.11 and see if that solves your problem. There are some
variances in kernels that cause netlink to behave strangely - which is why
I've had so many iterations trying to solve the user's can't login problem. I
think 0.9.11 finally solves that problem.
The change to kernel threads doing the listing does affect auditctl because of
the ack being sent instead of the list.
-Steve
6:58 p.m.
Steve Grubb wrote:
You might try 0.9.11 and see if that solves your problem. There are
some
variances in kernels that cause netlink to behave strangely - which is why
I've had so many iterations trying to solve the user's can't login problem. I
think 0.9.11 finally solves that problem.
I am on an SMP x86_64 platform (kernel .65)
I tried the 0.9.11 audit ... and it hung (waited on it for 7.5 minutes
but I was able to do ctrl-z to stop the test) ... however I believe the
run left the system in an unstable state considering it wouldn't respond
to a reboot command, and had to be force rebooted anyway. Before I
rebooted .. I got this ps -ef | grep audit output:
root 2311 11 0 18:38 ? 00:00:00 [kauditd]
root 3000 2946 0 18:40 pts/1 00:00:00 /bin/bash
/etc/rc.d/init.d/auditd stop
root 3008 3000 99 18:40 pts/1 00:01:17 /sbin/auditctl -D
root 3009 13 19 18:40 ? 00:00:14 [audit_list_rule]
root 3017 2899 0 18:41 pts/0 00:00:00 grep audit
I went back to the 0.9.10 version and it worked but slowly ... I did end
up with a lot of hanging processes regarding [audit_list_watch] and
[audit_list_rules] ... When I tried to do kill -9 on any of those
processes ... it didn't have any effect.
Sample (ps -ef | grep audit). Notice auditd isn't even running:
root 2311 11 0 18:38 ? 00:00:00 [kauditd]
root 3008 1 99 18:40 pts/1 00:07:00 /sbin/auditctl -D
root 3009 13 25 18:40 ? 00:01:49 [audit_list_rule]
root 3048 11 0 18:43 ? 00:00:00 [audit_list_rule]
root 3049 13 0 18:43 ? 00:00:00 [audit_list_watc]
root 3050 13 0 18:43 ? 00:00:00 [audit_list_rule]
root 3051 11 0 18:43 ? 00:00:00 [audit_list_watc]
.....
root 3820 13 0 18:46 ? 00:00:00 [audit_list_rule]
root 3821 11 0 18:46 ? 00:00:00 [audit_list_watc]
root 3826 2899 0 18:47 pts/0 00:00:00 grep audit
- Loulwa
4:58 p.m.
Loulwa Salem wrote: [Tue Jun 21 2005, 07:58:28PM EDT]
I am on an SMP x86_64 platform (kernel .65)
I tried the 0.9.11 audit ... and it hung (waited on it for 7.5 minutes
but I was able to do ctrl-z to stop the test) ... however I believe the
run left the system in an unstable state considering it wouldn't respond
to a reboot command, and had to be force rebooted anyway.
I have a little more info on the long pathname hang that Loulwa found.
I was able to reproduce it several times on a 2 CPU x86_64 box this
afternoon. I'm running with the 0.70 kernel and the 0.9.13 tools.
It doesn't seem to matter whether or not the audit subsystem is
enabled, or whether auditd is running.
I get the same ps output:
root 2536 6 0 17:14 ? 00:00:00 [kauditd]
root 2719 2580 0 17:15 pts/0 00:00:00 /bin/bash /etc/init.d/auditd
+restart
root 2727 2719 99 17:15 pts/0 00:00:43 /sbin/auditctl -D
root 2728 7 22 17:15 ? 00:00:10 [audit_list_rule]
On shutdown, the system gets stuck on 'Stopping pcmcia'. At one point
it provided a little more info:
Stopping pcmcia: warning: many lost ticks.
Your time source seems to be instable or some driver is hogging interupts
rip __do_softirq+0x4d/0xd0
I'm attaching sysrq -t output and straces of auditctl and auditd.
There wasn't any info available for sysrq -p. Note that the sysrq -t
output is actually from the 0.68 kernel. The output I got using the
0.70 kernel wasn't complete for auditctl.
Hope this helps,
Amy
5:20 p.m.
On Fri, 2005-06-24 at 17:58 -0400, Amy Griffis wrote:
I have a little more info on the long pathname hang that Loulwa
found.
I was able to reproduce it several times on a 2 CPU x86_64 box this
afternoon. I'm running with the 0.70 kernel and the 0.9.13 tools.
Thanks. The 'long pathname' part of it really ought to be a red herring
-- if you look at the strace you'll see the auditctl aborts without ever
trying to send the watch to the kernel. It makes the decision about
length on its own. Is this _really_ necessary to reproduce the problem?
This looks like a spinlock deadlock between auditctl and the
audit_list_rules thread. But I really can't see anywhere that those two
would be contending for any locks. This isn't easy to debug without
knowing _where_ each CPU is, I'm concerned that SysRq-P isn't working --
does it work _before_ you get the system into this state?
Do you have other options, like crash dump? Reproducing it on a PPC64
LPAR might let us poke at it more useful from the hypervisor.
We could try just adding printks throughout the audit_list_rules()
function so that we can attempt to see how far it got, I suppose...
Can you confirm my understanding of the steps required to reproduce
this... this ought to suffice?
while true ; do \
/sbin/auditctl -w
/tmp/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
-k good-key ; \
/sbin/service auditd restart ; \
done
It's running on my dual i686 test box now...
--
dwmw2
5:26 p.m.
Thanks. The 'long pathname' part of it really ought to be a
red herring
-- if you look at the strace you'll see the auditctl aborts without ever
trying to send the watch to the kernel. It makes the decision about
length on its own. Is this _really_ necessary to reproduce the problem?
I thought the same thing but couldn't reproduce it without that in
there. I think Amy has seen weird behavior under other circumstances
though.
I'm concerned that SysRq-P isn't working --
does it work _before_ you get the system into this state?
Not on my ia64 box.
Can you confirm my understanding of the steps required to reproduce
this... this ought to suffice?
Hangs on the first iteration on my box.
-- ljk
5:44 p.m.
On Fri, 2005-06-24 at 18:26 -0400, Linda Knippers wrote:
> I'm concerned that SysRq-P isn't working --
> does it work _before_ you get the system into this state?
Not on my ia64 box.
Hmmm. Does SysRq-W work if you apply the patch at
http://david.woodhou.se/sysrq-show-all-cpus.patch ?
--
dwmw2
4:56 a.m.
On Fri, 2005-06-24 at 18:26 -0400, Linda Knippers wrote:
> Can you confirm my understanding of the steps required to
reproduce
> this... this ought to suffice?
Hangs on the first iteration on my box.
Hmm. Still running on mine. Has this been seen on anything but ia64 and
x86_64? Perhaps it's a 64-bit thing, although I really don't see how
that could be the case. I'll try it on ppc64.
--
dwmw2
7119
days inactive
7123
days old
linux-audit@lists.linux-audit.osci.io
15 comments
5 participants
participants (5)
-
Amy Griffis -
David Woodhouse -
Linda Knippers -
Loulwa Salem -
Steve Grubb