10:50 a.m.
On Thu, 2014-12-18 at 11:45 -0500, Valdis.Kletnieks(a)vt.edu wrote:
On Tue, 16 Dec 2014 20:09:54 -0500, Valdis Kletnieks said:
> Spotted these two while booting single-user on 20141216. 20141208
> doesn't throw these, so it's something in the last week or so..
Gaah! Turns out that 20141208 *is* susceptible - it had been booting
just fine for several days, but it went around the bend, apparently due
to a userspace or initrd change.
$5 says you updated systemd?
Richard?
egrep 'BUG|Linux vers' from my syslog:
Dec 9 12:19:53 turing-police kernel: [ 0.000000] Linux version 3.18.0-next-20141208
(source(a)turing-police.cc.vt.edu) (gcc version 4.9.2 20141101 (Red Hat 4.9.2-1) (GCC) ) #27
SMP PREEMPT Mon Dec 8 22:20:07 EST 2014
Dec 9 21:19:53 turing-police kernel: [ 0.000000] Linux version 3.18.0-next-20141208
(source(a)turing-police.cc.vt.edu) (gcc version 4.9.2 20141101 (Red Hat 4.9.2-1) (GCC) ) #27
SMP PREEMPT Mon Dec 8 22:20:07 EST 2014
Dec 10 12:39:45 turing-police kernel: [ 0.000000] Linux version 3.18.0-next-20141208
(source(a)turing-police.cc.vt.edu) (gcc version 4.9.2 20141101 (Red Hat 4.9.2-1) (GCC) ) #27
SMP PREEMPT Mon Dec 8 22:20:07 EST 2014
Dec 10 20:56:28 turing-police kernel: [ 0.000000] Linux version 3.18.0-next-20141208
(source(a)turing-police.cc.vt.edu) (gcc version 4.9.2 20141101 (Red Hat 4.9.2-1) (GCC) ) #27
SMP PREEMPT Mon Dec 8 22:20:07 EST 2014
Dec 11 10:46:49 turing-police kernel: [ 0.000000] Linux version 3.18.0-next-20141208
(source(a)turing-police.cc.vt.edu) (gcc version 4.9.2 20141101 (Red Hat 4.9.2-1) (GCC) ) #27
SMP PREEMPT Mon Dec 8 22:20:07 EST 2014
Dec 11 23:53:10 turing-police kernel: [ 0.000000] Linux version 3.18.0-next-20141208
(source(a)turing-police.cc.vt.edu) (gcc version 4.9.2 20141101 (Red Hat 4.9.2-1) (GCC) ) #27
SMP PREEMPT Mon Dec 8 22:20:07 EST 2014
Dec 12 11:13:19 turing-police kernel: [ 0.000000] Linux version 3.18.0-next-20141208
(source(a)turing-police.cc.vt.edu) (gcc version 4.9.2 20141101 (Red Hat 4.9.2-1) (GCC) ) #27
SMP PREEMPT Mon Dec 8 22:20:07 EST 2014
Dec 12 19:26:24 turing-police kernel: [ 0.000000] Linux version 3.18.0-next-20141208
(source(a)turing-police.cc.vt.edu) (gcc version 4.9.2 20141101 (Red Hat 4.9.2-1) (GCC) ) #27
SMP PREEMPT Mon Dec 8 22:20:07 EST 2014
Dec 12 19:33:32 turing-police kernel: [ 0.000000] Linux version 3.18.0-next-20141208
(source(a)turing-police.cc.vt.edu) (gcc version 4.9.2 20141101 (Red Hat 4.9.2-1) (GCC) ) #27
SMP PREEMPT Mon Dec 8 22:20:07 EST 2014
Dec 12 19:42:30 turing-police kernel: [ 0.000000] Linux version 3.18.0-next-20141208
(source(a)turing-police.cc.vt.edu) (gcc version 4.9.2 20141101 (Red Hat 4.9.2-1) (GCC) ) #27
SMP PREEMPT Mon Dec 8 22:20:07 EST 2014
Dec 12 20:00:39 turing-police kernel: [ 1109.635328] BUG: sleeping function called from
invalid context at mm/slab.c:2849
Dec 12 20:00:43 turing-police kernel: [ 1113.680912] BUG: sleeping function called from
invalid context at mm/slab.c:2849
Dec 12 20:33:15 turing-police kernel: [ 3062.345461] BUG: sleeping function called from
invalid context at mm/slab.c:2849
Dec 12 20:37:48 turing-police kernel: [ 3335.788891] BUG: sleeping function called from
invalid context at mm/slab.c:2849
Dec 12 20:41:57 turing-police kernel: [ 3584.265255] BUG: sleeping function called from
invalid context at mm/slab.c:2849
Dec 12 20:42:47 turing-police kernel: [ 3633.863552] BUG: sleeping function called from
invalid context at mm/slab.c:2849
Dec 12 20:51:33 turing-police kernel: [ 0.000000] Linux version 3.18.0-next-20141208
(source(a)turing-police.cc.vt.edu) (gcc version 4.9.2 20141101 (Red Hat 4.9.2-1) (GCC) ) #27
SMP PREEMPT Mon Dec 8 22:20:07 EST 2014
Dec 12 21:51:04 turing-police kernel: [ 3587.132867] BUG: sleeping function called from
invalid context at mm/slab.c:2849
Dec 12 22:20:01 turing-police kernel: [ 5322.313024] BUG: sleeping function called from
invalid context at mm/slab.c:2849
Dec 12 23:06:00 turing-police kernel: [ 8077.463289] BUG: sleeping function called from
invalid context at mm/slab.c:2849
Dec 13 00:00:05 turing-police kernel: [11318.405826] BUG: sleeping function called from
invalid context at mm/slab.c:2849
I need to figure out what changed around 7:30PM on the 12th.
11:46 a.m.
New subject: linux-next 20141216 BUG: sleeping function called from invalid context at mm/slab.c:2849
On 14/12/18, Eric Paris wrote:
On Thu, 2014-12-18 at 11:45 -0500, Valdis.Kletnieks(a)vt.edu wrote:
> On Tue, 16 Dec 2014 20:09:54 -0500, Valdis Kletnieks said:
> > Spotted these two while booting single-user on 20141216. 20141208
> > doesn't throw these, so it's something in the last week or so..
>
> Gaah! Turns out that 20141208 *is* susceptible - it had been booting
> just fine for several days, but it went around the bend, apparently due
> to a userspace or initrd change.
$5 says you updated systemd?
Richard?
Ok, so if you are correct, then either we justify dropping the lock (I
assume the one commone to both BUG reports [sig->cred_guard_mutex] ),
or we make yet another queue were were hoping to avoid...
It would also be good to narrow it down to a rule that triggers this.
> egrep 'BUG|Linux vers' from my syslog:
>
> Dec 9 12:19:53 turing-police kernel: [ 0.000000] Linux version
3.18.0-next-20141208 (source(a)turing-police.cc.vt.edu) (gcc version 4.9.2 20141101 (Red Hat
4.9.2-1) (GCC) ) #27 SMP PREEMPT Mon Dec 8 22:20:07 EST 2014
...
> Dec 12 19:42:30 turing-police kernel: [ 0.000000] Linux
version 3.18.0-next-20141208 (source(a)turing-police.cc.vt.edu) (gcc version 4.9.2 20141101
(Red Hat 4.9.2-1) (GCC) ) #27 SMP PREEMPT Mon Dec 8 22:20:07 EST 2014
> Dec 12 20:00:39 turing-police kernel: [ 1109.635328] BUG: sleeping function called
from invalid context at mm/slab.c:2849
...
> Dec 12 20:42:47 turing-police kernel: [ 3633.863552] BUG:
sleeping function called from invalid context at mm/slab.c:2849
> Dec 12 20:51:33 turing-police kernel: [ 0.000000] Linux version
3.18.0-next-20141208 (source(a)turing-police.cc.vt.edu) (gcc version 4.9.2 20141101 (Red Hat
4.9.2-1) (GCC) ) #27 SMP PREEMPT Mon Dec 8 22:20:07 EST 2014
> Dec 12 21:51:04 turing-police kernel: [ 3587.132867] BUG: sleeping function called
from invalid context at mm/slab.c:2849
...
> I need to figure out what changed around 7:30PM on the 12th.
- RGB
--
Richard Guy Briggs <rbriggs(a)redhat.com>
Senior Software Engineer, Kernel Security, AMER ENG Base Operating Systems, Red Hat
Remote, Ottawa, Canada
Voice: +1.647.777.2635, Internal: (81) 32635, Alt: +1.613.693.0684x3545
11:50 a.m.
New subject: linux-next 20141216 BUG: sleeping function called from invalid context at mm/slab.c:2849
On Thu, 2014-12-18 at 12:46 -0500, Richard Guy Briggs wrote:
On 14/12/18, Eric Paris wrote:
> On Thu, 2014-12-18 at 11:45 -0500, Valdis.Kletnieks(a)vt.edu wrote:
> > On Tue, 16 Dec 2014 20:09:54 -0500, Valdis Kletnieks said:
> > > Spotted these two while booting single-user on 20141216. 20141208
> > > doesn't throw these, so it's something in the last week or so..
> >
> > Gaah! Turns out that 20141208 *is* susceptible - it had been booting
> > just fine for several days, but it went around the bend, apparently due
> > to a userspace or initrd change.
>
> $5 says you updated systemd?
>
> Richard?
Ok, so if you are correct, then either we justify dropping the lock (I
assume the one commone to both BUG reports [sig->cred_guard_mutex] ),
or we make yet another queue were were hoping to avoid...
It would also be good to narrow it down to a rule that triggers this.
I thought the first message was enough to find the problem, but:
static void kauditd_send_multicast_skb(struct sk_buff *skb)
{
...
nlmsg_multicast(sock, copy, 0, AUDIT_NLGRP_READLOG, GFP_KERNEL);
...
}
Since kauditd_send_multicast_skb() gets called in audit_log_end(), which
can come from any context (aka even a sleeping context) you can't use
GFP_KERNEL. The audit_buffer know what context it should use. So pass
that down and use that.
-Eric
> > egrep 'BUG|Linux vers' from my syslog:
> >
> > Dec 9 12:19:53 turing-police kernel: [ 0.000000] Linux version
3.18.0-next-20141208 (source(a)turing-police.cc.vt.edu) (gcc version 4.9.2 20141101 (Red Hat
4.9.2-1) (GCC) ) #27 SMP PREEMPT Mon Dec 8 22:20:07 EST 2014
...
> > Dec 12 19:42:30 turing-police kernel: [ 0.000000] Linux version
3.18.0-next-20141208 (source(a)turing-police.cc.vt.edu) (gcc version 4.9.2 20141101 (Red Hat
4.9.2-1) (GCC) ) #27 SMP PREEMPT Mon Dec 8 22:20:07 EST 2014
> > Dec 12 20:00:39 turing-police kernel: [ 1109.635328] BUG: sleeping function
called from invalid context at mm/slab.c:2849
...
> > Dec 12 20:42:47 turing-police kernel: [ 3633.863552] BUG: sleeping function
called from invalid context at mm/slab.c:2849
> > Dec 12 20:51:33 turing-police kernel: [ 0.000000] Linux version
3.18.0-next-20141208 (source(a)turing-police.cc.vt.edu) (gcc version 4.9.2 20141101 (Red Hat
4.9.2-1) (GCC) ) #27 SMP PREEMPT Mon Dec 8 22:20:07 EST 2014
> > Dec 12 21:51:04 turing-police kernel: [ 3587.132867] BUG: sleeping function
called from invalid context at mm/slab.c:2849
...
> > I need to figure out what changed around 7:30PM on the 12th.
- RGB
--
Richard Guy Briggs <rbriggs(a)redhat.com>
Senior Software Engineer, Kernel Security, AMER ENG Base Operating Systems, Red Hat
Remote, Ottawa, Canada
Voice: +1.647.777.2635, Internal: (81) 32635, Alt: +1.613.693.0684x3545
12:44 p.m.
New subject: linux-next 20141216 BUG: sleeping function called from invalid context at mm/slab.c:2849
On 14/12/18, Eric Paris wrote:
On Thu, 2014-12-18 at 12:46 -0500, Richard Guy Briggs wrote:
> On 14/12/18, Eric Paris wrote:
> > On Thu, 2014-12-18 at 11:45 -0500, Valdis.Kletnieks(a)vt.edu wrote:
> > > On Tue, 16 Dec 2014 20:09:54 -0500, Valdis Kletnieks said:
> > > > Spotted these two while booting single-user on 20141216. 20141208
> > > > doesn't throw these, so it's something in the last week or
so..
> > >
> > > Gaah! Turns out that 20141208 *is* susceptible - it had been booting
> > > just fine for several days, but it went around the bend, apparently due
> > > to a userspace or initrd change.
> >
> > $5 says you updated systemd?
> >
> > Richard?
>
> Ok, so if you are correct, then either we justify dropping the lock (I
> assume the one commone to both BUG reports [sig->cred_guard_mutex] ),
> or we make yet another queue were were hoping to avoid...
>
> It would also be good to narrow it down to a rule that triggers this.
I thought the first message was enough to find the problem, but:
static void kauditd_send_multicast_skb(struct sk_buff *skb)
{
...
nlmsg_multicast(sock, copy, 0, AUDIT_NLGRP_READLOG, GFP_KERNEL);
...
}
Since kauditd_send_multicast_skb() gets called in audit_log_end(), which
can come from any context (aka even a sleeping context) you can't use
GFP_KERNEL. The audit_buffer know what context it should use. So pass
that down and use that.
Ok, that looks more obvious now... We just need to change the internal
interface to kauditd_send_multicast_skb() to accept an audit_buffer
instead of just the skb and use the gfp_mask value from there instead of
using our own...
Thanks, Eric.
-Eric
> > > egrep 'BUG|Linux vers' from my syslog:
> > >
> > > Dec 9 12:19:53 turing-police kernel: [ 0.000000] Linux version
3.18.0-next-20141208 (source(a)turing-police.cc.vt.edu) (gcc version 4.9.2 20141101 (Red Hat
4.9.2-1) (GCC) ) #27 SMP PREEMPT Mon Dec 8 22:20:07 EST 2014
> ...
> > > Dec 12 19:42:30 turing-police kernel: [ 0.000000] Linux version
3.18.0-next-20141208 (source(a)turing-police.cc.vt.edu) (gcc version 4.9.2 20141101 (Red Hat
4.9.2-1) (GCC) ) #27 SMP PREEMPT Mon Dec 8 22:20:07 EST 2014
> > > Dec 12 20:00:39 turing-police kernel: [ 1109.635328] BUG: sleeping
function called from invalid context at mm/slab.c:2849
> ...
> > > Dec 12 20:42:47 turing-police kernel: [ 3633.863552] BUG: sleeping
function called from invalid context at mm/slab.c:2849
> > > Dec 12 20:51:33 turing-police kernel: [ 0.000000] Linux version
3.18.0-next-20141208 (source(a)turing-police.cc.vt.edu) (gcc version 4.9.2 20141101 (Red Hat
4.9.2-1) (GCC) ) #27 SMP PREEMPT Mon Dec 8 22:20:07 EST 2014
> > > Dec 12 21:51:04 turing-police kernel: [ 3587.132867] BUG: sleeping
function called from invalid context at mm/slab.c:2849
> ...
> > > I need to figure out what changed around 7:30PM on the 12th.
>
> - RGB
- RGB
--
Richard Guy Briggs <rbriggs(a)redhat.com>
Senior Software Engineer, Kernel Security, AMER ENG Base Operating Systems, Red Hat
Remote, Ottawa, Canada
Voice: +1.647.777.2635, Internal: (81) 32635, Alt: +1.613.693.0684x3545
1:04 p.m.
New subject: linux-next 20141216 BUG: sleeping function called from invalid context at mm/slab.c:2849
On Thu, 2014-12-18 at 13:44 -0500, Richard Guy Briggs wrote:
On 14/12/18, Eric Paris wrote:
> On Thu, 2014-12-18 at 12:46 -0500, Richard Guy Briggs wrote:
> > On 14/12/18, Eric Paris wrote:
> > > On Thu, 2014-12-18 at 11:45 -0500, Valdis.Kletnieks(a)vt.edu wrote:
> > > > On Tue, 16 Dec 2014 20:09:54 -0500, Valdis Kletnieks said:
> > > > > Spotted these two while booting single-user on 20141216.
20141208
> > > > > doesn't throw these, so it's something in the last week
or so..
> > > >
> > > > Gaah! Turns out that 20141208 *is* susceptible - it had been
booting
> > > > just fine for several days, but it went around the bend, apparently
due
> > > > to a userspace or initrd change.
> > >
> > > $5 says you updated systemd?
> > >
> > > Richard?
> >
> > Ok, so if you are correct, then either we justify dropping the lock (I
> > assume the one commone to both BUG reports [sig->cred_guard_mutex] ),
> > or we make yet another queue were were hoping to avoid...
> >
> > It would also be good to narrow it down to a rule that triggers this.
>
> I thought the first message was enough to find the problem, but:
>
> static void kauditd_send_multicast_skb(struct sk_buff *skb)
> {
> ...
> nlmsg_multicast(sock, copy, 0, AUDIT_NLGRP_READLOG, GFP_KERNEL);
> ...
> }
>
> Since kauditd_send_multicast_skb() gets called in audit_log_end(), which
> can come from any context (aka even a sleeping context) you can't use
> GFP_KERNEL. The audit_buffer know what context it should use. So pass
> that down and use that.
Ok, that looks more obvious now... We just need to change the internal
interface to kauditd_send_multicast_skb() to accept an audit_buffer
instead of just the skb and use the gfp_mask value from there instead of
using our own...
Thanks, Eric.
I'd suggest just sending the GFP type, not the who audit_buffer, but
that's up to you.
-Eric
2:03 p.m.
New subject: linux-next 20141216 BUG: sleeping function called from invalid context at mm/slab.c:2849
On Thursday, December 18, 2014 02:04:46 PM Eric Paris wrote:
I'd suggest just sending the GFP type, not the who audit_buffer,
but
that's up to you.
That would be my preference too, especially since we will want to send this to
stable and smaller is generally better there.
--
paul moore
security and virtualization @ redhat
1:21 p.m.
New subject: linux-next 20141216 BUG: sleeping function called from invalid context at mm/slab.c:2849
On Thu, 18 Dec 2014 11:50:20 -0500, Eric Paris said:
On Thu, 2014-12-18 at 11:45 -0500, Valdis.Kletnieks(a)vt.edu wrote:
> On Tue, 16 Dec 2014 20:09:54 -0500, Valdis Kletnieks said:
>
> > Spotted these two while booting single-user on 20141216. 20141208
> > doesn't throw these, so it's something in the last week or so..
>
> Gaah! Turns out that 20141208 *is* susceptible - it had been booting
> just fine for several days, but it went around the bend, apparently due
> to a userspace or initrd change.
$5 says you updated systemd?
Actually, yeah. yum.log says:
Dec 12 14:08:09 Updated: systemd-218-1.fc22.x86_64
and things started going downhill that evening. Damned if I know what 218
was doing to cause the issues. (Other RPM update that looked vaguely
related was for selinux-policy, but again, damned if I know what a policy
file could contain that would change the behavior...)
Is it worth backing it to -217, or do we have a handle on the issue and
systemd is just the messenger?
1:24 p.m.
New subject: linux-next 20141216 BUG: sleeping function called from invalid context at mm/slab.c:2849
On 14/12/18, Valdis.Kletnieks(a)vt.edu wrote:
On Thu, 18 Dec 2014 11:50:20 -0500, Eric Paris said:
> On Thu, 2014-12-18 at 11:45 -0500, Valdis.Kletnieks(a)vt.edu wrote:
> > On Tue, 16 Dec 2014 20:09:54 -0500, Valdis Kletnieks said:
> > > Spotted these two while booting single-user on 20141216. 20141208
> > > doesn't throw these, so it's something in the last week or so..
> >
> > Gaah! Turns out that 20141208 *is* susceptible - it had been booting
> > just fine for several days, but it went around the bend, apparently due
> > to a userspace or initrd change.
>
> $5 says you updated systemd?
Actually, yeah. yum.log says:
Dec 12 14:08:09 Updated: systemd-218-1.fc22.x86_64
and things started going downhill that evening. Damned if I know what 218
was doing to cause the issues. (Other RPM update that looked vaguely
related was for selinux-policy, but again, damned if I know what a policy
file could contain that would change the behavior...)
Is it worth backing it to -217, or do we have a handle on the issue and
systemd is just the messenger?
I've got a potential fix since the problem looks pretty obvious now.
This is stable branch material...
- RGB
--
Richard Guy Briggs <rbriggs(a)redhat.com>
Senior Software Engineer, Kernel Security, AMER ENG Base Operating Systems, Red Hat
Remote, Ottawa, Canada
Voice: +1.647.777.2635, Internal: (81) 32635, Alt: +1.613.693.0684x3545
3605
days inactive
3605
days old
linux-audit@lists.linux-audit.osci.io
7 comments
4 participants
participants (4)
-
Eric Paris -
Paul Moore -
Richard Guy Briggs -
Valdis.Kletnieks＠vt.edu