Hey Audit team,

I've got a kernel null pointer deref oops on Amazon Linux kernel 4.9.51-10.52.amzn1.x86_64. After the oops all new cron processes that spawned were stuck in D wait on some audit related syscall. This exhausted system file handles and the box ended up needing to be rebooted out-of-bounds. The audit handle was held by https://github.com/facebook/osquery/. It looks like there was an issue with audit sending to osquery (netlink_unicast sending to pid 21233, error -111) followed by something (presumably also osquery) attempting to 'op="add_rule" key=(null)'.

Here's from /var/log/messages. > Aug 13 14:34:54 packer_default-10-180-21-24 kernel: [4749520.133904] audit: netlink_unicast sending to audit_pid=21233 returned error: -111 > Aug 13 14:34:54 packer_default-10-180-21-24 kernel: [4749520.137982] audit_log_lost: 12 callbacks suppressed > Aug 13 14:34:54 packer_default-10-180-21-24 kernel: [4749520.137983] audit: audit_lost=36081041 audit_rate_limit=8192 audit_backlog_limit=4096 > Aug 13 14:34:54 packer_default-10-180-21-24 kernel: [4749520.137985] audit: type=1305 audit(1534170894.852:117649229): audit_pid=21233 old=21233 auid=501 ses=102936 res=0 > Aug 13 14:34:54 packer_default-10-180-21-24 kernel: [4749520.137986] audit: type=1305 audit(1534170894.852:117649230): audit_enabled=1 old=1 auid=501 ses=102936 res=1 > Aug 13 14:34:54 packer_default-10-180-21-24 kernel: [4749520.137987] audit: type=1305 audit(1534170894.852:117649231): audit_backlog_wait_time=1 old=1 auid=501 ses=102936 res=1 > Aug 13 14:34:54 packer_default-10-180-21-24 kernel: [4749520.137989] audit: type=1305 audit(1534170894.852:117649232): audit_backlog_limit=4096 old=4096 auid=501 ses=102936 res=1 > Aug 13 14:34:54 packer_default-10-180-21-24 kernel: [4749520.137990] audit: type=1305 audit(1534170894.852:117649233): audit_failure=0 old=0 auid=501 ses=102936 res=1 > Aug 13 14:34:54 packer_default-10-180-21-24 kernel: [4749520.137991] audit: type=1305 audit(1534170894.852:117649234): auid=501 ses=102936 op="add_rule" key=(null) list=4 res=0 > Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.138007] BUG: unable to handle kernel NULL pointer dereference at 00000000000001e0 > Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.143872] IP: [<ffffffff814a76da>] netlink_unicast+0x4a/0x230 > Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.148336] PGD 0 > Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.149822] > Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.151230] Oops: 0000 [#1] SMP > Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.153736] Modules linked in: iptable_filter(E) ip_tables(E) x_tables(E) udp_diag(E) nfnetlink_queue(E) nfnetlink_log(E) nfnetlink(E) tcp_diag(E) inet_diag(E) isofs(E) ipv6(E) crc_ > Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.184036] CPU: 0 PID: 21372 Comm: osqueryd Tainted: G E 4.9.51-10.52.amzn1.x86_64 #1 > Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.190620] Hardware name: Xen HVM domU, BIOS 4.2.amazon 08/24/2006 > Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.195513] task: ffff8807fa1c3b00 task.stack: ffffc900079a4000 > Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.200179] RIP: 0010:[<ffffffff814a76da>] [<ffffffff814a76da>] netlink_unicast+0x4a/0x230 > Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.206822] RSP: 0018:ffffc900079a7c38 EFLAGS: 00010246 > Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.211065] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000 > Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.216779] RDX: 00001ffffffffffe RSI: 00000000024000c0 RDI: 0000000000000014 > Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.222473] RBP: ffffc900079a7c68 R08: 0000000000000004 R09: ffff88062a375414 > Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.228167] R10: ffff8808004032c0 R11: ffff88062a375400 R12: 0000000000000000 > Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.233738] R13: ffff8804200a0100 R14: 00000000000052f1 R15: 0000000000016d42 > Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.238999] FS: 00007f3d40cc4700(0000) GS:ffff880800a00000(0000) knlGS:0000000000000000 > Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.244796] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.249555] CR2: 00000000000001e0 CR3: 000000059e63d000 CR4: 00000000001406f0 > Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.255401] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.260725] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 > Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.266038] Stack: > Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.267680] 00000000079a7c58 00000000000052f1 00000000000052f1 00000000ffffffa1 > Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.273540] ffff88062a376800 ffff8804200a1d00 ffffc900079a7cf8 ffffffff811107d2 > Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.279379] ffffc90000000004 ffffffff811e2f85 ffffc900079a7ca8 ffffffff8145a08c > Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.285201] Call Trace: > Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.287137] [<ffffffff811107d2>] audit_receive_msg+0x992/0xc20 > Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.291561] [<ffffffff811e2f85>] ? __kmalloc_node_track_caller+0x35/0x260 > Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.296642] [<ffffffff8145a08c>] ? release_sock+0x8c/0xa0 > Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.300718] [<ffffffff81110ab2>] audit_receive+0x52/0xa0 > Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.304713] [<ffffffff814a77ef>] netlink_unicast+0x15f/0x230 > Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.308943] [<ffffffff814a7bdc>] netlink_sendmsg+0x31c/0x390 > Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.313236] [<ffffffff81455818>] sock_sendmsg+0x38/0x50 > Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.317219] [<ffffffff81455c4f>] SYSC_sendto+0xef/0x170 > Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.321142] [<ffffffff811156ab>] ? __audit_syscall_entry+0xeb/0x100 > Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.325853] [<ffffffff81003437>] ? syscall_trace_enter+0x1b7/0x290 > Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.330464] [<ffffffff811158b3>] ? __audit_syscall_exit+0x1f3/0x280 > Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.335189] [<ffffffff8145673e>] SyS_sendto+0xe/0x10 > Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.338952] [<ffffffff81003854>] do_syscall_64+0x54/0xc0 > Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.342995] [<ffffffff8153d32b>] entry_SYSCALL64_slow_path+0x25/0x25 > Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.347840] Code: 65 8b 05 a2 5b b6 7e 25 00 ff 00 00 83 f8 01 19 f6 81 e6 a0 00 38 00 81 c6 20 00 08 02 e8 af cf ff ff 49 89 c5 31 c0 85 db 75 08 <49> 8b 84 24 e0 01 00 00 48 89 45 > Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.368625] RIP [<ffffffff814a76da>] netlink_unicast+0x4a/0x230 > Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.373253] RSP <ffffc900079a7c38> > Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.376013] CR2: 00000000000001e0 > Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.379307] ---[ end trace 6a41b2274729ba2a ]--- > Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.663029] audit: type=1300 audit(1534170895.379:117649235): arch=c000003e syscall=59 success=yes exit=0 a0=271cce0 a1=2741dd0 a2=26dc3b0 a3=7ffdd7f1ea80 items=2 ppid=4556 pid=2407 > Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.682741] audit: type=1309 audit(1534170895.379:117649235): argc=2 a0="date" a1="+%s" > Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.688585] audit: type=1307 audit(1534170895.379:117649235): cwd="/" > Aug 13 14:34:55 packer_default-10-180-21-24 abrt-dump-oops: Reported 1 kernel oopses to Abrt -- Linux-audit mailing list Linux-audit(a)redhat.com https://www.redhat.com/mailman/listinfo/linux-audit

Hey Paul, My expectations are aligned to what you've pointed out already (not-vanilla old kernel, user space client other than auditd). Primarily what I'm looking for is a sanity check on the null pointer deref on current vanilla kernel code because I think the rest of the issues stem from that. My C isn't sharp enough to try my hand at a reproduction case, though I would hope someone with the requisite expertise could put one together relatively quickly. specifically this message: audit: type=1305 audit(1534170894.852:117649234): auid=501 ses=102936 > op="add_rule" key=(null) list=4 res=0

On Fri, Aug 17, 2018 at 2:12 PM Paul Moore <paul(a)paul-moore.com&gt; wrote: > On Fri, Aug 17, 2018 at 4:26 PM Preston Bennes > <preston.bennes(a)snowflake.net&gt; wrote: > > Hey Audit team, > > Hey Audit user! > > > I've got a kernel null pointer deref oops on Amazon Linux kernel > 4.9.51-10.52.amzn1.x86_64. After the oops all new cron processes that > spawned were stuck in D wait on some audit related syscall. This exhausted > system file handles and the box ended up needing to be rebooted > out-of-bounds. The audit handle was held by > https://github.com/facebook/osquery/. > > > > It looks like there was an issue with audit sending to osquery > (netlink_unicast sending to pid 21233, error -111) followed by something > (presumably also osquery) attempting to 'op="add_rule" key=(null)'. > > The first thing I need to comment on is that since this is a distro > specific kernel (4.9.51-10.52.amzn1.x86_64) and not an upstream kernel > there is a limit in the amount of help we can provide. Generally the > upstream list focuses on the upstream code and we leave support for > the distro specific builds/packages up to the distro. If you can > reproduce your problem on an upstream kernel, with a strong preference > for a *recent* upstream kernel, we can probably be of more help. I > simply don't know what patches Amazon merges into it's amzn1 kernels. > > With that out of the way, I will add that v4.9.x is a rather old > kernel, and we've made a number of relevant improvements since the > v4.9 release and it is unlikely all of them have been backported to > your kernel. If possible, I would suggest upgrading to a newer kernel > as it may resolve your problem. > > There is also a possible issue of osquery as an auditd replacement; > it's unlikely this is being caused by osquery, but we don't test with > osquery to that save level as we do with Steve's audit userspace > (auditd). > > > Here's from /var/log/messages. > > > >> Aug 13 14:34:54 packer_default-10-180-21-24 kernel: [4749520.133904] > audit: netlink_unicast sending to audit_pid=21233 returned error: -111 > >> Aug 13 14:34:54 packer_default-10-180-21-24 kernel: [4749520.137982] > audit_log_lost: 12 callbacks suppressed > >> Aug 13 14:34:54 packer_default-10-180-21-24 kernel: [4749520.137983] > audit: audit_lost=36081041 audit_rate_limit=8192 audit_backlog_limit=4096 > >> Aug 13 14:34:54 packer_default-10-180-21-24 kernel: [4749520.137985] > audit: type=1305 audit(1534170894.852:117649229): audit_pid=21233 old=21233 > auid=501 ses=102936 res=0 > >> Aug 13 14:34:54 packer_default-10-180-21-24 kernel: [4749520.137986] > audit: type=1305 audit(1534170894.852:117649230): audit_enabled=1 old=1 > auid=501 ses=102936 res=1 > >> Aug 13 14:34:54 packer_default-10-180-21-24 kernel: [4749520.137987] > audit: type=1305 audit(1534170894.852:117649231): audit_backlog_wait_time=1 > old=1 auid=501 ses=102936 res=1 > >> Aug 13 14:34:54 packer_default-10-180-21-24 kernel: [4749520.137989] > audit: type=1305 audit(1534170894.852:117649232): audit_backlog_limit=4096 > old=4096 auid=501 ses=102936 res=1 > >> Aug 13 14:34:54 packer_default-10-180-21-24 kernel: [4749520.137990] > audit: type=1305 audit(1534170894.852:117649233): audit_failure=0 old=0 > auid=501 ses=102936 res=1 > >> Aug 13 14:34:54 packer_default-10-180-21-24 kernel: [4749520.137991] > audit: type=1305 audit(1534170894.852:117649234): auid=501 ses=102936 > op="add_rule" key=(null) list=4 res=0 > >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.138007] > BUG: unable to handle kernel NULL pointer dereference at 00000000000001e0 > >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.143872] > IP: [<ffffffff814a76da>] netlink_unicast+0x4a/0x230 > >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.148336] > PGD 0 > >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.149822] > >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.151230] > Oops: 0000 [#1] SMP > >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.153736] > Modules linked in: iptable_filter(E) ip_tables(E) x_tables(E) udp_diag(E) > nfnetlink_queue(E) nfnetlink_log(E) nfnetlink(E) tcp_diag(E) inet_diag(E) > isofs(E) ipv6(E) crc_ > >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.184036] > CPU: 0 PID: 21372 Comm: osqueryd Tainted: G E > 4.9.51-10.52.amzn1.x86_64 #1 > >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.190620] > Hardware name: Xen HVM domU, BIOS 4.2.amazon 08/24/2006 > >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.195513] > task: ffff8807fa1c3b00 task.stack: ffffc900079a4000 > >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.200179] > RIP: 0010:[<ffffffff814a76da>] [<ffffffff814a76da>] > netlink_unicast+0x4a/0x230 > >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.206822] > RSP: 0018:ffffc900079a7c38 EFLAGS: 00010246 > >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.211065] > RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000 > >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.216779] > RDX: 00001ffffffffffe RSI: 00000000024000c0 RDI: 0000000000000014 > >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.222473] > RBP: ffffc900079a7c68 R08: 0000000000000004 R09: ffff88062a375414 > >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.228167] > R10: ffff8808004032c0 R11: ffff88062a375400 R12: 0000000000000000 > >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.233738] > R13: ffff8804200a0100 R14: 00000000000052f1 R15: 0000000000016d42 > >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.238999] > FS: 00007f3d40cc4700(0000) GS:ffff880800a00000(0000) knlGS:0000000000000000 > >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.244796] > CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.249555] > CR2: 00000000000001e0 CR3: 000000059e63d000 CR4: 00000000001406f0 > >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.255401] > DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.260725] > DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 > >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.266038] > Stack: > >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.267680] > 00000000079a7c58 00000000000052f1 00000000000052f1 00000000ffffffa1 > >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.273540] > ffff88062a376800 ffff8804200a1d00 ffffc900079a7cf8 ffffffff811107d2 > >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.279379] > ffffc90000000004 ffffffff811e2f85 ffffc900079a7ca8 ffffffff8145a08c > >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.285201] > Call Trace: > >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.287137] > [<ffffffff811107d2>] audit_receive_msg+0x992/0xc20 > >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.291561] > [<ffffffff811e2f85>] ? __kmalloc_node_track_caller+0x35/0x260 > >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.296642] > [<ffffffff8145a08c>] ? release_sock+0x8c/0xa0 > >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.300718] > [<ffffffff81110ab2>] audit_receive+0x52/0xa0 > >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.304713] > [<ffffffff814a77ef>] netlink_unicast+0x15f/0x230 > >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.308943] > [<ffffffff814a7bdc>] netlink_sendmsg+0x31c/0x390 > >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.313236] > [<ffffffff81455818>] sock_sendmsg+0x38/0x50 > >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.317219] > [<ffffffff81455c4f>] SYSC_sendto+0xef/0x170 > >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.321142] > [<ffffffff811156ab>] ? __audit_syscall_entry+0xeb/0x100 > >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.325853] > [<ffffffff81003437>] ? syscall_trace_enter+0x1b7/0x290 > >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.330464] > [<ffffffff811158b3>] ? __audit_syscall_exit+0x1f3/0x280 > >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.335189] > [<ffffffff8145673e>] SyS_sendto+0xe/0x10 > >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.338952] > [<ffffffff81003854>] do_syscall_64+0x54/0xc0 > >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.342995] > [<ffffffff8153d32b>] entry_SYSCALL64_slow_path+0x25/0x25 > >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.347840] > Code: 65 8b 05 a2 5b b6 7e 25 00 ff 00 00 83 f8 01 19 f6 81 e6 a0 00 38 00 > 81 c6 20 00 08 02 e8 af cf ff ff 49 89 c5 31 c0 85 db 75 08 <49> 8b 84 24 > e0 01 00 00 48 89 45 > >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.368625] > RIP [<ffffffff814a76da>] netlink_unicast+0x4a/0x230 > >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.373253] > RSP <ffffc900079a7c38> > >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.376013] > CR2: 00000000000001e0 > >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.379307] > ---[ end trace 6a41b2274729ba2a ]--- > >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.663029] > audit: type=1300 audit(1534170895.379:117649235): arch=c000003e syscall=59 > success=yes exit=0 a0=271cce0 a1=2741dd0 a2=26dc3b0 a3=7ffdd7f1ea80 items=2 > ppid=4556 pid=2407 > >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.682741] > audit: type=1309 audit(1534170895.379:117649235): argc=2 a0="date" a1="+%s" > >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.688585] > audit: type=1307 audit(1534170895.379:117649235): cwd="/" > >> Aug 13 14:34:55 packer_default-10-180-21-24 abrt-dump-oops: Reported 1 > kernel oopses to Abrt > > paul moore

On Fri, Aug 17, 2018 at 5:24 PM Preston Bennes <preston.bennes(a)snowflake.net&gt; wrote: > Hey Paul, > > My expectations are aligned to what you've pointed out already (not-vanilla old kernel, user space client other than auditd). > > Primarily what I'm looking for is a sanity check on the null pointer deref on current vanilla kernel code because I think the rest of the issues stem from that. My C isn't sharp enough to try my hand at a reproduction case, though I would hope someone with the requisite expertise could put one together relatively quickly. specifically this message: > >> audit: type=1305 audit(1534170894.852:117649234): auid=501 ses=102936 op="add_rule" key=(null) list=4 res=0 Unfortunately without digging into Amazon's source for that kernel build it is difficult to come to any strong conclusion. The audit message you are seeing above is an AUDIT_CONFIG_CHANGE (type == 1305), and based on the 'op="add_rule"' field it appears as though the system is adding an exit (list == 4) filter rule to the kernel. The operation failed (res == 0), which shouldn't be surprising given the kernel oops. I'm not very familiar with osquery, does it load it's own audit filter rules? Are you loading some of your own? > On Fri, Aug 17, 2018 at 2:12 PM Paul Moore <paul(a)paul-moore.com&gt; wrote: >> >> On Fri, Aug 17, 2018 at 4:26 PM Preston Bennes >> <preston.bennes(a)snowflake.net&gt; wrote: >> > Hey Audit team, >> >> Hey Audit user! >> >> > I've got a kernel null pointer deref oops on Amazon Linux kernel 4.9.51-10.52.amzn1.x86_64. After the oops all new cron processes that spawned were stuck in D wait on some audit related syscall. This exhausted system file handles and the box ended up needing to be rebooted out-of-bounds. The audit handle was held by https://github.com/facebook/osquery/. >> > >> > It looks like there was an issue with audit sending to osquery (netlink_unicast sending to pid 21233, error -111) followed by something (presumably also osquery) attempting to 'op="add_rule" key=(null)'. >> >> The first thing I need to comment on is that since this is a distro >> specific kernel (4.9.51-10.52.amzn1.x86_64) and not an upstream kernel >> there is a limit in the amount of help we can provide. Generally the >> upstream list focuses on the upstream code and we leave support for >> the distro specific builds/packages up to the distro. If you can >> reproduce your problem on an upstream kernel, with a strong preference >> for a *recent* upstream kernel, we can probably be of more help. I >> simply don't know what patches Amazon merges into it's amzn1 kernels. >> >> With that out of the way, I will add that v4.9.x is a rather old >> kernel, and we've made a number of relevant improvements since the >> v4.9 release and it is unlikely all of them have been backported to >> your kernel. If possible, I would suggest upgrading to a newer kernel >> as it may resolve your problem. >> >> There is also a possible issue of osquery as an auditd replacement; >> it's unlikely this is being caused by osquery, but we don't test with >> osquery to that save level as we do with Steve's audit userspace >> (auditd). >> >> > Here's from /var/log/messages. >> > >> >> Aug 13 14:34:54 packer_default-10-180-21-24 kernel: [4749520.133904] audit: netlink_unicast sending to audit_pid=21233 returned error: -111 >> >> Aug 13 14:34:54 packer_default-10-180-21-24 kernel: [4749520.137982] audit_log_lost: 12 callbacks suppressed >> >> Aug 13 14:34:54 packer_default-10-180-21-24 kernel: [4749520.137983] audit: audit_lost=36081041 audit_rate_limit=8192 audit_backlog_limit=4096 >> >> Aug 13 14:34:54 packer_default-10-180-21-24 kernel: [4749520.137985] audit: type=1305 audit(1534170894.852:117649229): audit_pid=21233 old=21233 auid=501 ses=102936 res=0 >> >> Aug 13 14:34:54 packer_default-10-180-21-24 kernel: [4749520.137986] audit: type=1305 audit(1534170894.852:117649230): audit_enabled=1 old=1 auid=501 ses=102936 res=1 >> >> Aug 13 14:34:54 packer_default-10-180-21-24 kernel: [4749520.137987] audit: type=1305 audit(1534170894.852:117649231): audit_backlog_wait_time=1 old=1 auid=501 ses=102936 res=1 >> >> Aug 13 14:34:54 packer_default-10-180-21-24 kernel: [4749520.137989] audit: type=1305 audit(1534170894.852:117649232): audit_backlog_limit=4096 old=4096 auid=501 ses=102936 res=1 >> >> Aug 13 14:34:54 packer_default-10-180-21-24 kernel: [4749520.137990] audit: type=1305 audit(1534170894.852:117649233): audit_failure=0 old=0 auid=501 ses=102936 res=1 >> >> Aug 13 14:34:54 packer_default-10-180-21-24 kernel: [4749520.137991] audit: type=1305 audit(1534170894.852:117649234): auid=501 ses=102936 op="add_rule" key=(null) list=4 res=0 >> >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.138007] BUG: unable to handle kernel NULL pointer dereference at 00000000000001e0 >> >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.143872] IP: [<ffffffff814a76da>] netlink_unicast+0x4a/0x230 >> >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.148336] PGD 0 >> >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.149822] >> >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.151230] Oops: 0000 [#1] SMP >> >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.153736] Modules linked in: iptable_filter(E) ip_tables(E) x_tables(E) udp_diag(E) nfnetlink_queue(E) nfnetlink_log(E) nfnetlink(E) tcp_diag(E) inet_diag(E) isofs(E) ipv6(E) crc_ >> >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.184036] CPU: 0 PID: 21372 Comm: osqueryd Tainted: G E 4.9.51-10.52.amzn1.x86_64 #1 >> >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.190620] Hardware name: Xen HVM domU, BIOS 4.2.amazon 08/24/2006 >> >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.195513] task: ffff8807fa1c3b00 task.stack: ffffc900079a4000 >> >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.200179] RIP: 0010:[<ffffffff814a76da>] [<ffffffff814a76da>] netlink_unicast+0x4a/0x230 >> >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.206822] RSP: 0018:ffffc900079a7c38 EFLAGS: 00010246 >> >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.211065] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000 >> >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.216779] RDX: 00001ffffffffffe RSI: 00000000024000c0 RDI: 0000000000000014 >> >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.222473] RBP: ffffc900079a7c68 R08: 0000000000000004 R09: ffff88062a375414 >> >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.228167] R10: ffff8808004032c0 R11: ffff88062a375400 R12: 0000000000000000 >> >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.233738] R13: ffff8804200a0100 R14: 00000000000052f1 R15: 0000000000016d42 >> >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.238999] FS: 00007f3d40cc4700(0000) GS:ffff880800a00000(0000) knlGS:0000000000000000 >> >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.244796] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 >> >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.249555] CR2: 00000000000001e0 CR3: 000000059e63d000 CR4: 00000000001406f0 >> >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.255401] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 >> >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.260725] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 >> >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.266038] Stack: >> >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.267680] 00000000079a7c58 00000000000052f1 00000000000052f1 00000000ffffffa1 >> >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.273540] ffff88062a376800 ffff8804200a1d00 ffffc900079a7cf8 ffffffff811107d2 >> >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.279379] ffffc90000000004 ffffffff811e2f85 ffffc900079a7ca8 ffffffff8145a08c >> >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.285201] Call Trace: >> >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.287137] [<ffffffff811107d2>] audit_receive_msg+0x992/0xc20 >> >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.291561] [<ffffffff811e2f85>] ? __kmalloc_node_track_caller+0x35/0x260 >> >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.296642] [<ffffffff8145a08c>] ? release_sock+0x8c/0xa0 >> >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.300718] [<ffffffff81110ab2>] audit_receive+0x52/0xa0 >> >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.304713] [<ffffffff814a77ef>] netlink_unicast+0x15f/0x230 >> >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.308943] [<ffffffff814a7bdc>] netlink_sendmsg+0x31c/0x390 >> >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.313236] [<ffffffff81455818>] sock_sendmsg+0x38/0x50 >> >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.317219] [<ffffffff81455c4f>] SYSC_sendto+0xef/0x170 >> >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.321142] [<ffffffff811156ab>] ? __audit_syscall_entry+0xeb/0x100 >> >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.325853] [<ffffffff81003437>] ? syscall_trace_enter+0x1b7/0x290 >> >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.330464] [<ffffffff811158b3>] ? __audit_syscall_exit+0x1f3/0x280 >> >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.335189] [<ffffffff8145673e>] SyS_sendto+0xe/0x10 >> >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.338952] [<ffffffff81003854>] do_syscall_64+0x54/0xc0 >> >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.342995] [<ffffffff8153d32b>] entry_SYSCALL64_slow_path+0x25/0x25 >> >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.347840] Code: 65 8b 05 a2 5b b6 7e 25 00 ff 00 00 83 f8 01 19 f6 81 e6 a0 00 38 00 81 c6 20 00 08 02 e8 af cf ff ff 49 89 c5 31 c0 85 db 75 08 <49> 8b 84 24 e0 01 00 00 48 89 45 >> >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.368625] RIP [<ffffffff814a76da>] netlink_unicast+0x4a/0x230 >> >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.373253] RSP <ffffc900079a7c38> >> >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.376013] CR2: 00000000000001e0 >> >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.379307] ---[ end trace 6a41b2274729ba2a ]--- >> >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.663029] audit: type=1300 audit(1534170895.379:117649235): arch=c000003e syscall=59 success=yes exit=0 a0=271cce0 a1=2741dd0 a2=26dc3b0 a3=7ffdd7f1ea80 items=2 ppid=4556 pid=2407 >> >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.682741] audit: type=1309 audit(1534170895.379:117649235): argc=2 a0="date" a1="+%s" >> >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.688585] audit: type=1307 audit(1534170895.379:117649235): cwd="/" >> >> Aug 13 14:34:55 packer_default-10-180-21-24 abrt-dump-oops: Reported 1 kernel oopses to Abrt >> > >> > -- >> > Linux-audit mailing list >> > Linux-audit(a)redhat.com >> > https://www.redhat.com/mailman/listinfo/linux-audit >> >> -- >> paul moore >> www.paul-moore.com -- paul moore www.paul-moore.com