3:04 p.m.
Hey Audit team,
I've got a kernel null pointer deref oops on Amazon Linux
kernel 4.9.51-10.52.amzn1.x86_64. After the oops all new cron processes
that spawned were stuck in D wait on some audit related syscall. This
exhausted system file handles and the box ended up needing to be rebooted
out-of-bounds. The audit handle was held by
https://github.com/facebook/osquery/.
It looks like there was an issue with audit sending to osquery
(netlink_unicast sending to pid 21233, error -111) followed by something
(presumably also osquery) attempting to 'op="add_rule" key=(null)'.
Here's from /var/log/messages.
Aug 13 14:34:54 packer_default-10-180-21-24 kernel: [4749520.133904] audit:
netlink_unicast sending to audit_pid=21233 returned error: -111
Aug 13 14:34:54 packer_default-10-180-21-24 kernel: [4749520.137982]
audit_log_lost: 12 callbacks suppressed
Aug 13 14:34:54 packer_default-10-180-21-24 kernel: [4749520.137983]
audit: audit_lost=36081041 audit_rate_limit=8192 audit_backlog_limit=4096
Aug 13 14:34:54 packer_default-10-180-21-24 kernel: [4749520.137985]
audit: type=1305 audit(1534170894.852:117649229): audit_pid=21233 old=21233
auid=501 ses=102936 res=0
Aug 13 14:34:54 packer_default-10-180-21-24 kernel: [4749520.137986]
audit: type=1305 audit(1534170894.852:117649230): audit_enabled=1 old=1
auid=501 ses=102936 res=1
Aug 13 14:34:54 packer_default-10-180-21-24 kernel: [4749520.137987]
audit: type=1305 audit(1534170894.852:117649231): audit_backlog_wait_time=1
old=1 auid=501 ses=102936 res=1
Aug 13 14:34:54 packer_default-10-180-21-24 kernel: [4749520.137989]
audit: type=1305 audit(1534170894.852:117649232): audit_backlog_limit=4096
old=4096 auid=501 ses=102936 res=1
Aug 13 14:34:54 packer_default-10-180-21-24 kernel: [4749520.137990]
audit: type=1305 audit(1534170894.852:117649233): audit_failure=0 old=0
auid=501 ses=102936 res=1
Aug 13 14:34:54 packer_default-10-180-21-24 kernel: [4749520.137991]
audit: type=1305 audit(1534170894.852:117649234): auid=501 ses=102936
op="add_rule" key=(null) list=4 res=0
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.138007] BUG:
unable to handle kernel NULL pointer dereference at 00000000000001e0
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.143872] IP:
[<ffffffff814a76da>] netlink_unicast+0x4a/0x230
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.148336] PGD 0
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.149822]
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.151230] Oops:
0000 [#1] SMP
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.153736]
Modules linked in: iptable_filter(E) ip_tables(E) x_tables(E) udp_diag(E)
nfnetlink_queue(E) nfnetlink_log(E) nfnetlink(E) tcp_diag(E) inet_diag(E)
isofs(E) ipv6(E) crc_
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.184036] CPU:
0 PID: 21372 Comm: osqueryd Tainted: G E
4.9.51-10.52.amzn1.x86_64 #1
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.190620]
Hardware name: Xen HVM domU, BIOS 4.2.amazon 08/24/2006
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.195513] task:
ffff8807fa1c3b00 task.stack: ffffc900079a4000
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.200179] RIP:
0010:[<ffffffff814a76da>] [<ffffffff814a76da>] netlink_unicast+0x4a/0x230
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.206822] RSP:
0018:ffffc900079a7c38 EFLAGS: 00010246
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.211065] RAX:
0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.216779] RDX:
00001ffffffffffe RSI: 00000000024000c0 RDI: 0000000000000014
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.222473] RBP:
ffffc900079a7c68 R08: 0000000000000004 R09: ffff88062a375414
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.228167] R10:
ffff8808004032c0 R11: ffff88062a375400 R12: 0000000000000000
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.233738] R13:
ffff8804200a0100 R14: 00000000000052f1 R15: 0000000000016d42
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.238999] FS:
00007f3d40cc4700(0000) GS:ffff880800a00000(0000) knlGS:0000000000000000
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.244796] CS:
0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.249555] CR2:
00000000000001e0 CR3: 000000059e63d000 CR4: 00000000001406f0
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.255401] DR0:
0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.260725] DR3:
0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.266038] Stack:
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.267680]
00000000079a7c58 00000000000052f1 00000000000052f1 00000000ffffffa1
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.273540]
ffff88062a376800 ffff8804200a1d00 ffffc900079a7cf8 ffffffff811107d2
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.279379]
ffffc90000000004 ffffffff811e2f85 ffffc900079a7ca8 ffffffff8145a08c
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.285201] Call
Trace:
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.287137]
[<ffffffff811107d2>] audit_receive_msg+0x992/0xc20
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.291561]
[<ffffffff811e2f85>] ? __kmalloc_node_track_caller+0x35/0x260
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.296642]
[<ffffffff8145a08c>] ? release_sock+0x8c/0xa0
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.300718]
[<ffffffff81110ab2>] audit_receive+0x52/0xa0
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.304713]
[<ffffffff814a77ef>] netlink_unicast+0x15f/0x230
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.308943]
[<ffffffff814a7bdc>] netlink_sendmsg+0x31c/0x390
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.313236]
[<ffffffff81455818>] sock_sendmsg+0x38/0x50
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.317219]
[<ffffffff81455c4f>] SYSC_sendto+0xef/0x170
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.321142]
[<ffffffff811156ab>] ? __audit_syscall_entry+0xeb/0x100
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.325853]
[<ffffffff81003437>] ? syscall_trace_enter+0x1b7/0x290
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.330464]
[<ffffffff811158b3>] ? __audit_syscall_exit+0x1f3/0x280
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.335189]
[<ffffffff8145673e>] SyS_sendto+0xe/0x10
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.338952]
[<ffffffff81003854>] do_syscall_64+0x54/0xc0
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.342995]
[<ffffffff8153d32b>] entry_SYSCALL64_slow_path+0x25/0x25
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.347840] Code:
65 8b 05 a2 5b b6 7e 25 00 ff 00 00 83 f8 01 19 f6 81 e6 a0 00 38 00 81 c6
20 00 08 02 e8 af cf ff ff 49 89 c5 31 c0 85 db 75 08 <49> 8b 84 24 e0 01
00 00 48 89 45
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.368625] RIP
[<ffffffff814a76da>] netlink_unicast+0x4a/0x230
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.373253] RSP
<ffffc900079a7c38>
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.376013] CR2:
00000000000001e0
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.379307] ---[
end trace 6a41b2274729ba2a ]---
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.663029]
audit: type=1300 audit(1534170895.379:117649235): arch=c000003e syscall=59
success=yes exit=0 a0=271cce0 a1=2741dd0 a2=26dc3b0 a3=7ffdd7f1ea80 items=2
ppid=4556 pid=2407
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.682741]
audit: type=1309 audit(1534170895.379:117649235): argc=2 a0="date"
a1="+%s"
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.688585]
audit: type=1307 audit(1534170895.379:117649235): cwd="/"
Aug 13 14:34:55 packer_default-10-180-21-24 abrt-dump-oops: Reported 1
kernel oopses to Abrt