I posted initial change that allows auditd to log BPF program load/unload events, it's in here: https://github.com/linux-audit/audit-userspace/pull/104

We tried to push pure AUDIT interface for BPF program notification, but it was denied, the discussion is in here: https://marc.info/?t=153866123200003&r=1&w=2

The outcome of the discussion was to use perf event interface for BPF notification and use it in some deamon.. audit was our first choice. thoughts?

On Fri, Aug 09, 2019 at 01:45:21PM -0400, Steve Grubb wrote: > Hello, > > On Friday, August 9, 2019 10:18:31 AM EDT Jiri Olsa wrote: > > I posted initial change that allows auditd to log BPF program > > > > load/unload events, it's in here: > > https://github.com/linux-audit/audit-userspace/pull/104 > > Thanks for the patch...but we probably should have talked a bit more > before undertaking this effort. We normally do not audit from user space > what happens in the kernel. Doing this can be racy and it keeps auditd > from doing the one job it has - which is to grab events and record them > to disk and send them out the realtime interface. > > > We tried to push pure AUDIT interface for BPF program notification, > > > > but it was denied, the discussion is in here: > > https://marc.info/?t=153866123200003&r=1&w=2 > > Hmm. The email I remember was here: > https://www.redhat.com/archives/linux-audit/2018-October/msg00053.html > > and was only 2 emails long with no answer to my question. :-) oops, sry about that, your question was: > I'm not sure exactly what the issue is. You can audit for specific > syscall > and argument. So, if you want to see loads, then you can make a rule > like: > > -a always,exit -F arch=b64 -S bpf -F a0=5 The problem with above for us is that we also: - need to log also other properties of the BPF program, which are not visible from BPF syscall arguments, like its ID, JIT status

or license info

- need to see BPF program UNLOAD, which is not done via syscall, so those would be unvisible at all

> > The outcome of the discussion was to use perf event interface > > for BPF notification and use it in some deamon.. audit was our > > first choice. > > > > thoughts? > > I'd like to understand what the basic problem is that needs to be solved. we need a way for administrators to see the history of loaded BPF programs, to help investigating issues related to BPF.. and the only BPF interface for this data is through perf ring buffer

On Mon, Aug 12, 2019 at 09:49:43AM -0400, Steve Grubb wrote: > On Monday, August 12, 2019 3:59:22 AM EDT Jiri Olsa wrote: > > On Fri, Aug 09, 2019 at 01:45:21PM -0400, Steve Grubb wrote: > > > Hello, > > > > > > On Friday, August 9, 2019 10:18:31 AM EDT Jiri Olsa wrote: > > > > I posted initial change that allows auditd to log BPF program > > > > > > > > load/unload events, it's in here: > > > > https://github.com/linux-audit/audit-userspace/pull/104 > > > > > > Thanks for the patch...but we probably should have talked a bit more > > > before undertaking this effort. We normally do not audit from user space > > > what happens in the kernel. Doing this can be racy and it keeps auditd > > > from doing the one job it has - which is to grab events and record them > > > to disk and send them out the realtime interface. > > > > > > > We tried to push pure AUDIT interface for BPF program notification, > > > > > > > > but it was denied, the discussion is in here: > > > > https://marc.info/?t=153866123200003&r=1&w=2 > > > > > > Hmm. The email I remember was here: > > > https://www.redhat.com/archives/linux-audit/2018-October/msg00053.html > > > > > > and was only 2 emails long with no answer to my question. :-) > > > > oops, sry about that, your question was: > > > I'm not sure exactly what the issue is. You can audit for specific > > > syscall > > > and argument. So, if you want to see loads, then you can make a rule > > > like: > > > > > > -a always,exit -F arch=b64 -S bpf -F a0=5 > > > > The problem with above for us is that we also: > > > > - need to log also other properties of the BPF program, > > which are not visible from BPF syscall arguments, like > > its ID, JIT status > > The way this is normally done is to add a supplemental record. For example, > when auditing the open syscall, we also get CWD & PATH supplemental records. > When auditing connect, we get a SOCKADDR supplemental record. We have > requirements around selective audit whereby the admin is in control of what > is selected for audit via audit rules. So, what one could do is set a rule > for the bpf syscall and then when it triggers, we get these other records > added to the bpf syscall event. right, that was the initial plan, but BPF guys wanted just single notification system without specific hooks for audit, so we ended up with perf specific interface > > or license info > > This ^^ is not a security issue. > > > > - need to see BPF program UNLOAD, which is not done > > via syscall, so those would be unvisible at all > > Is there a place in the kernel where this happens? I could see abnormal > termination being something we might want. Does the program go through > something like an exit syscall internally? it's happening in here (kernel/bpf/syscall.c): bpf_prog_put __bpf_prog_put { if (atomic_dec_and_test(&prog->aux->refcnt)) { perf_event_bpf_event(prog, PERF_BPF_EVENT_PROG_UNLOAD, 0); ... } BPF program is released when it drops the reference count, which is normally when its file descriptor is closed. However it might get pinned and stay alive even when the initial file descriptor is closed.. and then there's the networking world, which might have some other specific ways.. but it all ends up in bpf_prog_put and zero reference count. > > > > The outcome of the discussion was to use perf event interface > > > > for BPF notification and use it in some deamon.. audit was our > > > > first choice. > > > > > > > > thoughts? > > > > > > I'd like to understand what the basic problem is that needs to be solved. > > > > we need a way for administrators to see the history of loaded BPF > > programs, to help investigating issues related to BPF.. and the > > only BPF interface for this data is through perf ring buffer > > That is really not the audit way. Let's keep talking to see where this ends > up. Would you see some other auditing daemon/app in place for this kind of data? thanks, jirka

cc-ing Petr Matousek jirka On Wed, Aug 14, 2019 at 09:33:34AM +0200, Jiri Olsa wrote: > hi, > Adding Vlad Dronov to the loop, because he asked > about this functionality some time ago. > > Vlad, the full thread can be found in here: > https://www.redhat.com/archives/linux-audit/2019-August/msg00004.html > > Any thoughts on this? > > thanks, > jirka > > On Mon, Aug 12, 2019 at 04:33:10PM +0200, Jiri Olsa wrote: > > On Mon, Aug 12, 2019 at 09:49:43AM -0400, Steve Grubb wrote: > > > On Monday, August 12, 2019 3:59:22 AM EDT Jiri Olsa wrote: > > > > On Fri, Aug 09, 2019 at 01:45:21PM -0400, Steve Grubb wrote: > > > > > Hello, > > > > > > > > > > On Friday, August 9, 2019 10:18:31 AM EDT Jiri Olsa wrote: > > > > > > I posted initial change that allows auditd to log BPF program > > > > > > > > > > > > load/unload events, it's in here: > > > > > > https://github.com/linux-audit/audit-userspace/pull/104 > > > > > > > > > > Thanks for the patch...but we probably should have talked a bit more > > > > > before undertaking this effort. We normally do not audit from user space > > > > > what happens in the kernel. Doing this can be racy and it keeps auditd > > > > > from doing the one job it has - which is to grab events and record them > > > > > to disk and send them out the realtime interface. > > > > > > > > > > > We tried to push pure AUDIT interface for BPF program notification, > > > > > > > > > > > > but it was denied, the discussion is in here: > > > > > > https://marc.info/?t=153866123200003&r=1&w=2 > > > > > > > > > > Hmm. The email I remember was here: > > > > > https://www.redhat.com/archives/linux-audit/2018-October/msg00053.html > > > > > > > > > > and was only 2 emails long with no answer to my question. :-) > > > > > > > > oops, sry about that, your question was: > > > > > I'm not sure exactly what the issue is. You can audit for specific > > > > > syscall > > > > > and argument. So, if you want to see loads, then you can make a rule > > > > > like: > > > > > > > > > > -a always,exit -F arch=b64 -S bpf -F a0=5 > > > > > > > > The problem with above for us is that we also: > > > > > > > > - need to log also other properties of the BPF program, > > > > which are not visible from BPF syscall arguments, like > > > > its ID, JIT status > > > > > > The way this is normally done is to add a supplemental record. For example, > > > when auditing the open syscall, we also get CWD & PATH supplemental records. > > > When auditing connect, we get a SOCKADDR supplemental record. We have > > > requirements around selective audit whereby the admin is in control of what > > > is selected for audit via audit rules. So, what one could do is set a rule > > > for the bpf syscall and then when it triggers, we get these other records > > > added to the bpf syscall event. > > > > right, that was the initial plan, but BPF guys wanted just > > single notification system without specific hooks for audit, > > so we ended up with perf specific interface > > > > > > or license info > > > > > > This ^^ is not a security issue. > > > > > > > > > > - need to see BPF program UNLOAD, which is not done > > > > via syscall, so those would be unvisible at all > > > > > > Is there a place in the kernel where this happens? I could see abnormal > > > termination being something we might want. Does the program go through > > > something like an exit syscall internally? > > > > it's happening in here (kernel/bpf/syscall.c): > > > > bpf_prog_put > > __bpf_prog_put > > { > > if (atomic_dec_and_test(&prog->aux->refcnt)) { > > perf_event_bpf_event(prog, PERF_BPF_EVENT_PROG_UNLOAD, 0); > > ... > > } > > > > BPF program is released when it drops the reference count, > > which is normally when its file descriptor is closed. > > > > However it might get pinned and stay alive even when the initial > > file descriptor is closed.. and then there's the networking world, > > which might have some other specific ways.. but it all ends up > > in bpf_prog_put and zero reference count. > > > > > > > > The outcome of the discussion was to use perf event interface > > > > > > for BPF notification and use it in some deamon.. audit was our > > > > > > first choice. > > > > > > > > > > > > thoughts? > > > > > > > > > > I'd like to understand what the basic problem is that needs to be solved. > > > > > > > > we need a way for administrators to see the history of loaded BPF > > > > programs, to help investigating issues related to BPF.. and the > > > > only BPF interface for this data is through perf ring buffer > > > > > > That is really not the audit way. Let's keep talking to see where this ends > > > up. > > > > Would you see some other auditing daemon/app in place for this kind of data? > > > > thanks, > > jirka

From: "Jiri Benc" <jbenc(a)redhat.com&gt; To: "Jiri Olsa" <jolsa(a)redhat.com&gt; Cc: "Steve Grubb" <sgrubb(a)redhat.com&gt;, linux-audit(a)redhat.com, "Stanislav Kozina" <skozina(a)redhat.com&gt;, "Yauheni Kaliuta" <yauheni.kaliuta(a)redhat.com&gt;, "Toke Høiland-Jørgensen" <toke(a)redhat.com&gt;, "Arnaldo Carvalho de Melo" <acme(a)redhat.com&gt;, "Jesper Dangaard Brouer" <brouer(a)redhat.com&gt;, "Vlad Dronov" <vdronov(a)redhat.com&gt;, "Petr Matousek" <pmatouse(a)redhat.com&gt;, "Rashid Khan" <rkhan(a)redhat.com&gt; Sent: Monday, November 4, 2019 2:05:18 PM Subject: Re: [RFC] audit support for BPF notification Seems there have been no reply to this... Jiri, what is the current status? Vlad, what is the Product Security's view on this? Is the audit support for bpf programs loading/unloading a requirement for full support of eBPF (as opposed to tech preview)? Thanks, Jiri On Tue, 20 Aug 2019 15:54:53 +0200, Jiri Olsa wrote: > cc-ing Petr Matousek > > jirka > > On Wed, Aug 14, 2019 at 09:33:34AM +0200, Jiri Olsa wrote: > > hi, > > Adding Vlad Dronov to the loop, because he asked > > about this functionality some time ago. > > > > Vlad, the full thread can be found in here: > > https://www.redhat.com/archives/linux-audit/2019-August/msg00004.html > > > > Any thoughts on this? > > > > thanks, > > jirka > > > > On Mon, Aug 12, 2019 at 04:33:10PM +0200, Jiri Olsa wrote: > > > On Mon, Aug 12, 2019 at 09:49:43AM -0400, Steve Grubb wrote: > > > > On Monday, August 12, 2019 3:59:22 AM EDT Jiri Olsa wrote: > > > > > On Fri, Aug 09, 2019 at 01:45:21PM -0400, Steve Grubb wrote: > > > > > > Hello, > > > > > > > > > > > > On Friday, August 9, 2019 10:18:31 AM EDT Jiri Olsa wrote: > > > > > > > I posted initial change that allows auditd to log BPF program > > > > > > > > > > > > > > load/unload events, it's in here: > > > > > > > https://github.com/linux-audit/audit-userspace/pull/104 > > > > > > > > > > > > Thanks for the patch...but we probably should have talked a bit > > > > > > more > > > > > > before undertaking this effort. We normally do not audit from > > > > > > user space > > > > > > what happens in the kernel. Doing this can be racy and it keeps > > > > > > auditd > > > > > > from doing the one job it has - which is to grab events and > > > > > > record them > > > > > > to disk and send them out the realtime interface. > > > > > > > > > > > > > We tried to push pure AUDIT interface for BPF program > > > > > > > notification, > > > > > > > > > > > > > > but it was denied, the discussion is in here: > > > > > > > https://marc.info/?t=153866123200003&r=1&w=2 > > > > > > > > > > > > Hmm. The email I remember was here: > > > > > > https://www.redhat.com/archives/linux-audit/2018-October/msg00053.html > > > > > > > > > > > > and was only 2 emails long with no answer to my question. :-) > > > > > > > > > > oops, sry about that, your question was: > > > > > > I'm not sure exactly what the issue is. You can audit for > > > > > > specific > > > > > > syscall > > > > > > and argument. So, if you want to see loads, then you can make a > > > > > > rule > > > > > > like: > > > > > > > > > > > > -a always,exit -F arch=b64 -S bpf -F a0=5 > > > > > > > > > > The problem with above for us is that we also: > > > > > > > > > > - need to log also other properties of the BPF program, > > > > > which are not visible from BPF syscall arguments, like > > > > > its ID, JIT status > > > > > > > > The way this is normally done is to add a supplemental record. For > > > > example, > > > > when auditing the open syscall, we also get CWD & PATH supplemental > > > > records. > > > > When auditing connect, we get a SOCKADDR supplemental record. We have > > > > requirements around selective audit whereby the admin is in control > > > > of what > > > > is selected for audit via audit rules. So, what one could do is set a > > > > rule > > > > for the bpf syscall and then when it triggers, we get these other > > > > records > > > > added to the bpf syscall event. > > > > > > right, that was the initial plan, but BPF guys wanted just > > > single notification system without specific hooks for audit, > > > so we ended up with perf specific interface > > > > > > > > or license info > > > > > > > > This ^^ is not a security issue. > > > > > > > > > > > > > - need to see BPF program UNLOAD, which is not done > > > > > via syscall, so those would be unvisible at all > > > > > > > > Is there a place in the kernel where this happens? I could see > > > > abnormal > > > > termination being something we might want. Does the program go > > > > through > > > > something like an exit syscall internally? > > > > > > it's happening in here (kernel/bpf/syscall.c): > > > > > > bpf_prog_put > > > __bpf_prog_put > > > { > > > if (atomic_dec_and_test(&prog->aux->refcnt)) { > > > perf_event_bpf_event(prog, PERF_BPF_EVENT_PROG_UNLOAD, 0); > > > ... > > > } > > > > > > BPF program is released when it drops the reference count, > > > which is normally when its file descriptor is closed. > > > > > > However it might get pinned and stay alive even when the initial > > > file descriptor is closed.. and then there's the networking world, > > > which might have some other specific ways.. but it all ends up > > > in bpf_prog_put and zero reference count. > > > > > > > > > > The outcome of the discussion was to use perf event interface > > > > > > > for BPF notification and use it in some deamon.. audit was our > > > > > > > first choice. > > > > > > > > > > > > > > thoughts? > > > > > > > > > > > > I'd like to understand what the basic problem is that needs to be > > > > > > solved. > > > > > > > > > > we need a way for administrators to see the history of loaded BPF > > > > > programs, to help investigating issues related to BPF.. and the > > > > > only BPF interface for this data is through perf ring buffer > > > > > > > > That is really not the audit way. Let's keep talking to see where > > > > this ends > > > > up. > > > > > > Would you see some other auditing daemon/app in place for this kind of > > > data? > > > > > > thanks, > > > jirka

Seems there have been no reply to this... Jiri, what is the current status? Vlad, what is the Product Security's view on this? Is the audit support for bpf programs loading/unloading a requirement for full support of eBPF (as opposed to tech preview)?