Hi, With 4.11.6 (that has been uploaded in debian unstable) I get a lot of messages in dmesg like [100052.120468] audit: audit_lost=66041 audit_rate_limit=0 audit_backlog_limit=8192 [100052.120470] audit: kauditd hold queue overflow And it also seems that the messages are not stored in auditd logs anymore. https://git.kernel.org/linus/264d509637d95f9404e52ced5003ad352e0f6a26 seems to be included in this release An idea?

Le 10/07/17 à 18:00, Paul Moore a écrit : > On Mon, Jul 10, 2017 at 10:59 AM, Laurent Bigonville <bigon(a)debian.org&gt; > wrote: >> >> Hi, >> >> With 4.11.6 (that has been uploaded in debian unstable) I get a lot of >> messages in dmesg like >> >> [100052.120468] audit: audit_lost=66041 audit_rate_limit=0 >> audit_backlog_limit=8192 >> [100052.120470] audit: kauditd hold queue overflow >> >> And it also seems that the messages are not stored in auditd logs >> anymore. >> >> https://git.kernel.org/linus/264d509637d95f9404e52ced5003ad352e0f6a26 >> seems >> to be included in this release >> >> An idea? > > I'm going to assume that your backlog limit is set to a sane value for > your system's configuration, so that leaves me with two commits that > may be of interest: > > * 1df30f8264b8 ("audit: fix the RCU locking for the auditd_connection > structure") > > This was a manual backport of a v4.12 patch to v4.11, looking now, I > see it should be in +v4.11.5 so that probably isn't your problem ... > > * c81be52a3ac0 ("audit: fix a race condition with the auditd tracking > code") > > This patch is relatively new and was just sent up to Linus during the > next merge window; it's a race condition fix so reproducing it can be > tricky, although it may be easily reproducible on your system at the > moment (luck you!). If you aren't in a position to apply the patch, > the workaround is rather simple: restart auditd. > > If none of the above works, let me know, but I strongly suspect you're > tripping over the race condition fixed in that last patch. > I didn't test the patch yet, but I restarted the auditd daemon 2 times and after that the queue has been flushed and I got all the message since this noon in the audit logs.

It's auditd 2.7.7 if that matters.

Le 11/07/17 à 00:23, Paul Moore a écrit : > On Mon, Jul 10, 2017 at 4:01 PM, Laurent Bigonville <bigon(a)debian.org&gt;

>> Le 10/07/17 à 18:00, Paul Moore a écrit : >>> On Mon, Jul 10, 2017 at 10:59 AM, Laurent Bigonville <bigon(a)debian.org&gt; >>> >>> wrote: >>>> Hi, >>>> >>>> With 4.11.6 (that has been uploaded in debian unstable) I get a lot of >>>> messages in dmesg like >>>> >>>> [100052.120468] audit: audit_lost=66041 audit_rate_limit=0 >>>> audit_backlog_limit=8192 >>>> [100052.120470] audit: kauditd hold queue overflow >>>> >>>> And it also seems that the messages are not stored in auditd logs >>>> anymore. >>>> >>>> https://git.kernel.org/linus/264d509637d95f9404e52ced5003ad352e0f6a26 >>>> seems >>>> to be included in this release >>>> >>>> An idea? >>> >>> 7 >>> I'm going to assume that your backlog limit is set to a sane value for >>> your system's configuration, so that leaves me with two commits that >>> may be of interest: >>> >>> * 1df30f8264b8 ("audit: fix the RCU locking for the auditd_connection >>> structure") >>> >>> This was a manual backport of a v4.12 patch to v4.11, looking now, I >>> see it should be in +v4.11.5 so that probably isn't your problem ... >>> >>> * c81be52a3ac0 ("audit: fix a race condition with the auditd tracking >>> code") >>> >>> This patch is relatively new and was just sent up to Linus during the >>> next merge window; it's a race condition fix so reproducing it can be >>> tricky, although it may be easily reproducible on your system at the >>> moment (luck you!). If you aren't in a position to apply the patch, >>> the workaround is rather simple: restart auditd. >>> >>> If none of the above works, let me know, but I strongly suspect you're >>> tripping over the race condition fixed in that last patch. >> >> I didn't test the patch yet, but I restarted the auditd daemon 2 times >> and >> after that the queue has been flushed and I got all the message since >> this >> noon in the audit logs. > > That sounds right; I'm guessing the patch above should be a more permanent > fix. The patch should be applied in 4.13-rc7 right? It seems to fix the main issue (all the audit messages being logged in dmesg) but I can still see from time to time the following message: [ 14.747565] audit: audit_lost=59 audit_rate_limit=0 audit_backlog_limit=64 [ 14.747566] audit: kauditd hold queue overflow

Le 09/09/17 à 16:22, Steve Grubb a écrit : > On Saturday, September 9, 2017 6:02:02 AM EDT Laurent Bigonville wrote: >> Le 11/07/17 à 00:23, Paul Moore a écrit : >>> On Mon, Jul 10, 2017 at 4:01 PM, Laurent Bigonville <bigon(a)debian.org&gt; > wrote: >>>> Le 10/07/17 à 18:00, Paul Moore a écrit : >>>>> On Mon, Jul 10, 2017 at 10:59 AM, Laurent Bigonville >>>>> <bigon(a)debian.org&gt; wrote: >>>>>> Hi, >>>>>> >>>>>> With 4.11.6 (that has been uploaded in debian unstable) I get a lot >>>>>> of messages in dmesg like >>>>>> >>>>>> [100052.120468] audit: audit_lost=66041 audit_rate_limit=0 >>>>>> audit_backlog_limit=8192 >>>>>> [100052.120470] audit: kauditd hold queue overflow >>>>>> >>>>>> And it also seems that the messages are not stored in auditd logs >>>>>> anymore. >>>>>> >>>>>> https://git.kernel.org/linus/264d509637d95f9404e52ced5003ad352e0f6a26 >>>>>> seems to be included in this release >>>>>> >>>>>> Any idea? >>>>> >>>>> I'm going to assume that your backlog limit is set to a sane value for >>>>> your system's configuration, so that leaves me with two commits that >>>>> may be of interest: >>>>> >>>>> * 1df30f8264b8 ("audit: fix the RCU locking for the auditd_connection >>>>> structure") >>>>> >>>>> This was a manual backport of a v4.12 patch to v4.11, looking now, I >>>>> see it should be in +v4.11.5 so that probably isn't your problem ... >>>>> >>>>> * c81be52a3ac0 ("audit: fix a race condition with the auditd tracking >>>>> code") >>>>> >>>>> This patch is relatively new and was just sent up to Linus during the >>>>> next merge window; it's a race condition fix so reproducing it can be >>>>> tricky, although it may be easily reproducible on your system at the >>>>> moment (luck you!). If you aren't in a position to apply the patch, >>>>> the workaround is rather simple: restart auditd. >>>>> >>>>> If none of the above works, let me know, but I strongly suspect you're >>>>> tripping over the race condition fixed in that last patch. >>>> >>>> I didn't test the patch yet, but I restarted the auditd daemon 2 times >>>> and after that the queue has been flushed and I got all the message since >>>> this noon in the audit logs. >>> >>> That sounds right; I'm guessing the patch above should be a more >>> permanent fix. >> >> The patch should be applied in 4.13-rc7 right? >> >> It seems to fix the main issue (all the audit messages being logged in >> dmesg) but I can still see from time to time the following message: >> >> [ 14.747565] audit: audit_lost=59 audit_rate_limit=0 >> audit_backlog_limit=64 [ 14.747566] audit: kauditd hold queue overflow > > That is a very low backlog_limit. Is this during boot? Yes that's during boot (64 is the kernel's default) In my audit.rules I have '-b 8192'