12:49 p.m.
Hi, I have started using 'auditd', mostly to monitor various directories
where packages get installed to check for changes in their contents,
with rules like:
-w /bin -p wa -k pkg-s
-w /boot -p wa -k pkg-s
-w /etc -p wa -k pkg-s
-w /lib -p wa -k pkg-s
-w /lib32 -p wa -k pkg-s
-w /lib64 -p wa -k pkg-s
-w /opt -p wa -k pkg-s
-w /usr -p wa -k pkg-s
-w /fs/sozan/loc -p wa -k pkg-l
-w /fs/sozan/loc32-el5 -p wa -k pkg-l
-w /fs/sozan/loc64-u12 -p wa -k pkg-l
-w /fs/sozan/com -p wa -k pkg-l
-w /fs/sozan/com32-el5 -p wa -k pkg-l
-w /fs/sozan/com64-u12 -p wa -k pkg-l
After setting them, I can verify that for example creating, updating and
deleting a file in '/boot' or '/opt' gets reported.
Wheat then happens is that even if I set 'auditctl -e 2' some of the
rules disappear, usually at around the same time as 'cron.daily' scripts
run, and some more disappear later. This usually seems to relate to
times where there some significant IO activity ('mlocate' scan, backup),
but this is a guess.
For example:
time->Thu Apr 17 07:58:44 2014
type=CONFIG_CHANGE msg=audit(1397717924.255:37148): op="remove rule"
dir="/boot" key="pkg-s" list=4 res=1
time->Thu Apr 17 07:59:04 2014
type=CONFIG_CHANGE msg=audit(1397717944.762:37151): op="remove rule"
dir="/opt" key="pkg-s" list=4 res=1
time->Thu Apr 17 10:01:02 2014
type=CONFIG_CHANGE msg=audit(1397725262.301:37157): op="remove rule"
dir="/fs/sozan/loc64-u12" key="pkg-l" list=4 res=1
time->Thu Apr 17 10:01:02 2014
type=CONFIG_CHANGE msg=audit(1397725262.301:37156): op="remove rule"
dir="/fs/sozan/loc32-el5" key="pkg-l" list=4 res=1
There is no equivalent line in 'dmesg'.
I understand that the 'audit' kernel modules may remove rules if they
refer to invalid paths, but all the relevant directories do exist, as
for example '/boot' and '/opt' are the standard usual directories in the
"root" tree itself:
$ ls -ldn /boot /opt /fs/sozan/loc64-u12 /fs/sozan/loc32-el5
drwxr-xr-x 3 0 0 4096 Apr 21 07:22 /boot
drwxrwsr-x 7 1 1 61 Jul 30 2011 /fs/sozan/loc32-el5
drwxrwsr-x 5 1 1 39 Oct 4 2011 /fs/sozan/loc64-u12
drwxr-xr-x 7 0 0 4096 Apr 20 14:52 /opt
$ df /boot/. /opt/. /fs/sozan/loc64-u12/. /fs/sozan/loc32-el5/.
Filesystem 1M-blocks Used Available Use% Mounted on
/dev/sda3 24815 16853 4106 81% /
/dev/sda3 24815 16853 4106 81% /
/dev/sda6 90048 82355 7694 92% /fs/sozan
/dev/sda6 90048 82355 7694 92% /fs/sozan
This is happening on two similarly configured Ubuntu 12.04 systems with
both 3.2 and 3.11 Ubuntu "official" kernels. I also have an AppArmor
configuration which seem to trigger bugs in AppArmor, but all the
relative profiles are essentially unchanged.
Eventually around almost all of the rules I have set "disappear". For
example of all these rules:
LIST_RULES: exit,always dir=/fs/sozan/search (0x10) perm=r key=pkg-r
LIST_RULES: exit,always dir=/fs/sozan/mlocate (0x11) perm=r key=pkg-r
....
LIST_RULES: exit,always dir=/bin (0x4) perm=wa key=pkg-s
LIST_RULES: exit,always dir=/boot (0x5) perm=wa key=pkg-s
LIST_RULES: exit,always dir=/etc (0x4) perm=wa key=pkg-s
LIST_RULES: exit,always dir=/lib (0x4) perm=wa key=pkg-s
LIST_RULES: exit,always dir=/lib32 (0x6) perm=wa key=pkg-s
LIST_RULES: exit,always dir=/lib64 (0x6) perm=wa key=pkg-s
LIST_RULES: exit,always dir=/opt (0x4) perm=wa key=pkg-s
LIST_RULES: exit,always dir=/usr (0x4) perm=wa key=pkg-s
LIST_RULES: exit,always dir=/fs/sozan/loc (0xd) perm=wa key=pkg-l
LIST_RULES: exit,always dir=/fs/sozan/loc32-el5 (0x13) perm=wa key=pkg-l
LIST_RULES: exit,always dir=/fs/sozan/loc64-u12 (0x13) perm=wa key=pkg-l
LIST_RULES: exit,always dir=/fs/sozan/com (0xd) perm=wa key=pkg-l
LIST_RULES: exit,always dir=/fs/sozan/com32-el5 (0x13) perm=wa key=pkg-l
LIST_RULES: exit,always dir=/fs/sozan/com64-u12 (0x13) perm=wa key=pkg-l
Only the first two have not "disappeared" on one of the systems.
This is rather peculiar, please let me know if it is a configuration
error, an issue, and any fixes or workaround if available (other than
running 'auditctl -R /etc/audit/audit.rules' every few minutes via CRON).
1:28 p.m.
On Monday, April 21, 2014 06:49:24 PM Peter Grandi wrote:
Hi, I have started using 'auditd', mostly to monitor various
directories
where packages get installed to check for changes in their contents,
with rules like:
-w /bin -p wa -k pkg-s
-w /boot -p wa -k pkg-s
-w /etc -p wa -k pkg-s
-w /lib -p wa -k pkg-s
-w /lib32 -p wa -k pkg-s
-w /lib64 -p wa -k pkg-s
-w /opt -p wa -k pkg-s
-w /usr -p wa -k pkg-s
-w /fs/sozan/loc -p wa -k pkg-l
-w /fs/sozan/loc32-el5 -p wa -k pkg-l
-w /fs/sozan/loc64-u12 -p wa -k pkg-l
-w /fs/sozan/com -p wa -k pkg-l
-w /fs/sozan/com32-el5 -p wa -k pkg-l
-w /fs/sozan/com64-u12 -p wa -k pkg-l
After setting them, I can verify that for example creating, updating and
deleting a file in '/boot' or '/opt' gets reported.
Wheat then happens is that even if I set 'auditctl -e 2' some of the
rules disappear, usually at around the same time as 'cron.daily' scripts
run, and some more disappear later. This usually seems to relate to
times where there some significant IO activity ('mlocate' scan, backup),
but this is a guess.
For example:
time->Thu Apr 17 07:58:44 2014
type=CONFIG_CHANGE msg=audit(1397717924.255:37148): op="remove rule"
dir="/boot" key="pkg-s" list=4 res=1 time->Thu Apr 17 07:59:04
2014
type=CONFIG_CHANGE msg=audit(1397717944.762:37151): op="remove rule"
dir="/opt" key="pkg-s" list=4 res=1 time->Thu Apr 17 10:01:02
2014
type=CONFIG_CHANGE msg=audit(1397725262.301:37157): op="remove rule"
dir="/fs/sozan/loc64-u12" key="pkg-l" list=4 res=1 time->Thu Apr
17
10:01:02 2014
type=CONFIG_CHANGE msg=audit(1397725262.301:37156): op="remove rule"
dir="/fs/sozan/loc32-el5" key="pkg-l" list=4 res=1
There is no equivalent line in 'dmesg'.
I understand that the 'audit' kernel modules may remove rules if they
refer to invalid paths, but all the relevant directories do exist, as
for example '/boot' and '/opt' are the standard usual directories in the
"root" tree itself:
What happens is that the text path that you put in a watch is a human
convenience. The kernel doesn't understand strings, it understands numbers. It
changes the path into device and inode information. This is, technically, what
your watch is on. If the inode disappears, then the rule is ejected. Rules can
survive across renames but not deletions.
I don't know what is managing your system, but its probably deleting paths.
You might do a ls -i to get the inode number of the directories and then when
you notice the rules being ejected, re-run the ls -i and see if the inodes
have changed.
-Steve
$ ls -ldn /boot /opt /fs/sozan/loc64-u12 /fs/sozan/loc32-el5
drwxr-xr-x 3 0 0 4096 Apr 21 07:22 /boot
drwxrwsr-x 7 1 1 61 Jul 30 2011 /fs/sozan/loc32-el5
drwxrwsr-x 5 1 1 39 Oct 4 2011 /fs/sozan/loc64-u12
drwxr-xr-x 7 0 0 4096 Apr 20 14:52 /opt
$ df /boot/. /opt/. /fs/sozan/loc64-u12/. /fs/sozan/loc32-el5/.
Filesystem 1M-blocks Used Available Use% Mounted on
/dev/sda3 24815 16853 4106 81% /
/dev/sda3 24815 16853 4106 81% /
/dev/sda6 90048 82355 7694 92% /fs/sozan
/dev/sda6 90048 82355 7694 92% /fs/sozan
This is happening on two similarly configured Ubuntu 12.04 systems with
both 3.2 and 3.11 Ubuntu "official" kernels. I also have an AppArmor
configuration which seem to trigger bugs in AppArmor, but all the
relative profiles are essentially unchanged.
Eventually around almost all of the rules I have set "disappear". For
example of all these rules:
LIST_RULES: exit,always dir=/fs/sozan/search (0x10) perm=r key=pkg-r
LIST_RULES: exit,always dir=/fs/sozan/mlocate (0x11) perm=r key=pkg-r
....
LIST_RULES: exit,always dir=/bin (0x4) perm=wa key=pkg-s
LIST_RULES: exit,always dir=/boot (0x5) perm=wa key=pkg-s
LIST_RULES: exit,always dir=/etc (0x4) perm=wa key=pkg-s
LIST_RULES: exit,always dir=/lib (0x4) perm=wa key=pkg-s
LIST_RULES: exit,always dir=/lib32 (0x6) perm=wa key=pkg-s
LIST_RULES: exit,always dir=/lib64 (0x6) perm=wa key=pkg-s
LIST_RULES: exit,always dir=/opt (0x4) perm=wa key=pkg-s
LIST_RULES: exit,always dir=/usr (0x4) perm=wa key=pkg-s
LIST_RULES: exit,always dir=/fs/sozan/loc (0xd) perm=wa key=pkg-l
LIST_RULES: exit,always dir=/fs/sozan/loc32-el5 (0x13) perm=wa key=pkg-l
LIST_RULES: exit,always dir=/fs/sozan/loc64-u12 (0x13) perm=wa key=pkg-l
LIST_RULES: exit,always dir=/fs/sozan/com (0xd) perm=wa key=pkg-l
LIST_RULES: exit,always dir=/fs/sozan/com32-el5 (0x13) perm=wa key=pkg-l
LIST_RULES: exit,always dir=/fs/sozan/com64-u12 (0x13) perm=wa key=pkg-l
Only the first two have not "disappeared" on one of the systems.
This is rather peculiar, please let me know if it is a configuration
error, an issue, and any fixes or workaround if available (other than
running 'auditctl -R /etc/audit/audit.rules' every few minutes via CRON).
--
Linux-audit mailing list
Linux-audit(a)redhat.com
https://www.redhat.com/mailman/listinfo/linux-audit
1:35 p.m.
On Apr 21, 2014, at 11:28 AM, Steve Grubb <sgrubb(a)redhat.com> wrote:
What happens is that the text path that you put in a watch is a human
convenience. The kernel doesn't understand strings, it understands numbers. It
changes the path into device and inode information.
Cool. So I am guessing the rule works even if someone creates a hard link to the same
watched path and access files through that other path?
Todd
2:03 p.m.
On Mon, 2014-04-21 at 11:35 -0700, lists_todd(a)mac.com wrote:
On Apr 21, 2014, at 11:28 AM, Steve Grubb <sgrubb(a)redhat.com> wrote:
> What happens is that the text path that you put in a watch is a
> human
> convenience. The kernel doesn't understand strings, it understands
> numbers. It
> changes the path into device and inode information.
Cool. So I am guessing the rule works even if someone creates a hard
link to the same watched path and access files through that other
path?
As I remember, and it's been a long time, watches should survive even if
the object being watched is deleted and recreated. I seemed to remember
it was only if the parent directory is deleted that rules get evicted.
So that doesn't explain it for /boot! Pretty darn hard to delete /!
But it could easily make sense for your other areas being watched...
But yes, if you watch /etc/shadow and someone accesses that inode
through another hard link, you will get audit records...
3:49 p.m.
[ ... ]
> For example:
> time->Thu Apr 17 07:58:44 2014
> type=CONFIG_CHANGE msg=audit(1397717924.255:37148): op="remove rule"
dir="/boot" key="pkg-s" list=4 res=1
> time->Thu Apr 17 07:59:04 2014
> type=CONFIG_CHANGE msg=audit(1397717944.762:37151): op="remove rule"
dir="/opt" key="pkg-s" list=4 res=1
> time->Thu Apr 17 10:01:02 2014
> type=CONFIG_CHANGE msg=audit(1397725262.301:37157):
op="remove rule" dir="/fs/sozan/loc64-u12" key="pkg-l"
list=4 res=1
> time->Thu Apr 17 10:01:02 2014
> type=CONFIG_CHANGE msg=audit(1397725262.301:37156): op="remove rule"
dir="/fs/sozan/loc32-el5" key="pkg-l" list=4 res=1
[ ... ] device and inode information. This is, technically,
what your watch is on. If the inode disappears, then the rule
is ejected. Rules can survive across renames but not
deletions.
I don't know what is managing your system, but its probably
deleting paths.
I am the sole user (as far as I know...) of both systems, and I
am pretty sure I was asleep at least at some of the reported
times, and I can't imagine any of the system scripts deleting
and recreating '/boot' and '/opt', for example. Also I checked
the 'm' times of '/' and '/fs/sozan' and they are a few weeks
old. None of the "disappeared" paths seems to have been modified
in any way.
BTW this has happened also on a far smaller scale on a Debian
7/Wheezy system. Again a system where I am the sysadm and only
user, and it seems roughly at the same time as a treewalk.
The vague impression I have in both cases is that there is some
reason why under high load 'audit' just loses or deletes watches.
Note that this was not under high _logging_ load, because the
watches for '/opt' and '/boot' are to log in case of writing or
attribute changes, and the treewalk is entirely (on one side)
read-only. Indeed I see not even a trace of logging about those
accesses.
Anyhow, I have now recorded the inos of the watched directories,
and I shall also run 'inotifywait -m /' to catch if possible any
changes in '/opt' and '/boot'.
3:53 p.m.
> I don't know what is managing your system, but its probably
> deleting paths.
I am the sole user (as far as I know...) of both systems, [
... ] None of the "disappeared" paths seems to have been
modified in any way. [ ... ] Anyhow, I have now recorded the
inos of the watched directories, and I shall also run
'inotifywait -m /' to catch if possible any changes in '/opt'
and '/boot'.
I have done this and this morning during 'mlocate' treewalking
some of the usual paths disappeared; I verified the inos and
the 'inotify' output and no inos changed nor any of the watched
directories changed.
Since the list of directories that *do not* disappear is
usually:
LIST_RULES: exit,always dir=/bin (0x4) perm=wa key=pkg-s
LIST_RULES: exit,always dir=/etc (0x4) perm=wa key=pkg-s
LIST_RULES: exit,always dir=/lib (0x4) perm=wa key=pkg-s
LIST_RULES: exit,always dir=/usr (0x4) perm=wa key=pkg-s
LIST_RULES: exit,always dir=/fs/sozan/loc (0xd) perm=wa key=pkg-l
LIST_RULES: exit,always dir=/fs/sozan/com (0xd) perm=wa key=pkg-l
and those that disappear tend to be far less frequently used
directories like '/boot', '/opt', '/lib32'. Rereading this:
> [ ... ] device and inode information. This is, technically,
> what your watch is on. If the inode disappears, then the rule
> is ejected. Rules can survive across renames but not deletions.
it appears that I misread earlier: this says "inode", not
"inum". Also it says "inode disappears", which is not
necessarily always because the on-disk inode is deleted.
Thus I have come up with a potential explanation:
* The 'audit' module does not identify the watched file and
directory by (device,ino) but by a pointer to an inode table
entry, a bit like a filesystem module would.
* During treewalks a lot of inodes get cached in the in-memory
inode table.
* This creates pressure on the inode tables and thus the least
used (in some sense) inodes get evicted, and this includes
those for the "disappearing directories".
* When these least used inodes are evicted the 'audit' module
sees it as if it was a removal of the inode.
If the above is the right explanation it is a pretty big deal,
because it means that a way to disable many/most 'audit' watches
on files is to create and access a lot of inodes, which is
pretty easy to do.
4:46 p.m.
On Tuesday, April 22, 2014 09:53:15 PM Peter Grandi wrote:
>> I don't know what is managing your system, but its
probably
>> deleting paths.
>
> I am the sole user (as far as I know...) of both systems, [
> ... ] None of the "disappeared" paths seems to have been
> modified in any way. [ ... ] Anyhow, I have now recorded the
> inos of the watched directories, and I shall also run
> 'inotifywait -m /' to catch if possible any changes in '/opt'
> and '/boot'.
I have done this and this morning during 'mlocate' treewalking
some of the usual paths disappeared; I verified the inos and
the 'inotify' output and no inos changed nor any of the watched
directories changed.
Since the list of directories that *do not* disappear is
usually:
LIST_RULES: exit,always dir=/bin (0x4) perm=wa key=pkg-s
LIST_RULES: exit,always dir=/etc (0x4) perm=wa key=pkg-s
LIST_RULES: exit,always dir=/lib (0x4) perm=wa key=pkg-s
LIST_RULES: exit,always dir=/usr (0x4) perm=wa key=pkg-s
LIST_RULES: exit,always dir=/fs/sozan/loc (0xd) perm=wa key=pkg-l
LIST_RULES: exit,always dir=/fs/sozan/com (0xd) perm=wa key=pkg-l
and those that disappear tend to be far less frequently used
directories like '/boot', '/opt', '/lib32'. Rereading this:
>> [ ... ] device and inode information. This is, technically,
>> what your watch is on. If the inode disappears, then the rule
>> is ejected. Rules can survive across renames but not deletions.
it appears that I misread earlier: this says "inode", not
"inum". Also it says "inode disappears", which is not
necessarily always because the on-disk inode is deleted.
Thus I have come up with a potential explanation:
* The 'audit' module does not identify the watched file and
directory by (device,ino) but by a pointer to an inode table
entry, a bit like a filesystem module would.
* During treewalks a lot of inodes get cached in the in-memory
inode table.
* This creates pressure on the inode tables and thus the least
used (in some sense) inodes get evicted, and this includes
those for the "disappearing directories".
* When these least used inodes are evicted the 'audit' module
sees it as if it was a removal of the inode.
If the above is the right explanation it is a pretty big deal,
I don't know if that is in fact what happens. But if it is, I would agree with
your conclusion.
-Steve
because it means that a way to disable many/most 'audit'
watches
on files is to create and access a lot of inodes, which is
pretty easy to do.
3:04 a.m.
[ ... ]
Thus I have come up with a potential explanation:
* The 'audit' module does not identify the watched file
and
directory by (device,ino) but by a pointer to an inode table
entry, a bit like a filesystem module would.
I had a look at the code and it seems it relies on 'inotify' and
the code does get pointers at the relevant in-memory inode
descriptors.
* During treewalks a lot of inodes get cached in the in-memory
inode table.
* This creates pressure on the inode tables and thus the least
used (in some sense) inodes get evicted, and this includes
those for the "disappearing directories".
* When these least used inodes are evicted the 'audit' module
sees it as if it was a removal of the inode.
To corroborate this I have been running:
while true
do
for D in $(< audit-names.txt)
do (cd "$D" && exec sleep 3001)&
done
sleep 3001
done
Which has the effect of marking the relevant directories as the
active current directories of each 'sleep' process, and none of
those directories were "disappeared" from the 'audit' active
rules list.
The 'inotify' code has a comment that claims:
* inode: Pinned so long as the inode is associated with a watch,
from
* inotify_add_watch() to the final put_inotify_watch().
They use 'igrab'/'iput', and 'audit_tree.c' and
'audit_watch.c'
uses them, so I wonder what is missing.
9:34 a.m.
What's the kernel in question? audit hasn't used "inotify" in a long
time. We now use "fsnotify". but in either case, the inodes aren't
supposed to be able to be kicked out of core...
On Wed, 2014-04-23 at 09:04 +0100, Peter Grandi wrote:
[ ... ]
> Thus I have come up with a potential explanation:
> * The 'audit' module does not identify the watched file and
> directory by (device,ino) but by a pointer to an inode table
> entry, a bit like a filesystem module would.
I had a look at the code and it seems it relies on 'inotify' and
the code does get pointers at the relevant in-memory inode
descriptors.
> * During treewalks a lot of inodes get cached in the in-memory
> inode table.
> * This creates pressure on the inode tables and thus the least
> used (in some sense) inodes get evicted, and this includes
> those for the "disappearing directories".
> * When these least used inodes are evicted the 'audit' module
> sees it as if it was a removal of the inode.
To corroborate this I have been running:
while true
do
for D in $(< audit-names.txt)
do (cd "$D" && exec sleep 3001)&
done
sleep 3001
done
Which has the effect of marking the relevant directories as the
active current directories of each 'sleep' process, and none of
those directories were "disappeared" from the 'audit' active
rules list.
The 'inotify' code has a comment that claims:
> * inode: Pinned so long as the inode is associated with a watch, from
> * inotify_add_watch() to the final put_inotify_watch().
They use 'igrab'/'iput', and 'audit_tree.c' and
'audit_watch.c'
uses them, so I wonder what is missing.
--
Linux-audit mailing list
Linux-audit(a)redhat.com
https://www.redhat.com/mailman/listinfo/linux-audit
3:33 p.m.
[ .... ]
What's the kernel in question?
Ubuntu 12.04's 3.2 and SteamOS 3.10.
audit hasn't used "inotify" in a long time. We now
use
"fsnotify".
Out of laziness I used 'inotify' to mean both; also at one point
I was looking at some 2.6.x sources as there seemed to be
relevant changes in some mailing list.
but in either case, the inodes aren't supposed to be able to
be kicked out of core...
But on 3 different system I have they really seem to be evicted,
and with regularity, and this does not happen if the inodes are
kept open.
From the source I have looked at, the *notify code seems to
attempt to hold on to the inodes that are watched, but perhaps
it has some hidden assumptions that the 'audit' module does not
satisfy.
10:55 a.m.
On 14/04/27, Peter Grandi wrote:
> but in either case, the inodes aren't supposed to be able
to
> be kicked out of core...
But on 3 different system I have they really seem to be evicted,
and with regularity, and this does not happen if the inodes are
kept open.
From the source I have looked at, the *notify code seems to
attempt to hold on to the inodes that are watched, but perhaps
it has some hidden assumptions that the 'audit' module does not
satisfy.
Do you have a reproducer to detect this quickly?
Miklos Szeredi appears to have found the likely cause:
https://lkml.org/lkml/2014/11/4/246
- RGB
--
Richard Guy Briggs <rbriggs(a)redhat.com>
Senior Software Engineer, Kernel Security, AMER ENG Base Operating Systems, Red Hat
Remote, Ottawa, Canada
Voice: +1.647.777.2635, Internal: (81) 32635, Alt: +1.613.693.0684x3545
3699
days inactive
3897
days old
linux-audit@lists.linux-audit.osci.io
10 comments
5 participants
participants (5)
-
Eric Paris -
lists_todd＠mac.com -
Peter Grandi -
Richard Guy Briggs -
Steve Grubb