Hi all, I debugged a bit further to identify distributions that are affected by the issue. I repeated the same experiment with sshd from 3 more distributions. CentOS Linux release 7.1.1503 (64-bit, 3.10.0-229.el7.x86_64): Problem NOT reproduced CentOS release 6.6 (64-bit, 2.6.32-504.el6.x86_64): Problem NOT reproduced Ubuntu 12.04.5 LTS (64-bit, 3.13.0-32-generic): Problem reproduced

After all, Ubuntu family are affected by the issue and I could confirm that results are inconsistent across two different distribution families.

If you can let us know how can we workaround the issue, it will be a great help. Regards, Kangkook > On Sep 9, 2015, at 11:50 PM, Kangkook Jee <aixer77(a)gmail.com&gt; wrote: > > Dear all, > > We are developing custom user space audit agent to gather system wide system > call trace. While experimenting with various programs, we found out that > processes (daemons) that started early (along with the system bootstrapping) do > not report any audit events at all. These processes typically fall into PID > range of less than 2000. Here’s how I reproduced the symptom with sshd daemon. > > 1. Reboot the system > > 2. Add and enable audit events > # /sbin/auditctl -a exit,always -F arch=b64 -S clone -S close -S creat -S dup > -S dup2 -S dup3 -S execve -S exit -S exit_group -S fork -S open -S openat > -S unlink -S unlinkat -S vfork -S 288 -S accept -S bind -S connect > -S listen -S socket -S socketpair > # /sbin/auditctl -e1 -b 102400 > > 3. Connect to the system via ssh > Audit messages generated only from child processes and none are seen from > the original daemon. > > 4. Restart sshd > # restart ssh > > 5. Connect again to the system via ssh > Now, we see audit messages from both parent and child processes. > > I did the experiment from Ubuntu 14.04.2 LTS distribution (64-bit, kernel > version 3.13.0-58-generic). > > I first wonder whether this is intended behavior of audit framework or > not. If it is intended, I also want to know how can we configure auditd > differently to capture system calls from all processes. > > Thanks a lot for your help in advance! > > Regards, Kangkook >

-- Linux-audit mailing list Linux-audit(a)redhat.com https://www.redhat.com/mailman/listinfo/linux-audit

On Sep 11, 2015, at 5:50 AM, Richard Guy Briggs <rgb(a)redhat.com&gt; wrote: On 15/09/10, Kangkook Jee wrote: > Hi all, > > I debugged a bit further to identify distributions that are affected by the issue. > I repeated the same experiment with sshd from 3 more distributions. > > CentOS Linux release 7.1.1503 (64-bit, 3.10.0-229.el7.x86_64): Problem NOT reproduced > CentOS release 6.6 (64-bit, 2.6.32-504.el6.x86_64): Problem NOT reproduced > Ubuntu 12.04.5 LTS (64-bit, 3.13.0-32-generic): Problem reproduced For each of these examples, what is the value of the kernel command line "audit=<value>" if it is even present? It is possible that the CentOS examples include "audit=1" while Ubuntu omits the line. "cat /proc/cmdline" should tell you the answer. > After all, Ubuntu family are affected by the issue and I could confirm > that results are inconsistent across two different distribution > families. I would be curious what your results are with a recent Debian and with a recent Fedora. > If you can let us know how can we workaround the issue, it will be a great help. > > Regards, Kangkook > > >> On Sep 9, 2015, at 11:50 PM, Kangkook Jee <aixer77(a)gmail.com&gt; wrote: >> >> Dear all, >> >> We are developing custom user space audit agent to gather system wide system >> call trace. While experimenting with various programs, we found out that >> processes (daemons) that started early (along with the system bootstrapping) do >> not report any audit events at all. These processes typically fall into PID >> range of less than 2000. Here’s how I reproduced the symptom with sshd daemon. >> >> 1. Reboot the system >> >> 2. Add and enable audit events >> # /sbin/auditctl -a exit,always -F arch=b64 -S clone -S close -S creat -S dup >> -S dup2 -S dup3 -S execve -S exit -S exit_group -S fork -S open -S openat >> -S unlink -S unlinkat -S vfork -S 288 -S accept -S bind -S connect >> -S listen -S socket -S socketpair >> # /sbin/auditctl -e1 -b 102400 >> >> 3. Connect to the system via ssh >> Audit messages generated only from child processes and none are seen from >> the original daemon. >> >> 4. Restart sshd >> # restart ssh >> >> 5. Connect again to the system via ssh >> Now, we see audit messages from both parent and child processes. >> >> I did the experiment from Ubuntu 14.04.2 LTS distribution (64-bit, kernel >> version 3.13.0-58-generic). >> >> I first wonder whether this is intended behavior of audit framework or >> not. If it is intended, I also want to know how can we configure auditd >> differently to capture system calls from all processes. >> >> Thanks a lot for your help in advance! >> >> Regards, Kangkook >> > > -- > Linux-audit mailing list > Linux-audit(a)redhat.com <mailto:Linux-audit@redhat.com> > https://www.redhat.com/mailman/listinfo/linux-audit <https://www.redhat.com/mailman/listinfo/linux-audit> - RGB -- Richard Guy Briggs <rbriggs(a)redhat.com <mailto:rbriggs@redhat.com>> Senior Software Engineer, Kernel Security, AMER ENG Base Operating Systems, Red Hat Remote, Ottawa, Canada Voice: +1.647.777.2635, Internal: (81) 32635, Alt: +1.613.693.0684x3545

On Sep 11, 2015, at 12:24 PM, Richard Guy Briggs <rgb(a)redhat.com&gt; wrote: On 15/09/11, Kangkook Jee wrote: > From the previous reply, I think I misunderstood your question regarding kernel command line. > Here’s "cat /proc/cmdline” results for distributions that I’ve experimented. > > Ubuntu 14.04 (64-bit): > BOOT_IMAGE=/boot/vmlinuz-3.13.0-58-generic root=UUID=7505f862-ce46-49e5-9d1c-e4e307844889 ro text quiet splash vt.handoff=7 > > Ubuntu 12.04 (64-bit): > BOOT_IMAGE=/boot/vmlinuz-3.13.0-32-generic root=UUID=5be789be-9b0c-463e-bd18-42bfa79fb24c ro quiet splash > > CentOS 7 (64-bit): > BOOT_IMAGE=/vmlinuz-3.10.0-229.el7.x86_64 root=/dev/mapper/centos-root ro rd.lvm.lv=centos/root rd.lvm.lv=centos/swap crashkernel=auto rhgb quiet LANG=en_US.UTF-8 > > CentOS 6 (64-bit): > ro root=UUID=a7d44560-adcc-4000-9584-8b9fcf2afd74 rd_NO_LUKS rd_NO_LVM LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=129M@0M KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet > > I don’t see any audit=<value> entries from all examples above. Yes, this is what I was seeking from you. And you are correct, none of them have audit=1 as I was hoping from at least CentOS. There is a chance that the CentOS kernel was compiled with audit=1 hardcoded, but I think that is a pretty small chance... I'll have to look at this closer... But any Debian and Fedora data points that you can provide would certainly be useful. > /Kangkook > >> On Sep 11, 2015, at 5:50 AM, Richard Guy Briggs <rgb(a)redhat.com&gt; wrote: >> >> On 15/09/10, Kangkook Jee wrote: >>> Hi all, >>> >>> I debugged a bit further to identify distributions that are affected by the issue. >>> I repeated the same experiment with sshd from 3 more distributions. >>> >>> CentOS Linux release 7.1.1503 (64-bit, 3.10.0-229.el7.x86_64): Problem NOT reproduced >>> CentOS release 6.6 (64-bit, 2.6.32-504.el6.x86_64): Problem NOT reproduced >>> Ubuntu 12.04.5 LTS (64-bit, 3.13.0-32-generic): Problem reproduced >> >> For each of these examples, what is the value of the kernel command line >> "audit=<value>" if it is even present? It is possible that the CentOS >> examples include "audit=1" while Ubuntu omits the line. "cat >> /proc/cmdline" should tell you the answer. >> >>> After all, Ubuntu family are affected by the issue and I could confirm >>> that results are inconsistent across two different distribution >>> families. >> >> I would be curious what your results are with a recent Debian and with a >> recent Fedora. >> >>> If you can let us know how can we workaround the issue, it will be a great help. >>> >>> Regards, Kangkook >>> >>> >>>> On Sep 9, 2015, at 11:50 PM, Kangkook Jee <aixer77(a)gmail.com&gt; wrote: >>>> >>>> Dear all, >>>> >>>> We are developing custom user space audit agent to gather system wide system >>>> call trace. While experimenting with various programs, we found out that >>>> processes (daemons) that started early (along with the system bootstrapping) do >>>> not report any audit events at all. These processes typically fall into PID >>>> range of less than 2000. Here’s how I reproduced the symptom with sshd daemon. >>>> >>>> 1. Reboot the system >>>> >>>> 2. Add and enable audit events >>>> # /sbin/auditctl -a exit,always -F arch=b64 -S clone -S close -S creat -S dup >>>> -S dup2 -S dup3 -S execve -S exit -S exit_group -S fork -S open -S openat >>>> -S unlink -S unlinkat -S vfork -S 288 -S accept -S bind -S connect >>>> -S listen -S socket -S socketpair >>>> # /sbin/auditctl -e1 -b 102400 >>>> >>>> 3. Connect to the system via ssh >>>> Audit messages generated only from child processes and none are seen from >>>> the original daemon. >>>> >>>> 4. Restart sshd >>>> # restart ssh >>>> >>>> 5. Connect again to the system via ssh >>>> Now, we see audit messages from both parent and child processes. >>>> >>>> I did the experiment from Ubuntu 14.04.2 LTS distribution (64-bit, kernel >>>> version 3.13.0-58-generic). >>>> >>>> I first wonder whether this is intended behavior of audit framework or >>>> not. If it is intended, I also want to know how can we configure auditd >>>> differently to capture system calls from all processes. >>>> >>>> Thanks a lot for your help in advance! >>>> >>>> Regards, Kangkook >>>> >>> >> >>> -- >>> Linux-audit mailing list >>> Linux-audit(a)redhat.com <mailto:Linux-audit@redhat.com> <mailto:Linux-audit@redhat.com <mailto:Linux-audit@redhat.com>> >>> https://www.redhat.com/mailman/listinfo/linux-audit <https://www.redhat.com/mailman/listinfo/linux-audit> <https://www.redhat.com/mailman/listinfo/linux-audit <https://www.redhat.com/mailman/listinfo/linux-audit>> >> >> >> - RGB >> >> -- >> Richard Guy Briggs <rbriggs(a)redhat.com <mailto:rbriggs@redhat.com> <mailto:rbriggs@redhat.com <mailto:rbriggs@redhat.com>>> >> Senior Software Engineer, Kernel Security, AMER ENG Base Operating Systems, Red Hat >> Remote, Ottawa, Canada >> Voice: +1.647.777.2635, Internal: (81) 32635, Alt: +1.613.693.0684x3545 > - RGB -- Richard Guy Briggs <rbriggs(a)redhat.com <mailto:rbriggs@redhat.com>> Senior Software Engineer, Kernel Security, AMER ENG Base Operating Systems, Red Hat Remote, Ottawa, Canada Voice: +1.647.777.2635, Internal: (81) 32635, Alt: +1.613.693.0684x3545

>> >> Kangkook, >> >> Perhaps you can re-test, but modify the kernel boot parameters to >> include audit=1 as an additional argument. >> >> Reading the auditd(8) manual, one sees >> >> A boot param of audit=1 should be added to ensure that all >> processes that run before the audit daemon starts is marked as >> auditable by the kernel. Not doing that will make a few >> processes impossible to properly audit.

On Sep 13, 2015, at 11:58 AM, Richard Guy Briggs <rgb(a)redhat.com <mailto:rgb@redhat.com>> wrote: On 15/09/11, Kangkook Jee wrote: > Hi Richard, Hi Kangkook, > I also did the same experiment for the latest distributions of Fedora > core and Debian and here’s the results. > > Fedora-22 (64-bit, 4.0.4-301.fc22.x86_64): Problem reproduced. > Debian-8 (64-bit, 3.16.0-4-amd64): Problem reproduced > > Btw, Burn Alting (burn(a)swtf.dyndns.org <mailto:burn@swtf.dyndns.org>) suggested me to append audit=1 > to kernel flag. I added the option to boot-loader (grub) and problem > went away. On all systems? This is expected behaviour. Sorry I was not more explicit in asking you to test that. I guess it was implied by asking what the settings for the kernel command line were. Now the surprising bit is that CentOS does not demonstrate the problem without audit=1 in the command line, which leads me to wonder if they have set "u32 audit_enabled = 1;" around line 83 of kernel/audit.c in their kernel source. It would surprise me if they did, but it would not be completely unreasonable. > Regards, Kangkook > >> On Sep 11, 2015, at 12:24 PM, Richard Guy Briggs <rgb(a)redhat.com <mailto:rgb@redhat.com>> wrote: >> On 15/09/11, Kangkook Jee wrote: >>> From the previous reply, I think I misunderstood your question regarding kernel command line. >>> Here’s "cat /proc/cmdline” results for distributions that I’ve experimented. >>> >>> Ubuntu 14.04 (64-bit): >>> BOOT_IMAGE=/boot/vmlinuz-3.13.0-58-generic root=UUID=7505f862-ce46-49e5-9d1c-e4e307844889 ro text quiet splash vt.handoff=7 >>> >>> Ubuntu 12.04 (64-bit): >>> BOOT_IMAGE=/boot/vmlinuz-3.13.0-32-generic root=UUID=5be789be-9b0c-463e-bd18-42bfa79fb24c ro quiet splash >>> >>> CentOS 7 (64-bit): >>> BOOT_IMAGE=/vmlinuz-3.10.0-229.el7.x86_64 root=/dev/mapper/centos-root ro rd.lvm.lv=centos/root rd.lvm.lv=centos/swap crashkernel=auto rhgb quiet LANG=en_US.UTF-8 >>> >>> CentOS 6 (64-bit): >>> ro root=UUID=a7d44560-adcc-4000-9584-8b9fcf2afd74 rd_NO_LUKS rd_NO_LVM LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=129M@0M KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet >>> >>> I don’t see any audit=<value> entries from all examples above. >> >> Yes, this is what I was seeking from you. And you are correct, none of >> them have audit=1 as I was hoping from at least CentOS. There is a >> chance that the CentOS kernel was compiled with audit=1 hardcoded, but I >> think that is a pretty small chance... >> >> I'll have to look at this closer... But any Debian and Fedora data >> points that you can provide would certainly be useful. >> >>> /Kangkook >>> >>>> On Sep 11, 2015, at 5:50 AM, Richard Guy Briggs <rgb(a)redhat.com <mailto:rgb@redhat.com>> wrote: >>>> >>>> On 15/09/10, Kangkook Jee wrote: >>>>> Hi all, >>>>> >>>>> I debugged a bit further to identify distributions that are affected by the issue. >>>>> I repeated the same experiment with sshd from 3 more distributions. >>>>> >>>>> CentOS Linux release 7.1.1503 (64-bit, 3.10.0-229.el7.x86_64): Problem NOT reproduced >>>>> CentOS release 6.6 (64-bit, 2.6.32-504.el6.x86_64): Problem NOT reproduced >>>>> Ubuntu 12.04.5 LTS (64-bit, 3.13.0-32-generic): Problem reproduced >>>> >>>> For each of these examples, what is the value of the kernel command line >>>> "audit=<value>" if it is even present? It is possible that the CentOS >>>> examples include "audit=1" while Ubuntu omits the line. "cat >>>> /proc/cmdline" should tell you the answer. >>>> >>>>> After all, Ubuntu family are affected by the issue and I could confirm >>>>> that results are inconsistent across two different distribution >>>>> families. >>>> >>>> I would be curious what your results are with a recent Debian and with a >>>> recent Fedora. >>>> >>>>> If you can let us know how can we workaround the issue, it will be a great help. >>>>> >>>>> Regards, Kangkook >>>>> >>>>> >>>>>> On Sep 9, 2015, at 11:50 PM, Kangkook Jee <aixer77(a)gmail.com <mailto:aixer77@gmail.com>> wrote: >>>>>> >>>>>> Dear all, >>>>>> >>>>>> We are developing custom user space audit agent to gather system wide system >>>>>> call trace. While experimenting with various programs, we found out that >>>>>> processes (daemons) that started early (along with the system bootstrapping) do >>>>>> not report any audit events at all. These processes typically fall into PID >>>>>> range of less than 2000. Here’s how I reproduced the symptom with sshd daemon. >>>>>> >>>>>> 1. Reboot the system >>>>>> >>>>>> 2. Add and enable audit events >>>>>> # /sbin/auditctl -a exit,always -F arch=b64 -S clone -S close -S creat -S dup >>>>>> -S dup2 -S dup3 -S execve -S exit -S exit_group -S fork -S open -S openat >>>>>> -S unlink -S unlinkat -S vfork -S 288 -S accept -S bind -S connect >>>>>> -S listen -S socket -S socketpair >>>>>> # /sbin/auditctl -e1 -b 102400 >>>>>> >>>>>> 3. Connect to the system via ssh >>>>>> Audit messages generated only from child processes and none are seen from >>>>>> the original daemon. >>>>>> >>>>>> 4. Restart sshd >>>>>> # restart ssh >>>>>> >>>>>> 5. Connect again to the system via ssh >>>>>> Now, we see audit messages from both parent and child processes. >>>>>> >>>>>> I did the experiment from Ubuntu 14.04.2 LTS distribution (64-bit, kernel >>>>>> version 3.13.0-58-generic). >>>>>> >>>>>> I first wonder whether this is intended behavior of audit framework or >>>>>> not. If it is intended, I also want to know how can we configure auditd >>>>>> differently to capture system calls from all processes. >>>>>> >>>>>> Thanks a lot for your help in advance! >>>>>> >>>>>> Regards, Kangkook >>>> >>>> - RGB >> >> - RGB - RGB -- Richard Guy Briggs <rbriggs(a)redhat.com <mailto:rbriggs@redhat.com>> Senior Software Engineer, Kernel Security, AMER ENG Base Operating Systems, Red Hat Remote, Ottawa, Canada Voice: +1.647.777.2635, Internal: (81) 32635, Alt: +1.613.693.0684x3545