Subject: kernel errors
From: Daniel Urist <durist@ucar.edu>
Date: Thu, 20 Oct 2016 15:26:19 -0600
Thu, 20 Oct 2016 15:26:19 -0600
Can anyone explain to me what these error messages mean?


> Oct 18 21:52:18 zweb5 kernel: vxW: [xid #773] !!! limit:
> ffff881027dd90b0[NOFILE,7] = 2 on exit.
> Oct 18 21:52:20 zweb5 kernel: vxW: [xid #758] !!! limit:
> ffff882033dbc0b0[NOFILE,7] = 4 on exit.
> Oct 18 21:52:20 zweb5 kernel: vxW: [xid #758] !!! limit:
> ffff882033dbc0b0[SHMEM,19] = 393 on exit.
> Oct 18 21:52:21 zweb5 kernel: vxW: [xid #761] !!! limit:
> ffff882033ceb0b0[NOFILE,7] = 4 on exit.
> Oct 18 21:52:21 zweb5 kernel: vxW: [xid #761] !!! limit:
> ffff882033ceb0b0[SHMEM,19] = 393 on exit.


OS is debian jessie with psand kernel v. 4.1.33-vs2.3.8.5.2-beng, guests
are also debian jessie.

A little background:

My vserver filesystems are hosted on a netapp, and are running in a
pacemaker/corosync cluster.

Occasionally, the filesystem monitor script that the cluster is using times
out, and the cluster moves the vservers around to different nodes in the
cluster. The timeout behavior is really odd-- the monitor script is a
simple perl script that reads /proc/self/mountinfo, but the first thing it
does when it runs is to log (via syslog) that it has started. When it times
out, it never even gets that far. The system never appears to be highly
loaded or low on memory, and there are no NFS errors.

The timing of these events seem to be roughly correlated with a mysqldump
that's running via cron in one of the vservers; but the mysqldump runs fine
when run by hand, so that might be a red herring.

The error messages above occurred about 1.5 minutes before the monitor
scripts timed out (and the cluster subsequently moved the vservers), so it
doesn't seem that they should be coming from the vserver shutdown.

Thanks in advance for any enlightenment.


Can anyone explain to me what these error messages mean? 
 
Oct 18 21:52:18 zweb5 kernel: vxW: [xid #773] !!! limit: ffff881027dd90b0[NOFILE,7] = 2 on exit.
Oct 18 21:52:20 zweb5 kernel: vxW: [xid #758] !!! limit: ffff882033dbc0b0[NOFILE,7] = 4 on exit.
Oct 18 21:52:20 zweb5 kernel: vxW: [xid #758] !!! limit: ffff882033dbc0b0[SHMEM,19] = 393 on exit.
Oct 18 21:52:21 zweb5 kernel: vxW: [xid #761] !!! limit: ffff882033ceb0b0[NOFILE,7] = 4 on exit.
Oct 18 21:52:21 zweb5 kernel: vxW: [xid #761] !!! limit: ffff882033ceb0b0[SHMEM,19] = 393 on exit.

OS is debian jessie with psand kernel v. 4.1.33-vs2.3.8.5.2-beng, guests are also debian jessie.

A little background:

My vserver filesystems are hosted on a netapp, and are running in a pacemaker/corosync cluster. 

Occasionally, the filesystem monitor script that the cluster is using times out, and the cluster moves the vservers around to different nodes in the cluster. The timeout behavior is really odd-- the monitor script is a simple perl script that reads /proc/self/mountinfo, but the first thing it does when it runs is to log (via syslog) that it has started. When it times out, it never even gets that far. The system never appears to be highly loaded or low on memory, and there are no NFS errors.

The timing of these events seem to be roughly correlated with a mysqldump that's running via cron in one of the vservers; but the mysqldump runs fine when run by hand, so that might be a red herring. 

The error messages above occurred about 1.5 minutes before the monitor scripts timed out (and the cluster subsequently moved the vservers), so it doesn't seem that they should be coming from the vserver shutdown.

Thanks in advance for any enlightenment.