Greetings, fellow Toaster-ites.
Hopefully this will distract you from the previous l1nux4411 "I'm 16 and Linux r00lz" troll-fest ...
We've got a creaky old F740 running 6.3.1R1 that's performed yeoman duty for us for many years.
All of a sudden it's having regular (at least once an hour) issues with "No buffer space available" errors that cause temporary NFS mount hiccup issues. I'm not sure how bad the situation is but it hasn't happened to us before (we also had our first panic on the F740 the other day, too) and more to the point, a reboot didn't fix the problem - it came right back shortly thereafter. Which is what worries me - a reboot should've fixed the issue, at least for awhile. I'm also troubled that there doesn't seem to be any tunable options parameter to increase the number of kernel buffers (or kernel buffer space in general) to deal with this issue.
Has any other Toaster-ite run into this before?
Here's the configuration and the error messages below.
Thanks,
- Greg Earle
------- Begin Forwarded Message
===== SYSCONFIG-A ===== NetApp Release 6.3.1R1: Thu Feb 20 04:06:12 PST 2003 System ID: 00XXXXXXXX (toaster) System Serial Number: XXXXX (toaster) slot 0: System Board (NetApp System Board V N4) Model Name: F740 Serial Number: NNNNNN Firmware release: 2.8_a2 Processors: 1 Memory Size: 512 MB
Apr 9 17:12:59 notatoaster1 automount[27576]: >> mount: toaster:/vol/ vol1/local failed, reason given by server: Permission denied Apr 9 17:12:59 toaster Mon Apr 9 17:12:59 PDT [nis_main:warning]: Bound to preferred NIS server NNN.NN.NN.69_n Apr 9 17:12:59 toaster Mon Apr 9 17:12:59 PDT [nis_poll:error]: Cannot send NIS binding request to NNN.NN.NN.70 : No buffer space available Apr 9 17:12:59 toaster Mon Apr 9 17:12:59 PDT [nis_poll:error]: Cannot send NIS binding request to NNN.NN.NN.69 : No buffer space available Apr 9 17:12:59 toaster Mon Apr 9 17:12:59 PDT [mountd_main:warning]: yp_match: clnt_call: RPC: Unable to send_ errno = No buffer space available Apr 9 17:12:59 toaster Mon Apr 9 17:12:59 PDT [nis_main:warning]: Preferred NIS Server NNN.NN.NN.69 not responding_n Apr 9 17:12:59 toaster Mon Apr 9 17:12:59 PDT [nis_main:info]: NIS server(s) not available Apr 9 17:12:59 toaster Mon Apr 9 17:12:59 PDT [nis_poll:info]: NIS server(s) not available Apr 9 17:12:59 notatoaster automount[27576]: mount(nfs): nfs: mount failure toaster:/vol/vol1/local on /usr/local Apr 9 17:12:59 toaster Mon Apr 9 17:12:59 PDT [nis_poll:warning]: Bound to preferred NIS server NNN.NN.NN.70_n Apr 9 17:14:19 notatoaster2 automount[15953]: >> mount: mount to NFS server 'toaster' failed: timed out (retrying). Apr 9 17:15:38 toaster Mon Apr 9 17:15:38 PDT [mountd_main:warning]: yp_match: clnt_call: RPC: Unable to send_ errno = No buffer space available Apr 9 17:15:38 toaster Mon Apr 9 17:15:38 PDT [nis_main:warning]: Preferred NIS Server NNN.NN.NN.70 not responding_n Apr 9 17:15:38 toaster Mon Apr 9 17:15:38 PDT [nis_main:warning]: Bound to preferred NIS server NNN.NN.NN.69_n Apr 9 17:15:38 toaster Mon Apr 9 17:15:38 PDT [nis_poll:error]: Cannot send NIS binding request to NNN.NN.NN.70 : No buffer space available Apr 9 17:15:38 toaster Mon Apr 9 17:15:38 PDT [nis_poll:error]: Cannot send NIS binding request to NNN.NN.NN.69 : No buffer space available Apr 9 17:15:38 toaster Mon Apr 9 17:15:38 PDT [nis_main:warning]: Preferred NIS Server NNN.NN.NN.69 not responding_n Apr 9 17:15:38 toaster Mon Apr 9 17:15:38 PDT [nis_main:info]: NIS server(s) not available Apr 9 17:15:38 toaster Mon Apr 9 17:15:38 PDT [nis_poll:info]: NIS server(s) not available Apr 9 17:15:40 toaster Mon Apr 9 17:15:40 PDT [nis_poll:warning]: Bound to preferred NIS server NNN.NN.NN.70_n
Have you thought about upgrading the current release? like 6.5.7something?
On 4/9/07, Greg Earle earle@isolar.dyndns.org wrote:
Greetings, fellow Toaster-ites.
Hopefully this will distract you from the previous l1nux4411 "I'm 16 and Linux r00lz" troll-fest ...
We've got a creaky old F740 running 6.3.1R1 that's performed yeoman duty for us for many years.
All of a sudden it's having regular (at least once an hour) issues with "No buffer space available" errors that cause temporary NFS mount hiccup issues. I'm not sure how bad the situation is but it hasn't happened to us before (we also had our first panic on the F740 the other day, too) and more to the point, a reboot didn't fix the problem - it came right back shortly thereafter. Which is what worries me - a reboot should've fixed the issue, at least for awhile. I'm also troubled that there doesn't seem to be any tunable options parameter to increase the number of kernel buffers (or kernel buffer space in general) to deal with this issue.
Has any other Toaster-ite run into this before?
Here's the configuration and the error messages below.
Thanks,
- Greg Earle------- Begin Forwarded Message
===== SYSCONFIG-A ===== NetApp Release 6.3.1R1: Thu Feb 20 04:06:12 PST 2003 System ID: 00XXXXXXXX (toaster) System Serial Number: XXXXX (toaster) slot 0: System Board (NetApp System Board V N4) Model Name: F740 Serial Number: NNNNNN Firmware release: 2.8_a2 Processors: 1 Memory Size: 512 MB
Apr 9 17:12:59 notatoaster1 automount[27576]: >> mount: toaster:/vol/ vol1/local failed, reason given by server: Permission denied Apr 9 17:12:59 toaster Mon Apr 9 17:12:59 PDT [nis_main:warning]: Bound to preferred NIS server NNN.NN.NN.69_n Apr 9 17:12:59 toaster Mon Apr 9 17:12:59 PDT [nis_poll:error]: Cannot send NIS binding request to NNN.NN.NN.70 : No buffer space available Apr 9 17:12:59 toaster Mon Apr 9 17:12:59 PDT [nis_poll:error]: Cannot send NIS binding request to NNN.NN.NN.69 : No buffer space available Apr 9 17:12:59 toaster Mon Apr 9 17:12:59 PDT [mountd_main:warning]: yp_match: clnt_call: RPC: Unable to send_ errno = No buffer space available Apr 9 17:12:59 toaster Mon Apr 9 17:12:59 PDT [nis_main:warning]: Preferred NIS Server NNN.NN.NN.69 not responding_n Apr 9 17:12:59 toaster Mon Apr 9 17:12:59 PDT [nis_main:info]: NIS server(s) not available Apr 9 17:12:59 toaster Mon Apr 9 17:12:59 PDT [nis_poll:info]: NIS server(s) not available Apr 9 17:12:59 notatoaster automount[27576]: mount(nfs): nfs: mount failure toaster:/vol/vol1/local on /usr/local Apr 9 17:12:59 toaster Mon Apr 9 17:12:59 PDT [nis_poll:warning]: Bound to preferred NIS server NNN.NN.NN.70_n Apr 9 17:14:19 notatoaster2 automount[15953]: >> mount: mount to NFS server 'toaster' failed: timed out (retrying). Apr 9 17:15:38 toaster Mon Apr 9 17:15:38 PDT [mountd_main:warning]: yp_match: clnt_call: RPC: Unable to send_ errno = No buffer space available Apr 9 17:15:38 toaster Mon Apr 9 17:15:38 PDT [nis_main:warning]: Preferred NIS Server NNN.NN.NN.70 not responding_n Apr 9 17:15:38 toaster Mon Apr 9 17:15:38 PDT [nis_main:warning]: Bound to preferred NIS server NNN.NN.NN.69_n Apr 9 17:15:38 toaster Mon Apr 9 17:15:38 PDT [nis_poll:error]: Cannot send NIS binding request to NNN.NN.NN.70 : No buffer space available Apr 9 17:15:38 toaster Mon Apr 9 17:15:38 PDT [nis_poll:error]: Cannot send NIS binding request to NNN.NN.NN.69 : No buffer space available Apr 9 17:15:38 toaster Mon Apr 9 17:15:38 PDT [nis_main:warning]: Preferred NIS Server NNN.NN.NN.69 not responding_n Apr 9 17:15:38 toaster Mon Apr 9 17:15:38 PDT [nis_main:info]: NIS server(s) not available Apr 9 17:15:38 toaster Mon Apr 9 17:15:38 PDT [nis_poll:info]: NIS server(s) not available Apr 9 17:15:40 toaster Mon Apr 9 17:15:40 PDT [nis_poll:warning]: Bound to preferred NIS server NNN.NN.NN.70_n
On Apr 9, 2007, at 5:51 PM, Greg Earle wrote:
Apr 9 17:12:59 notatoaster1 automount[27576]: >> mount: toaster:/ vol/vol1/local failed, reason given by server: Permission denied Apr 9 17:12:59 toaster Mon Apr 9 17:12:59 PDT [nis_main:warning]: Bound to preferred NIS server NNN.NN.NN.69_n Apr 9 17:12:59 toaster Mon Apr 9 17:12:59 PDT [nis_poll:error]: Cannot send NIS binding request to NNN.NN.NN.70 : No buffer space available Apr 9 17:12:59 toaster Mon Apr 9 17:12:59 PDT [nis_poll:error]: Cannot send NIS binding request to NNN.NN.NN.69 : No buffer space available Apr 9 17:12:59 toaster Mon Apr 9 17:12:59 PDT [mountd_main:warning]: yp_match: clnt_call: RPC: Unable to send_ errno = No buffer space available
The way I am reading the above (and its late so I could be completely out of it), this seems to me to be a problem when trying to send a NIS poll to whatever system you are using for NIS. Are you sure your NIS server is happy and has plenty of memory to allocate buffers to your NFS mount requests? Additionally, the way this is formatted (the verbatim wording) reminds me of a Solaris error, you using Sol for NIS?
http://www.adminschoice.com/docs/solaris_error_message_l_n.htm#errno141
On Apr 9, 2007, at 11:11 PM, Shane Garoutte wrote:
On Apr 9, 2007, at 5:51 PM, Greg Earle wrote:
Apr 9 17:12:59 notatoaster1 automount[27576]: >> mount: toaster:/ vol/vol1/local failed, reason given by server: Permission denied Apr 9 17:12:59 toaster Mon Apr 9 17:12:59 PDT [nis_main:warning]: Bound to preferred NIS server NNN.NN.NN.69_n Apr 9 17:12:59 toaster Mon Apr 9 17:12:59 PDT [nis_poll:error]: Cannot send NIS binding request to NNN.NN.NN.70 : No buffer space available Apr 9 17:12:59 toaster Mon Apr 9 17:12:59 PDT [nis_poll:error]: Cannot send NIS binding request to NNN.NN.NN.69 : No buffer space available Apr 9 17:12:59 toaster Mon Apr 9 17:12:59 PDT [mountd_main:warning]: yp_match: clnt_call: RPC: Unable to send_ errno = No buffer space available
The way I am reading the above (and its late so I could be completely out of it), this seems to me to be a problem when trying to send a NIS poll to whatever system you are using for NIS. Are you sure your NIS server is happy and has plenty of memory to allocate buffers to your NFS mount requests? Additionally, the way this is formatted (the verbatim wording) reminds me of a Solaris error, you using Sol for NIS?
http://www.adminschoice.com/docs/ solaris_error_message_l_n.htm#errno141
The 2 NIS slave servers (yes, both Solaris 9 boxes) do little else besides service NIS and some application Kerberos and LDAP services. They don't themselves serve out NFS, nor are they much in the way of NFS consumers.
There is only one instance in the logs of issues between the NIS servers and the Filer, i.e. an "NFS server not responding still trying" instance that righted itself about a minute later.
Since I highly doubt they both ran out of buffer space simultaneously, I'm assuming it's the NetApp Filer in question that did. ;)
We've never experienced these "No buffer space available" issues before, and you can tell from the age of the OS that we've had this puppy for quite a while now (and an F540 before that, too).
Someone from the list used the comparison tool to show me that there's literally hundreds upon hundreds of bugs fixed between 6.3.1R1 and 6.5.7. It looks like a couple of them might be relevant to this situation:
43372 SNMP reports "No buffer space available" 4
100694 WAFL ran out of buffers on low memory systems. WAFL 2
I'll plead with the Management to see if there's any way we can get a waiver (we're in a Flight Project Configuration Freeze - people get nervous when you tell them you're going to touch the Filer) to upgrade it.
Thanks for all your suggestions so far - including Nils' one for me to try and get a new NOW account for us. I'm pursuing that route as we speak ...
- Greg
-
Greg Earle -
Shane Garoutte -
tmac