LSA Errors
by lella.r.bennettï¼ lmco.com
10 Nov '00
10 Nov '00
Has anyone ever seen these errors before? We are seeing them consistently after the filer disconnects from its resource domain controller.
[CIFSAuthen]: LsaLookupNames: Exception rpc_s_connection_closed caught.
[CIFSAuthen]: LsaClose: Exception rpc_s_connection_closed caught.
-Lella Bennett
Lockheed Martin EIS
303-971-1055
Has anyone ever seen these errors before? We are seeing them
consistently after the filer disconnects from its resource domain
controller.
[CIFSAuthen]: LsaLookupNames: Exception rpc_s_connection_closed caught.
[CIFSAuthen]: LsaClose: Exception rpc_s_connection_closed caught.
Any info would be appreciated.
Thanks!
Lella Bennett
Lockheed Martin EIS
303-971-1055
1
0
LSA errors
by lella.r.bennettï¼ lmco.com
10 Nov '00
10 Nov '00
Has anyone ever seen these errors before? We are seeing them consistently after the filer disconnects from its resource domain controller.
[CIFSAuthen]: LsaLookupNames: Exception rpc_s_connection_closed caught.
[CIFSAuthen]: LsaClose: Exception rpc_s_connection_closed caught.
-Lella Bennett
Lockheed Martin EIS
303-971-1055
Have you looked to see if the drives are down? We are running NetApp Release 5.3.4R3P2 on our filers, NBU 3.2+latest patches.
There is a bug in this version of ontap that causes the tape drives to be downed at, we think, the end of jobs when it comes time to unload the tape or at the change of tapes on a job that needs to span tapes. I understand that 5.3.4 (base) does not have that problem. The thing is, we see this consistently on some filers and rarely on others.
As a work around, we ended up having a cron job up the drives hourly. This took care of the problem.
Look in the archives around 5/2/00, subject "Network Appliance Upgrade to ONTAP 5.3.4R3P2 and Backups" and you should see more on this problem.
Kelly
--
Kelly Wyatt, Kelly.Wyatt(a)SAS.com
Senior Systems Programmer
Strategic Recovery
SAS Institute Inc. / SAS Campus Drive / Cary, NC 27513
http://www.sas.com
SAS... The Power to Know
-----Original Message-----
From: Jeffrey Krueger [mailto:jkrueger@qualcomm.com]
Sent: Tuesday, November 07, 2000 10:18 PM
To: toasters(a)mathworks.com
Subject: Veritas NetBackup backup hangs
We have recently cut-over to using Veritas NetBackup with NDMP to perform a
mixture of backups to filer-attached-tape and 3-way backups to other filers
with tape. We are running DOT 5.3.4R3P2 on a mixture of 630's and 760's
against NBU 3.2 with a recent patch set (one that includes the 3-way fix).
In both instance, we have found that sometimes a filer completes a dump
(ndmp session goes away, backup snapshot is deleted, /etc/dumpdates is
updates), but the backup job hangs in Veritas-land. This job hangs
indefiately until manually killed, at which point it does not record the
file history making the backup un-restore-able.
We're growing very frustrated with this issue. BudTool has been backing
these filers up for 2 years with NDMP and relatively few problems. So far,
NBU support has been disappointing to say the least.
Has anyone seen this? Is there a workaround or fix?
Also, does anyone know the mechanics of the "/etc/ndmpdlog.*" files? They
are poorly documented in the DOT manuals. All of ours are zero length, but
its unclear whether this is supposed to be the case (no news is good
news). Does anyone know what they are supposed to look like?
-- Jeff
--
----------------------------------------------------------------------------
Jeff Krueger E-Mail: jeff(a)qualcomm.com
Senior Engineer Phone: 858-651-6709
NetApp Filers / UNIX Infrastructure Fax: 858-651-6627
QUALCOMM, Inc. IT Engineering Web: www.qualcomm.com
rkissel(a)netapp.com unsubscribe
Rich
--------------------------------------
Rich Kissel-----------Account Manager
Network Appliance
100 N. El Segundo Blvd.
Suite 1802, 18th Floor
El Segundo, CA 90245
310-426-1717 Office
310-426-1701 Fax
310-739-6577 Mobile
877-5-NETAPP Pager
rich.kissel(a)netapp.com
www.netapp.com
"NTAP could be next Cisco" CBS MarketWatch
http://cbs.marketwatch.com/archive/19991207/news/current/smith.htx?source=ht
Hi all,
I'd like to give a big thanks to all who showed me the option for
turning on NFS stats on a per-client basis. The NetApp 5.2 SysAdmin
manual doesn't mention this at all in the trouble shooting section.
They should update their docs better.
I'd also like to thank Darren for the hint on using 'prex' on solaris
to get more info on processes and which are doing what.
And another big thanks to Mike Horwath <drechsau(a)tiny.net> for sending
the 'netapp-top' script. I haven't used it yet, but it looks like
just the thing to help us out in the future. The script is at the end
of this message.
John
John Stoffel - Senior Unix Systems Administrator - Lucent Technologies
stoffel(a)lucent.com - http://www.lucent.com - 978-952-7548
--------------------------snip here---------------------------------
#!/usr/local/bin/perl
# netapp-top - display current top NFS clients for a NetApp
# Copyright (C) 1998 Daniel Quinlan
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
$default_interval = 60;
$prog = $0;
$prog =~ s@.*/@@;
require "getopts.pl";
&Getopts("hi:");
if (defined $ENV{"LINES"} && $ENV{"LINES"} ne "") {
$lines = $ENV{"LINES"};
}
else {
$lines = 24;
}
if ($opt_h) {
&usage;
exit 0;
}
if ($opt_i && $opt_i > 0) {
$interval = $opt_i;
}
else {
$interval = $default_interval;
}
if ($ARGV[0]) {
$host = $ARGV[0];
}
else {
&usage;
exit 1;
}
sub usage {
print <<EOF;
usage: $prog [options] filer
-h print this help
-i n report filer statistics every n seconds ($default_interval is default)
The NetApp must have the "nfs.per_client_stats.enable" option
"on" for this to work at all.
This program must be run this from an admin host as root.
EOF
}
sub print_top {
my $total = 0;
my $diff;
my %recent;
foreach $key (keys %nfs_ops_last) {
$diff = $nfs_ops{$key} - $nfs_ops_last{$key};
$recent{$key} = $diff;
$total += $diff;
}
$total = int ($total / $interval);
system("clear");
printf "%6s\t%s\n", "NFS/s",
"total NFS/s = $total | interval = $interval";
printf "%6s\t%s\n", "------", "----------------------------------------";
$i = 0;
foreach $key (sort {$recent{$b} <=> $recent{$a} } keys %recent) {
printf "%6d\t%s\n", int ($recent{$key} / $interval + 0.5),
$ip_name{$key};
last if ($i++ > ($lines - 5));
}
}
$first = 1;
for (;;) {
$time_0 = time();
open(NFSSTAT, "rsh $host nfsstat -l|");
while (<NFSSTAT>) {
@tmp = split;
# ip_address = tmp[0]
# ip_name = tmp[1]
# nfs_ops = tmp[4]
$ip_name{$tmp[0]} = $tmp[1];
$nfs_ops{$tmp[0]} = $tmp[4];
}
close(NFSSTAT);
if (! $first) {
&print_top;
}
else {
$first = 0;
}
%nfs_ops_last = %nfs_ops;
$time_1 = time();
sleep ($interval - ($time_1 - $time_0));
}
Hi all,
Just to drag this conversation back to a purely NetApp, purely NFS
scenario, I'd like to get some help and pointers on how I can solve a
problem I had this morning in a more general and useful way.
Let me give you the background details here.
We have a bunch of toasters here, various old F330s, an F520 (soon to
be retired) and some F740s. This morning a bunch of people were
complaining that their workstations were slow, that home directories
were timing out, etc. These people all had their home directories on
an F330 running OnTap 5.2.1. It has 192Mb of RAM, four shelves, each
with 7 x 4gb disks.
The poor system was simply pinned to the wall by a client. The CPU
was hovering between 85% and 100%, it was reading and writing around
2.3Mb/s to the disks constantly. The nfsstats told me that about 23%
of the traffic was writes, the rest was attr lookups and reads. The
usual mix of NFS traffic. The cache age was down around 4-5 (it's
normally much higher), so I knew it was getting hit hard with writes.
But since the system was on a direct link back to a switch, and since
I don't run the network at all and don't have access to it, I couldn't
tell which system(s) were beating it up.
We ended up putting in a PC on a repeater to sniff the link between
the switch and the NetApp to try and figure out which host(s) was the
bad boy.
Once we figured that out, it still didn't help since the two hosts
didn't look loaded at all, nor were there any runaway processes
sucking up IO that I could find. The clients were both quad processor
Suns running Solaris 2.5.1 or 2.6.
I use the following tools to try and figure out what was going on
here, and failed. We had to reboot the two systems to solve the
problem. Now as a Unix admin, this really pained me, since I should
have been able to find the culprits and just kill them off. We used
these on the solaris side:
snoop
tcpdump
lsof
top
ps (in all kinds of variations).
ethereal (found after the fact, will be used in the future).
And on the NetApp side I used:
nfsstat
netstat -n
netstat -r
sysstat 4
And while they all showed me something, none of them could show me
what I needed.
On the NetApp side I needed something to show me the top 10 NFS hosts
but IP address, but I couldn't get it to work. The output of 'netstat
-r' wasn't a help at all.
On the Solaris side, tcpdump showed me the traffic, but didn't give me
a way to relate it back to a specific process. And while lsof showed
me processes, it didn't show me which one was writing data and at what
rate.
Does anyone have any hints? I'm thinking of upgrading to 5.3.6 at
some point, just to bring the F330s upto date with the F740s, but I'm
not in a rush really.
Ideally, something on the NetApp side to show me the top NFS clients
in terms of Data Rate, or anything would be a god-send. Then
something on the Client side to figure out which process(es) were the
NFS hogs would also be good.
Thanks,
John
John Stoffel - Senior Unix Systems Administrator - Lucent Technologies
stoffel(a)lucent.com - http://www.lucent.com - 978-952-7548