toasters June 2011

toasters@lists.teaparty.net

24 participants
15 discussions

Re: Truncated SSH ouptut on 8.x?
by A Darren Dunham 31 Aug '11

31 Aug '11

On Wed, Jun 08, 2011 at 08:24:54AM +0200, Sander Klein wrote: > Maybe it's related to the ssh client you're using? Pretty sure it's not a network or SSH issue. I just got a nice reproduction, and the data I got really surprised me. (BTW, the filer I'm hitting below is running 8.0.1P2). I have a script that checks "qtree stats" periodically. And to save on SSH setup/teardown costs, it runs a sysstat in the same command. But the script then works through the data and hands back a summary. I don't have a copy of the raw stuff coming back. But I saw some oddities that made it appear that it was not getting all the data. So I ran the following command and was going to put it in a crontab to run mutliple times so I could see if it went off. "qtree stats; sysstat -c 1 5" Well, while testing the script, on the 4th time I ran it I got "short" output. On this filer, that command should return 179 lines. On the last try it only returned 10, but not simply the first 10. Here's a (slightly obfuscated) version of what I got back: -----BEGIN No qtrees are in use in Volume vol0 Volume Tree NFS ops CIFS ops -------- -------- ------- -------- perf [user]perf 1773 0 Volume Tree NFS ops CIFS ops -------- -------- ------- -------- test_vm_volume vm_vol 0 0 CPU NFS CIFS HTTP Net kB/s Disk kB/s Tape kB/s Cache in out read write read write age 77% 28350 0 0 290677 82179 194167 415548 0 0 6s -----END In reality, the "perf" volume has over 100 qtrees. The one shown above is the first line in the list when the command is complete. So it's not simply truncating the total output, it has managed to drop a portion of the output of one command. If it were a network or SSH problem, I wouldn't expect a gap in the middle of a command, nor would I expect it exactly on a nice line boundary. Looks like I can open a ticket now. -- Darren

4 9

Re: Finding out which client has mapped a share?
by John Stoffel 01 Jul '11

01 Jul '11

Kyle> How about cifs sessions *? supportnetapp01> cifs sessions * Doesn't seem to work. When I do this command, I don't see any clients mapping this share, but it still won't let me delete it. Very frustrating. John

6 8

Finding out which client has mapped a share?
by John Stoffel 30 Jun '11

30 Jun '11

Is there anyway to find out which client has mapped which shares? I'm trying to delete a share and even with: cifs shares -delete -f applications It won't go away and I get the following. Thu Jun 30 11:52:14 PDT [smbrpc.forceClose.share.timeout:warning]: CIFSRPC: Close of share Applications has not completed and has taken 3660 seconds. This is on a FAS2050 running 7.2.6.1P8, which I know I should upgrade sometime. Thanks, John

4 4

Re: Finding out which client has mapped a share?
by John Stoffel 30 Jun '11

30 Jun '11

Alexander> you could either terminate cifs quickly or check the open Alexander> files list of the filer using the "manage computer" mmc on Alexander> a windows workstation connected to your filer to force Alexander> closing of the open files on this share. We'll probably just terminate all of CIFS tonight after 6pm filer local time to see if that will clean things up. Too bad you can't rename a share, but I can see why not, since the protocol is tied so tight between the clients/server. Alexander> If this CIFS share is a volume on its own you might also Alexander> want to try to take the volume offline, but i'm not sure Alexander> about that. I don't want to get that extreme. *grin* Thanks for all the suggestions. John

1 0

RE: kicking an unhappy ssh daemon
by Ehrhart, Rick 29 Jun '11

29 Jun '11

Hi Chris - You're correct rshstat is not available in ONTAP 7.2.6.x. In attempt to isolate the problem, ssh is piggy-backed on top of rsh and telnet depending if ssh is interactive or not; so if rsh and telnet work then the problem is in ssh part. However if rsh does not work, hopefully, more information will be displayed. My thought is that all the rsh sessions and therefore ssh sessions have been consumed. The rshstat command would have told us. If all the rsh sessions are consumed, then unfortunately, a re-boot is needed. All the options below look fine. Since the telnet.access and rsh.access is legacy, please check options trusted.hosts to make sure this is set correctly. Regards, - Rick - -----Original Message----- From: Christopher B. Allison [mailto:cba@ccs.neu.edu] Sent: Wednesday, June 29, 2011 9:06 AM To: Ehrhart, Rick Cc: David N. Blank-Edelman; toasters(a)mathworks.com Subject: RE: kicking an unhappy ssh daemon Hi Rick, Thanks for your reply. I work for David, and am replying to this because David is in meetings all day. On Wed, 29 Jun 2011, Ehrhart, Rick wrote: > What is the version of ONTAP? NetApp Release 7.2.6.1P9: Wed Aug 19 12:37:58 PDT 2009 > Please show the output of "options ssh", ssh.access host!=192.168.200.50 ssh.enable on ssh.idle.timeout 60 ssh.passwd_auth.enable on ssh.port 22 ssh.pubkey_auth.enable on ssh1.enable off ssh2.enable on Note: We added the "host!=192.168.200.50" directive to block out our monitoring system when we thought it was consuming all the ssh slots. We've since discarded that theory, but haven't reverted the ssh block. > "options telnet" telnet.access legacy telnet.distinct.enable off telnet.enable on > "options rsh". rsh.access legacy rsh.enable on > Please show the output of "rshstat". "Do a "priv set advanced" first. rshstat doesn't appear to be available on this filer, either in admin or advanced privilege mode, though it is present on two filers we have running 8.0.x. Is rshstat available in ONTAP 7.2.6? If it matters: I'm still investigating this on our side, but I believe the onset of the problem was actually tied to an influx of bad ssh attempts (brute force attack), rather than any configuration change on our part. I'm happy to provide more information, including a detailed log of the attempts I've made to restart the ssh daemon, if it would be helpful. Best, -Chris Allison -- UNIX Systems Administrator College of Computer and Information Science Northeastern University, Boston, MA

1 0

kicking an unhappy ssh daemon
by David N. Blank-Edelman 29 Jun '11

29 Jun '11

Hi- We have an interesting situation with one of our filers. Earlier today, one of our monitoring systems managed to eat up all of the available ssh slots on the box. Once we got that under control, we have found we still can't get on to the box via ssh or ssh commands to it, they just hang. There is something listening and accepting connections, but it hangs once the connection is made. We've tried: - turning off and on the ssh options - changing the ssh port (and changing it back) - secureadmin setup ssh (and moving the keys back) Can you think of anything else I could do to kick the daemon short of rebooting the box (which is currently serving in production)? Perhaps something in a super-squirrel-secret priv mode? Thanks! -- dNb

4 4

Re: kicking an unhappy ssh daemon
by Bennett Todd 29 Jun '11

29 Jun '11

That suggests an idea for the possible cause, to me. Symptom: inbound ssh connections start hanging, and eventually, after a long time, recover. Each new ssh connection requires some high-quality random bits, unguessable absent knowledge that can't be externally observed. Suppose the ssh server is drawing them directly from an entropy pool that is only slowly, gradually refilled by the OS observing things like the rotational latency of spinning drives as that's slightly disturbed by turbulence? I don't know the OS that underlies netapps, so I don't know if that pool can be manually fed from commandline, nor if the ssh server can be redirected to a PRNG itself just seeded by the entropy pool. Or if so how to do it. -Bennett On Jun 29, 2011 1:31 AM, "Stefan Funke" <bundy(a)usage.de> wrote:

1 0

Re: kicking an unhappy ssh daemon
by David N. Blank-Edelman 29 Jun '11

29 Jun '11

Hi Felix- Thanks for your response. On Tuesday, June 28, 2011 at 6:08 PM, Felix Schröder wrote: > To kick out a stalled SSH session just type "ssh root@filer logout telnet" on a *nix machine, then type in the password - That will force the filer to logout any interactive (ssh) session. It works the same way with rsh. I must not have been clear in my message. Both interactive, and non-interactive ssh command the machine are hanging. I can't even do an ssh netapp help, nevermind execute the logout command. The connections never get that far. I should also mention that at the moment, according to netstat, there are no existing ssh connections so there isn't really anything to logout. -- dNb

2 1

Re: kicking an unhappy ssh daemon
by David N. Blank-Edelman 29 Jun '11

29 Jun '11

On Tuesday, June 28, 2011 at 7:48 PM, Felix Schröder wrote: > When a session has become hung, if a serial console session is possible, or > some other means of issuing commands is possible, then the hung connection > may be located in the output of the "netstat -a" command so that the "Swind" > size may be checked: if it is zero, then this problem may be occurring, and > the user may try to close and then re-open the session from the client. Hmm, so my read of this bug is "sometimes sessions in progress hang. If that happens, start another session" which doesn't seem quite like what I am seeing where I can't even start a session at this point. -- dNb

1 0

Issues with Compression?
by Milazzo Giacomo 25 Jun '11

25 Jun '11

Hi all, I've received a couple of alert from field with users using DoT 8.0.1 compression feature on Oracle production environment. Their application hang! NetApp representative here in Italy suggested us to avoid, for the moment, to use compression on Oracle and generally on DB/backup environment... Do you have experience? Positive and not I mean. And, of course, not only on Oracle/DB environment. Thank you in advance,

2 1

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

1999

1998

1997

toasters June 2011