Turns out there is a limit of 1 interactive SSH session (knew that) and 20 non-interactive sessions (did not know that). And there were hung sessions (monitoring, perfstats, etc) from several hosts. Rebooted two hosts and got the connection count down below 20 and all is well.

Thanks!

From: toasters-bounces@teaparty.net [mailto:toasters-bounces@teaparty.net] On Behalf Of Randy Rue
Sent: Wednesday, February 20, 2013 10:28 AM
To: toasters@teaparty.net
Subject: can't shoot SSH commands to a filer

Hello All,

I have a 3020 running 7.2.4 that is failing to respond to SSH commands when they’re sent one at a time. I can ssh in as root and get a shell via putty or ssh. But if I try to send an “uptime” command I get either “FATAL ERROR: Server unexpectedly closed network connection” (from plink) or “Connection to carbon-b closed by remote host.” from ssh.

Also, when I’m logged in interactively, I see lots of this blowing across the console:

carbon-b> receive signals: rsh protocol failure in circuit setup. 300 received

receive signals: rsh protocol failure in circuit setup. 300 received

Try again.

Try again.

receive signals: rsh protocol failure in circuit setup. 300 received

receive signals: rsh protocol failure in circuit setup. 300 received

receive signals: rsh protocol failure in circuit setup. 300 received

receive signals: rsh protocol failure in circuit setup. 300 received

receive signals: rsh protocol failure in circuit setup. 300 received

Try again.

receive signals: rsh protocol failure in circuit setup. 300 received

Try again.

receive signals: rsh protocol failure in circuit setup. 300 received

Try again.

receive signals: rsh protocol failure in circuit setup. 300 received

Try again.

Try again.

Try again.

receive signals: rsh protocol failure in circuit setup. 300 received

Try again.

Try again.

Try again.

Try again.

Try again.

Try again.

Try again.

Try again.

receive signals: rsh protocol failure in circuit setup. 300 received

Try again.

receive signals: rsh protocol failure in circuit setup. 300 received

Try again.

receive signals: rsh protocol failure in circuit setup. 300 received

Try again.

Try again.

receive signals: rsh protocol failure in circuit setup. 300 received

Try again.

Try again.

Try again.

Try again.

Try again.

Try again.

receive signals: rsh protocol failure in circuit setup. 300 received

Try again.

Try again.

Try again.

Try again.

Try again.

receive signals: rsh protocol failure in circuit setup. 300 received

Try again.

Try again.

Try again.

Try again.

Try again.

Try again.

receive signals: rsh protocol failure in circuit setup. 300 received

We have lots of processes (monitoring, alerting, scripted processes) that touch this box via non-interactive SSH, is each of these errors some failed attempt to connect?

If I understand the inner workings correctly, pretty much all admin traffic on a filer channels into an internal RSH pipe. Are there different pipes for interactive and non-interactive SSH? I have tried stopping and restarting SSH (options ssh.enable off and on). Also RSH (options rsh.enable off and on). No love.

Anyone have any ideas?

Randy

Seattle