a systat -s -u 1 gives me the following output:
tigershark> sysstat -s -u 1 CPU Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP Disk ops/s in out read write read write age hit time ty util 49% 2255 70895 2009 3306 20734 0 0 47 100% 78% : 22% 37% 2329 74366 2150 76 0 0 0 47 99% 0% - 12% 38% 2642 74641 2460 100 24 0 0 47 97% 0% - 3% 39% 2880 67314 3378 920 0 0 0 47 96% 0% - 13% 54% 3730 61661 3252 4645 8 0 0 47 95% 0% - 31% 48% 5236 70522 3813 541 24 0 0 47 97% 0% - 6% 47% 4534 73570 3512 350 0 0 0 47 98% 0% - 4% 48% 5146 75076 3949 381 0 0 0 47 97% 0% - 4% 48% 4896 73286 3505 465 24 0 0 47 98% 0% - 6% 44% 3996 74177 3050 254 8 0 0 47 99% 0% - 5% 51% 3580 66122 2863 9770 1391 0 0 47 100% 26% T 11% 57% 5194 69234 3855 4230 23416 0 0 47 99% 100% : 23% 44% 3128 73071 3734 905 1428 0 0 47 98% 17% : 10% 46% 4474 74810 3876 392 0 0 0 47 99% 0% - 4% 41% 3437 76071 2899 60 32 0 0 47 99% 0% - 2% -- Summary Statistics ( 15 samples 1 secs/sample) CPU Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP Disk ops/s in out read write read write age hit time ty util Min 37% 2255 61661 2009 60 0 0 0 47 95% 0% * 2% Avg 46% 3830 71654 3220 1759 3139 0 0 47 98% 14% * 10% Max 57% 5236 76071 3949 9770 23416 0 0 47 100% 100% * 31%
does anybody now why the net input is incredibly high compared to the written disk data?
thx,
christian
Hi
I have seen very high CPU utilisation (up to 99%) when users were attempting to connect via CIFS (initial browsing only, no significant data transfers). Packet traces at the time showed unusually high numbers of fragmented packets which pointed to local networking issues and our NT4 (!) domain controllers. The problem got fixed by a power outage which rebooted all our edge switches!
I am still concerned that a network issue between clients and the filer should cause such high (debilitating) processor load on the filer.
Regards Francois
Christian Mikovits wrote:
a systat -s -u 1 gives me the following output:
tigershark> sysstat -s -u 1 CPU Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP Disk ops/s in out read write read write age hit time ty util 49% 2255 70895 2009 3306 20734 0 0 47 100% 78% : 22% 37% 2329 74366 2150 76 0 0 0 47 99% 0% - 12% 38% 2642 74641 2460 100 24 0 0 47 97% 0% - 3% 39% 2880 67314 3378 920 0 0 0 47 96% 0% - 13% 54% 3730 61661 3252 4645 8 0 0 47 95% 0% - 31% 48% 5236 70522 3813 541 24 0 0 47 97% 0% - 6% 47% 4534 73570 3512 350 0 0 0 47 98% 0% - 4% 48% 5146 75076 3949 381 0 0 0 47 97% 0% - 4% 48% 4896 73286 3505 465 24 0 0 47 98% 0% - 6% 44% 3996 74177 3050 254 8 0 0 47 99% 0% - 5% 51% 3580 66122 2863 9770 1391 0 0 47 100% 26% T 11% 57% 5194 69234 3855 4230 23416 0 0 47 99% 100% : 23% 44% 3128 73071 3734 905 1428 0 0 47 98% 17% : 10% 46% 4474 74810 3876 392 0 0 0 47 99% 0% - 4% 41% 3437 76071 2899 60 32 0 0 47 99% 0% - 2% -- Summary Statistics ( 15 samples 1 secs/sample) CPU Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP Disk ops/s in out read write read write age hit time ty util Min 37% 2255 61661 2009 60 0 0 0 47 95% 0%
- 2% Avg
46% 3830 71654 3220 1759 3139 0 0 47 98% 14% * 10% Max 57% 5236 76071 3949 9770 23416 0 0 47 100% 100% * 31%
does anybody now why the net input is incredibly high compared to the written disk data?
thx,
christian
Update:
Now the output in an attachement, becauso of mail line wrap at 80 characters ...
It could be just overhead of the apps. Or a lot of transactions that don't translate directly into disk write operations.
On Dec 18, 2007 6:30 AM, Christian Mikovits christian.mikovits@cubit.at wrote:
Update:
Now the output in an attachement, becauso of mail line wrap at 80 characters ...
Right. Your best bet would be to do a packet trace and see what's actually happening. It could be lots of reasons, some perfectly fine (such as very small cached operations), and others not so nice (lots of network errors). But the wire doesn't lie, so see what's on there.
-- Adam Fox adamfox@netapp.com
-----Original Message----- From: Blake Golliher [mailto:thelastman@gmail.com] Sent: Tuesday, December 18, 2007 11:43 AM To: Christian Mikovits Cc: toasters@mathworks.com Subject: Re: Weird sysstat output
It could be just overhead of the apps. Or a lot of transactions that don't translate directly into disk write operations.
On Dec 18, 2007 6:30 AM, Christian Mikovits christian.mikovits@cubit.at wrote:
Update:
Now the output in an attachement, becauso of mail line wrap at 80 characters ...
I configured one of 3040s to be backed to tape through directly attached NDMP using direct attached tape drive, the issue is that after data written, from Netbackup the session never finish, but on the filer, ndmp session shows it finished and tape is unloaded.
Netbackup 5.1MP2 Filer: 3040 with NTAP 7.2.3
Did anybody see similar issue on your filer?
Thanks jackie
Is there a firewall between your backup server and the filer?
If so, you may be blowing the session timeout, so the filer isn't communicating back to the backup server that it has finished.
Paul
Did you verify that the ndmp session truly did not complete? If the filer is indicating that the NDMP session is completed, but Netbackup never finishes, is there any messages on the Netbackup indicating the reason for the failure?
James
-----Original Message----- From: owner-toasters@mathworks.com [mailto:owner-toasters@mathworks.com] On Behalf Of Paul Keating Sent: Tuesday, December 18, 2007 12:28 PM To: Li, Jackie (Yanhui); toasters@mathworks.com Subject: RE: filer NDMP backup issue
Is there a firewall between your backup server and the filer?
If so, you may be blowing the session timeout, so the filer isn't communicating back to the backup server that it has finished.
Paul
There is firewall.
Every time near the end of backup, we saw following error in the ndmp log on filer: Dec 20 18:43:06 GMT [ndmpd:5]: Fatal top level Exception Dec 20 18:43:06 GMT [ndmpd:5]: General Exception: String index out of range: -1
-----Original Message----- From: Johnson, James A [HDS] [mailto:James.Johnson8@hdsupply.com] Sent: Friday, December 21, 2007 8:09 AM To: Paul Keating; Li, Jackie (Yanhui); toasters@mathworks.com Subject: RE: filer NDMP backup issue
Did you verify that the ndmp session truly did not complete? If the filer is indicating that the NDMP session is completed, but Netbackup never finishes, is there any messages on the Netbackup indicating the reason for the failure?
James
-----Original Message----- From: owner-toasters@mathworks.com [mailto:owner-toasters@mathworks.com] On Behalf Of Paul Keating Sent: Tuesday, December 18, 2007 12:28 PM To: Li, Jackie (Yanhui); toasters@mathworks.com Subject: RE: filer NDMP backup issue
Is there a firewall between your backup server and the filer?
If so, you may be blowing the session timeout, so the filer isn't communicating back to the backup server that it has finished.
Paul
Ndmp does not work through firewalls directly. I think if you made a vpn tha( encapsulates all ports, you might be able to ndmp through a firewall.
--tmac
Sent from my Verizon Wireless BlackBerry
-----Original Message----- From: "Li, Jackie (Yanhui)" YLi@ea.com
Date: Fri, 21 Dec 2007 10:10:41 To:"Johnson, James A [HDS]" James.Johnson8@hdsupply.com, "Paul Keating" pkeating@bank-banque-canada.ca, toasters@mathworks.com Subject: RE: filer NDMP backup issue
There is firewall.
Every time near the end of backup, we saw following error in the ndmp log on filer: Dec 20 18:43:06 GMT [ndmpd:5]: Fatal top level Exception Dec 20 18:43:06 GMT [ndmpd:5]: General Exception: String index out of range: -1
-----Original Message----- From: Johnson, James A [HDS] [mailto:James.Johnson8@hdsupply.com] Sent: Friday, December 21, 2007 8:09 AM To: Paul Keating; Li, Jackie (Yanhui); toasters@mathworks.com Subject: RE: filer NDMP backup issue
Did you verify that the ndmp session truly did not complete? If the filer is indicating that the NDMP session is completed, but Netbackup never finishes, is there any messages on the Netbackup indicating the reason for the failure?
James
-----Original Message----- From: owner-toasters@mathworks.com [mailto:owner-toasters@mathworks.com] On Behalf Of Paul Keating Sent: Tuesday, December 18, 2007 12:28 PM To: Li, Jackie (Yanhui); toasters@mathworks.com Subject: RE: filer NDMP backup issue
Is there a firewall between your backup server and the filer?
If so, you may be blowing the session timeout, so the filer isn't communicating back to the backup server that it has finished.
Paul
--
-----Original Message----- From: owner-toasters@mathworks.com [mailto:owner-toasters@mathworks.com] On Behalf Of Li, Jackie (Yanhui) Sent: December 18, 2007 2:55 PM To: toasters@mathworks.com Subject: filer NDMP backup issue
I configured one of 3040s to be backed to tape through directly attached NDMP using direct attached tape drive, the issue is that after data written, from Netbackup the session never finish, but on the filer, ndmp session shows it finished and tape is unloaded.
Netbackup 5.1MP2 Filer: 3040 with NTAP 7.2.3
Did anybody see similar issue on your filer?
Thanks jackie
====================================================================================
La version française suit le texte anglais.
------------------------------------------------------------------------------------
This email may contain privileged and/or confidential information, and the Bank of Canada does not waive any related rights. Any distribution, use, or copying of this email or the information it contains by other than the intended recipient is unauthorized. If you received this email in error please delete it immediately from your system and notify the sender promptly by email that you have done so.
------------------------------------------------------------------------------------
Le présent courriel peut contenir de l'information privilégiée ou confidentielle. La Banque du Canada ne renonce pas aux droits qui s'y rapportent. Toute diffusion, utilisation ou copie de ce courriel ou des renseignements qu'il contient par une personne autre que le ou les destinataires désignés est interdite. Si vous recevez ce courriel par erreur, veuillez le supprimer immédiatement et envoyer sans délai à l'expéditeur un message électronique pour l'aviser que vous avez éliminé de votre ordinateur toute copie du courriel reçu.
Sorry I meant there is NO firewall, the backup server and filer is on the same subnet.
-----Original Message----- From: Tim McCarthy [mailto:tmacmd@gmail.com] Sent: Friday, December 21, 2007 10:45 AM To: Li, Jackie (Yanhui); owner-toasters@mathworks.com; Johnson, James A [HDS]; Paul Keating; NetApp Toasters List Subject: Re: filer NDMP backup issue
Ndmp does not work through firewalls directly. I think if you made a vpn tha( encapsulates all ports, you might be able to ndmp through a firewall.
--tmac
Sent from my Verizon Wireless BlackBerry
-----Original Message----- From: "Li, Jackie (Yanhui)" YLi@ea.com
Date: Fri, 21 Dec 2007 10:10:41 To:"Johnson, James A [HDS]" James.Johnson8@hdsupply.com, "Paul Keating" pkeating@bank-banque-canada.ca, toasters@mathworks.com Subject: RE: filer NDMP backup issue
There is firewall.
Every time near the end of backup, we saw following error in the ndmp log on filer: Dec 20 18:43:06 GMT [ndmpd:5]: Fatal top level Exception Dec 20 18:43:06 GMT [ndmpd:5]: General Exception: String index out of range: -1
-----Original Message----- From: Johnson, James A [HDS] [mailto:James.Johnson8@hdsupply.com] Sent: Friday, December 21, 2007 8:09 AM To: Paul Keating; Li, Jackie (Yanhui); toasters@mathworks.com Subject: RE: filer NDMP backup issue
Did you verify that the ndmp session truly did not complete? If the filer is indicating that the NDMP session is completed, but Netbackup never finishes, is there any messages on the Netbackup indicating the reason for the failure?
James
-----Original Message----- From: owner-toasters@mathworks.com [mailto:owner-toasters@mathworks.com] On Behalf Of Paul Keating Sent: Tuesday, December 18, 2007 12:28 PM To: Li, Jackie (Yanhui); toasters@mathworks.com Subject: RE: filer NDMP backup issue
Is there a firewall between your backup server and the filer?
If so, you may be blowing the session timeout, so the filer isn't communicating back to the backup server that it has finished.
Paul
--
-----Original Message----- From: owner-toasters@mathworks.com [mailto:owner-toasters@mathworks.com] On Behalf Of Li, Jackie (Yanhui) Sent: December 18, 2007 2:55 PM To: toasters@mathworks.com Subject: filer NDMP backup issue
I configured one of 3040s to be backed to tape through directly attached NDMP using direct attached tape drive, the issue is that after data written, from Netbackup the session never finish, but on the filer, ndmp session shows it finished and tape is unloaded.
Netbackup 5.1MP2 Filer: 3040 with NTAP 7.2.3
Did anybody see similar issue on your filer?
Thanks jackie
====================================================================================
La version française suit le texte anglais.
------------------------------------------------------------------------------------
This email may contain privileged and/or confidential information, and the Bank of Canada does not waive any related rights. Any distribution, use, or copying of this email or the information it contains by other than the intended recipient is unauthorized. If you received this email in error please delete it immediately from your system and notify the sender promptly by email that you have done so.
------------------------------------------------------------------------------------
Le présent courriel peut contenir de l'information privilégiée ou confidentielle. La Banque du Canada ne renonce pas aux droits qui s'y rapportent. Toute diffusion, utilisation ou copie de ce courriel ou des renseignements qu'il contient par une personne autre que le ou les destinataires désignés est interdite. Si vous recevez ce courriel par erreur, veuillez le supprimer immédiatement et envoyer sans délai à l'expéditeur un message électronique pour l'aviser que vous avez éliminé de votre ordinateur toute copie du courriel reçu.
-
Blake Golliher -
Christian Mikovits -
Fox, Adam -
Francois Joubert -
Johnson, James A [HDS] -
Li, Jackie (Yanhui) -
Paul Keating -
Tim McCarthy