Hello toasters,
I am having an issue where I periodically find that some xp desktop users are doing a large amount of other ops to a cifs share with a large amount of files for a sustained period. If I let them go, they will continue for days.
In most cases, it seems the users do a search and the search takes too long, so they close the window, but it is running in some kind of stuck state in the background. Killing the session or closing open files via connection to the toaster via computer management does nothing; the session just starts back up and pounds the crap out of the filler. Rebooting the desktop has been my solution so far, though I suspect restarting explorer.exe may be sufficient.
I am running 8.1.1P2 and this is an IBM rebranded FAS3070. Any ideas? Has anyone seen similar behavior?
cifs top -n 10 ops/s reads(n, KB/s) writes(n, KB/s) suspect/s IP Name 1859 | 0 0 | 0 0 | 0 | 10.10.100.223 domain\someuser2 1418 | 0 0 | 0 0 | 0 | 10.10.100.227 domain\someuser1 1673 | 0 0 | 0 0 | 0 | 10.10.180.230 domain\someuser3
qtree status share1 vol1 ntfs enabled normal
vol status -v share1 Volume State Status Options Share1 online raid_dp, flex nosnap=off, nosnapdir=off, minra=off, sis no_atime_update=off, nvfail=off, 64-bit ignore_inconsistent=off, snapmirrored=off, create_ucode=on, convert_ucode=off, maxdirsize=73400, schedsnapname=ordinal, fs_size_fixed=on, guarantee=none, svo_enable=off, svo_checksum=off, svo_allow_rman=off, svo_reject_errors=off, no_i2p=off, fractional_reserve=0, extent=off, try_first=volume_grow, read_realloc=off, snapshot_clone_dependency=off, dlog_hole_reserve=off, nbu_archival_snap=off Volume UUID: 0fbdb127-a774-11e2-ab63-123478563412 Containing aggregate: 'sataaggr0'
Plex /sataaggr0/plex0: online, normal, active RAID group /sataaggr0/plex0/rg0: normal, block checksums RAID group /sataaggr0/plex0/rg1: normal, block checksums RAID group /sataaggr0/plex0/rg2: normal, block checksums RAID group /sataaggr0/plex0/rg3: normal, block checksums
Snapshot autodelete settings for share1: state=off commitment=try trigger=volume target_free_space=20% delete_order=oldest_first defer_delete=user_created prefix=(not specified) destroy_list=none Volume autosize settings: state=off Hybrid Cache: Eligibility=read-write
For the fourth consecutive year, Independent Health is the highest ranked health insurance plan in the New York/New Jersey region by J.D. Power and Associates 2013 Member Health Plan StudySM.
For the sixth consecutive year, Independent Health was named one of the best companies to work for in New York State.
Go to www.independenthealth.com to view careers and available positions.
We are an Equal Opportunity Employer.
CONFIDENTIALITY NOTICE: This e-mail and its attachments (collectively referred to as "e-mail") may contain confidential information that is privileged and protected from disclosure by Federal and State confidentiality laws, rules or regulations. This e-mail is intended for the designated addressee only. If you are not the designated addressee, you are notified that any disclosure, copying or distribution of this e-mail may be unlawful and may subject you to legal consequences. If you have received this e-mail in error, please contact me immediately by telephone at (716) 631 - 3001 and delete the e-mail from your computer immediately. Thank you for your attention.
Does anyone have any ideas or have seen similar behavior to this?
From: toasters-bounces@teaparty.net [mailto:toasters-bounces@teaparty.net] On Behalf Of Jordan Slingerland Sent: Wednesday, June 05, 2013 3:01 PM To: toasters@teaparty.net Subject: large amount of CIFS other ops from xp desktops after canceled search
Hello toasters,
I am having an issue where I periodically find that some xp desktop users are doing a large amount of other ops to a cifs share with a large amount of files for a sustained period. If I let them go, they will continue for days.
In most cases, it seems the users do a search and the search takes too long, so they close the window, but it is running in some kind of stuck state in the background. Killing the session or closing open files via connection to the toaster via computer management does nothing; the session just starts back up and pounds the crap out of the filler. Rebooting the desktop has been my solution so far, though I suspect restarting explorer.exe may be sufficient.
I am running 8.1.1P2 and this is an IBM rebranded FAS3070. Any ideas? Has anyone seen similar behavior?
cifs top -n 10 ops/s reads(n, KB/s) writes(n, KB/s) suspect/s IP Name 1859 | 0 0 | 0 0 | 0 | 10.10.100.223 domain\someuser2 1418 | 0 0 | 0 0 | 0 | 10.10.100.227 domain\someuser1 1673 | 0 0 | 0 0 | 0 | 10.10.180.230 domain\someuser3
qtree status share1 vol1 ntfs enabled normal
vol status -v share1 Volume State Status Options Share1 online raid_dp, flex nosnap=off, nosnapdir=off, minra=off, sis no_atime_update=off, nvfail=off, 64-bit ignore_inconsistent=off, snapmirrored=off, create_ucode=on, convert_ucode=off, maxdirsize=73400, schedsnapname=ordinal, fs_size_fixed=on, guarantee=none, svo_enable=off, svo_checksum=off, svo_allow_rman=off, svo_reject_errors=off, no_i2p=off, fractional_reserve=0, extent=off, try_first=volume_grow, read_realloc=off, snapshot_clone_dependency=off, dlog_hole_reserve=off, nbu_archival_snap=off Volume UUID: 0fbdb127-a774-11e2-ab63-123478563412 Containing aggregate: 'sataaggr0'
Plex /sataaggr0/plex0: online, normal, active RAID group /sataaggr0/plex0/rg0: normal, block checksums RAID group /sataaggr0/plex0/rg1: normal, block checksums RAID group /sataaggr0/plex0/rg2: normal, block checksums RAID group /sataaggr0/plex0/rg3: normal, block checksums
Snapshot autodelete settings for share1: state=off commitment=try trigger=volume target_free_space=20% delete_order=oldest_first defer_delete=user_created prefix=(not specified) destroy_list=none Volume autosize settings: state=off Hybrid Cache: Eligibility=read-write
For the fourth consecutive year, Independent Health is the highest ranked health insurance plan in the New York/New Jersey region by J.D. Power and Associates 2013 Member Health Plan StudySM.
For the sixth consecutive year, Independent Health was named one of the best companies to work for in New York State.
Go to www.independenthealth.comhttp://www.independenthealth.com to view careers and available positions.
We are an Equal Opportunity Employer.
CONFIDENTIALITY NOTICE: This e-mail and its attachments (collectively referred to as "e-mail") may contain confidential information that is privileged and protected from disclosure by Federal and State confidentiality laws, rules or regulations. This e-mail is intended for the designated addressee only. If you are not the designated addressee, you are notified that any disclosure, copying or distribution of this e-mail may be unlawful and may subject you to legal consequences. If you have received this e-mail in error, please contact me immediately by telephone at (716) 631 - 3001 and delete the e-mail from your computer immediately. Thank you for your attention.
Looks like metadata crawls (which is why the IOPS are so high compared to the read/write throughput.
________________________________________ From: toasters-bounces@teaparty.net [toasters-bounces@teaparty.net] on behalf of Jordan Slingerland [Jordan.Slingerland@independenthealth.com] Sent: Friday, June 28, 2013 4:05 PM To: Jordan Slingerland; toasters@teaparty.net Subject: RE: large amount of CIFS other ops from xp desktops after canceled search
Does anyone have any ideas or have seen similar behavior to this?
From: toasters-bounces@teaparty.net [mailto:toasters-bounces@teaparty.net] On Behalf Of Jordan Slingerland Sent: Wednesday, June 05, 2013 3:01 PM To: toasters@teaparty.net Subject: large amount of CIFS other ops from xp desktops after canceled search
Hello toasters,
I am having an issue where I periodically find that some xp desktop users are doing a large amount of other ops to a cifs share with a large amount of files for a sustained period. If I let them go, they will continue for days.
In most cases, it seems the users do a search and the search takes too long, so they close the window, but it is running in some kind of stuck state in the background. Killing the session or closing open files via connection to the toaster via computer management does nothing; the session just starts back up and pounds the crap out of the filler. Rebooting the desktop has been my solution so far, though I suspect restarting explorer.exe may be sufficient.
I am running 8.1.1P2 and this is an IBM rebranded FAS3070. Any ideas? Has anyone seen similar behavior?
cifs top -n 10 ops/s reads(n, KB/s) writes(n, KB/s) suspect/s IP Name 1859 | 0 0 | 0 0 | 0 | 10.10.100.223 domain\someuser2 1418 | 0 0 | 0 0 | 0 | 10.10.100.227 domain\someuser1 1673 | 0 0 | 0 0 | 0 | 10.10.180.230 domain\someuser3
Please be advised that this email may contain confidential information. If you are not the intended recipient, please notify us by email by replying to the sender and delete this message. The sender disclaims that the content of this email constitutes an offer to enter into, or the acceptance of, any agreement; provided that the foregoing does not invalidate the binding effect of any digital or other electronic reproduction of a manual signature that is included in any attachment.
Yes, I believe they are windows searches.
However, I would not expect a search to take more than a few hours, if that. These continue for days. When questioned, the users usually say they did a search earlier and it took too long so they closed the window. Goes away when they reboot, yet, a week later I notice iops climbing so I investigate and find the same thing happened to a different user.
--JMS
-----Original Message----- From: Page, Jeremy [mailto:jeremy.page@gilbarco.com] Sent: Friday, June 28, 2013 4:12 PM To: Jordan Slingerland; toasters@teaparty.net Subject: RE: large amount of CIFS other ops from xp desktops after canceled search
Looks like metadata crawls (which is why the IOPS are so high compared to the read/write throughput.
________________________________________ From: toasters-bounces@teaparty.net [toasters-bounces@teaparty.net] on behalf of Jordan Slingerland [Jordan.Slingerland@independenthealth.com] Sent: Friday, June 28, 2013 4:05 PM To: Jordan Slingerland; toasters@teaparty.net Subject: RE: large amount of CIFS other ops from xp desktops after canceled search
Does anyone have any ideas or have seen similar behavior to this?
From: toasters-bounces@teaparty.net [mailto:toasters-bounces@teaparty.net] On Behalf Of Jordan Slingerland Sent: Wednesday, June 05, 2013 3:01 PM To: toasters@teaparty.net Subject: large amount of CIFS other ops from xp desktops after canceled search
Hello toasters,
I am having an issue where I periodically find that some xp desktop users are doing a large amount of other ops to a cifs share with a large amount of files for a sustained period. If I let them go, they will continue for days.
In most cases, it seems the users do a search and the search takes too long, so they close the window, but it is running in some kind of stuck state in the background. Killing the session or closing open files via connection to the toaster via computer management does nothing; the session just starts back up and pounds the crap out of the filler. Rebooting the desktop has been my solution so far, though I suspect restarting explorer.exe may be sufficient.
I am running 8.1.1P2 and this is an IBM rebranded FAS3070. Any ideas? Has anyone seen similar behavior?
cifs top -n 10 ops/s reads(n, KB/s) writes(n, KB/s) suspect/s IP Name 1859 | 0 0 | 0 0 | 0 | 10.10.100.223 domain\someuser2 1418 | 0 0 | 0 0 | 0 | 10.10.100.227 domain\someuser1 1673 | 0 0 | 0 0 | 0 | 10.10.180.230 domain\someuser3
Please be advised that this email may contain confidential information. If you are not the intended recipient, please notify us by email by replying to the sender and delete this message. The sender disclaims that the content of this email constitutes an offer to enter into, or the acceptance of, any agreement; provided that the foregoing does not invalidate the binding effect of any digital or other electronic reproduction of a manual signature that is included in any attachment.
I did some pktt captures and may have found the cause. I am optimistic... it was happening to 2 ips today and when I was looking through the trace I noticed a pattern that ended on the same file name. Some zip file. I then looked back at a trace I took a month back and found the same file name at the end of what appears to be a loop.
Well, since I could not open this zip file because it is reported by winzip and winrar to be corrupt (and all snapshots of it) I deleted it. The loop is still in progress but this time with the message "STATUS_OBJECT_NAME_NOT_FOUND" rather than "NT Create AndX Request"
I have asked the 2 users currently "looping" on this file to reboot at the end of the day and hopefully...it doesn't happen again.
Side note:
I migrated this share from a large file server vm to a netapp cifs share a few months back using OSSV and snapmirror. I have not deleted that VM yet, I may fire it back up and see if the file is also inaccessible on that system.
--Jordan
-----Original Message----- From: toasters-bounces@teaparty.net [mailto:toasters-bounces@teaparty.net] On Behalf Of Jordan Slingerland Sent: Friday, June 28, 2013 4:15 PM To: Page, Jeremy; toasters@teaparty.net Subject: RE: large amount of CIFS other ops from xp desktops after canceled search
Yes, I believe they are windows searches.
However, I would not expect a search to take more than a few hours, if that. These continue for days. When questioned, the users usually say they did a search earlier and it took too long so they closed the window. Goes away when they reboot, yet, a week later I notice iops climbing so I investigate and find the same thing happened to a different user.
--JMS
-----Original Message----- From: Page, Jeremy [mailto:jeremy.page@gilbarco.com] Sent: Friday, June 28, 2013 4:12 PM To: Jordan Slingerland; toasters@teaparty.net Subject: RE: large amount of CIFS other ops from xp desktops after canceled search
Looks like metadata crawls (which is why the IOPS are so high compared to the read/write throughput.
________________________________________ From: toasters-bounces@teaparty.net [toasters-bounces@teaparty.net] on behalf of Jordan Slingerland [Jordan.Slingerland@independenthealth.com] Sent: Friday, June 28, 2013 4:05 PM To: Jordan Slingerland; toasters@teaparty.net Subject: RE: large amount of CIFS other ops from xp desktops after canceled search
Does anyone have any ideas or have seen similar behavior to this?
From: toasters-bounces@teaparty.net [mailto:toasters-bounces@teaparty.net] On Behalf Of Jordan Slingerland Sent: Wednesday, June 05, 2013 3:01 PM To: toasters@teaparty.net Subject: large amount of CIFS other ops from xp desktops after canceled search
Hello toasters,
I am having an issue where I periodically find that some xp desktop users are doing a large amount of other ops to a cifs share with a large amount of files for a sustained period. If I let them go, they will continue for days.
In most cases, it seems the users do a search and the search takes too long, so they close the window, but it is running in some kind of stuck state in the background. Killing the session or closing open files via connection to the toaster via computer management does nothing; the session just starts back up and pounds the crap out of the filler. Rebooting the desktop has been my solution so far, though I suspect restarting explorer.exe may be sufficient.
I am running 8.1.1P2 and this is an IBM rebranded FAS3070. Any ideas? Has anyone seen similar behavior?
cifs top -n 10 ops/s reads(n, KB/s) writes(n, KB/s) suspect/s IP Name 1859 | 0 0 | 0 0 | 0 | 10.10.100.223 domain\someuser2 1418 | 0 0 | 0 0 | 0 | 10.10.100.227 domain\someuser1 1673 | 0 0 | 0 0 | 0 | 10.10.180.230 domain\someuser3
Please be advised that this email may contain confidential information. If you are not the intended recipient, please notify us by email by replying to the sender and delete this message. The sender disclaims that the content of this email constitutes an offer to enter into, or the acceptance of, any agreement; provided that the foregoing does not invalidate the binding effect of any digital or other electronic reproduction of a manual signature that is included in any attachment.
_______________________________________________ Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters
[cid:image001.png@01CE7421.6676FD60]
-----Original Message----- From: Jordan Slingerland Sent: Friday, June 28, 2013 4:55 PM To: Jordan Slingerland; Page, Jeremy; toasters@teaparty.net Subject: RE: large amount of CIFS other ops from xp desktops after canceled search
I did some pktt captures and may have found the cause. I am optimistic... it was happening to 2 ips today and when I was looking through the trace I noticed a pattern that ended on the same file name. Some zip file. I then looked back at a trace I took a month back and found the same file name at the end of what appears to be a loop.
Well, since I could not open this zip file because it is reported by winzip and winrar to be corrupt (and all snapshots of it) I deleted it. The loop is still in progress but this time with the message "STATUS_OBJECT_NAME_NOT_FOUND" rather than "NT Create AndX Request"
I have asked the 2 users currently "looping" on this file to reboot at the end of the day and hopefully...it doesn't happen again.
Side note:
I migrated this share from a large file server vm to a netapp cifs share a few months back using OSSV and snapmirror. I have not deleted that VM yet, I may fire it back up and see if the file is also inaccessible on that system.
--Jordan
-----Original Message-----
From: toasters-bounces@teaparty.netmailto:toasters-bounces@teaparty.net [mailto:toasters-bounces@teaparty.net] On Behalf Of Jordan Slingerland
Sent: Friday, June 28, 2013 4:15 PM
To: Page, Jeremy; toasters@teaparty.netmailto:toasters@teaparty.net
Subject: RE: large amount of CIFS other ops from xp desktops after canceled search
Yes, I believe they are windows searches.
However, I would not expect a search to take more than a few hours, if that. These continue for days. When questioned, the users usually say they did a search earlier and it took too long so they closed the window. Goes away when they reboot, yet, a week later I notice iops climbing so I investigate and find the same thing happened to a different user.
--JMS
-----Original Message-----
From: Page, Jeremy [mailto:jeremy.page@gilbarco.com]
Sent: Friday, June 28, 2013 4:12 PM
To: Jordan Slingerland; toasters@teaparty.netmailto:toasters@teaparty.net
Subject: RE: large amount of CIFS other ops from xp desktops after canceled search
Looks like metadata crawls (which is why the IOPS are so high compared to the read/write throughput.
________________________________________
From: toasters-bounces@teaparty.netmailto:toasters-bounces@teaparty.net [toasters-bounces@teaparty.net] on behalf of Jordan Slingerland [Jordan.Slingerland@independenthealth.com]
Sent: Friday, June 28, 2013 4:05 PM
To: Jordan Slingerland; toasters@teaparty.netmailto:toasters@teaparty.net
Subject: RE: large amount of CIFS other ops from xp desktops after canceled search
Does anyone have any ideas or have seen similar behavior to this?
From: toasters-bounces@teaparty.netmailto:toasters-bounces@teaparty.net [mailto:toasters-bounces@teaparty.net] On Behalf Of Jordan Slingerland
Sent: Wednesday, June 05, 2013 3:01 PM
To: toasters@teaparty.netmailto:toasters@teaparty.net
Subject: large amount of CIFS other ops from xp desktops after canceled search
Hello toasters,
I am having an issue where I periodically find that some xp desktop users are doing a large amount of other ops to a cifs share with a large amount of files for a sustained period. If I let them go, they will continue for days.
In most cases, it seems the users do a search and the search takes too long, so they close the window, but it is running in some kind of stuck state in the background. Killing the session or closing open files via connection to the toaster via computer management does nothing; the session just starts back up and pounds the crap out of the filler. Rebooting the desktop has been my solution so far, though I suspect restarting explorer.exe may be sufficient.
I am running 8.1.1P2 and this is an IBM rebranded FAS3070. Any ideas? Has anyone seen similar behavior?
cifs top -n 10
ops/s reads(n, KB/s) writes(n, KB/s) suspect/s IP Name
1859 | 0 0 | 0 0 | 0 | 10.10.100.223 domain\someuser2
1418 | 0 0 | 0 0 | 0 | 10.10.100.227 domain\someuser1
1673 | 0 0 | 0 0 | 0 | 10.10.180.230 domain\someuser3
Please be advised that this email may contain confidential information. If you are not the intended recipient, please notify us by email by replying to the sender and delete this message. The sender disclaims that the content of this email constitutes an offer to enter into, or the acceptance of, any agreement; provided that the foregoing does not invalidate the binding effect of any digital or other electronic reproduction of a manual signature that is included in any attachment.
_______________________________________________
Toasters mailing list
Toasters@teaparty.netmailto:Toasters@teaparty.net
-
Jordan Slingerland -
Page, Jeremy