Hi folks,
I'm at a loss. I have an unmaintained (fas2020) 7-mode HA pair in a site that's slated to be decommed, but for now, I need to keep the lights on. There is a qtree snapmirror that, aside from Mondays, runs in about 3-4 hours. Mondays, it takes 2-4 days. The change rate on the source doesn't appear to be any different. Also, this was working happily for years and only started acting up lately. There are only 5 disks on the system, and while they're at 100% now, they're also at 100% during a sample taken while this context wasn't updating, and while it was updating in a way that finished in 4 hours.
So far, I've checked statit during the event and while idle, and I couldn't see anything that stuck out as a massive difference, except maybe name cache hits? I have 68% name cache hits during this transfer, and 100% while idle.
I don't see anything suspicious in the wafl scan status, and reallocate status -v shows:
Reallocation scans are on No reallocation status.
CPU is idle, network has no errors. Does anyone have any suggestions? What kinds of things should I be looking at?
Thanks,
Basil
************************************************************************* This message and any attachments (the "message") are confidential, intended solely for the addressee(s), and may contain legally privileged information. Any unauthorized use or dissemination is prohibited. E-mails are susceptible to alteration. Neither SOCIETE GENERALE nor any of its subsidiaries or affiliates shall be liable for the message if altered, changed or falsified. Please visit http://sgasdisclosure.com for important information regarding SG Americas Securities, LLC (“SGAS”). Please visit http://swapdisclosure.sgcib.com for important information regarding swap transactions with SOCIETE GENERALE. *************************************************************************
Are there any other snapmirror qtrees happening on the same volume on Mondays. If yes the locks on snapshots might be the problem.
J
On 12/13/2016 11:04 AM, BERNTSEN Basil wrote:
Hi folks,
I'm at a loss. I have an unmaintained (fas2020) 7-mode HA pair in a site that's slated to be decommed, but for now, I need to keep the lights on. There is a qtree snapmirror that, aside from Mondays, runs in about 3-4 hours. Mondays, it takes 2-4 days. The change rate on the source doesn't appear to be any different. Also, this was working happily for years and only started acting up lately. There are only 5 disks on the system, and while they're at 100% now, they're also at 100% during a sample taken while this context wasn't updating, and while it was updating in a way that finished in 4 hours.
So far, I've checked statit during the event and while idle, and I couldn't see anything that stuck out as a massive difference, except maybe name cache hits? I have 68% name cache hits during this transfer, and 100% while idle.
I don't see anything suspicious in the wafl scan status, and reallocate status -v shows:
Reallocation scans are on
No reallocation status.
CPU is idle, network has no errors. Does anyone have any suggestions? What kinds of things should I be looking at?
Thanks,
Basil
This message and any attachments (the "message") are confidential, intended solely for the addressee(s), and may contain legally privileged information. Any unauthorized use or dissemination is prohibited. E-mails are susceptible to alteration. Neither SOCIETE GENERALE nor any of its subsidiaries or affiliates shall be liable for the message if altered, changed or falsified. Please visit http://sgasdisclosure.com for important information regarding SG Americas Securities, LLC (“SGAS”). Please visit http://swapdisclosure.sgcib.com for important information regarding swap transactions with SOCIETE GENERALE.
Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters
Check the etc/log/snapmirror file on the destination filer for clues.
J
On 12/14/2016 03:28 PM, Joel Krajden wrote:
Are there any other snapmirror qtrees happening on the same volume on Mondays. If yes the locks on snapshots might be the problem.
J
On 12/13/2016 11:04 AM, BERNTSEN Basil wrote:
Hi folks,
I'm at a loss. I have an unmaintained (fas2020) 7-mode HA pair in a site that's slated to be decommed, but for now, I need to keep the lights on. There is a qtree snapmirror that, aside from Mondays, runs in about 3-4 hours. Mondays, it takes 2-4 days. The change rate on the source doesn't appear to be any different. Also, this was working happily for years and only started acting up lately. There are only 5 disks on the system, and while they're at 100% now, they're also at 100% during a sample taken while this context wasn't updating, and while it was updating in a way that finished in 4 hours.
So far, I've checked statit during the event and while idle, and I couldn't see anything that stuck out as a massive difference, except maybe name cache hits? I have 68% name cache hits during this transfer, and 100% while idle.
I don't see anything suspicious in the wafl scan status, and reallocate status -v shows:
Reallocation scans are on
No reallocation status.
CPU is idle, network has no errors. Does anyone have any suggestions? What kinds of things should I be looking at?
Thanks,
Basil
This message and any attachments (the "message") are confidential, intended solely for the addressee(s), and may contain legally privileged information. Any unauthorized use or dissemination is prohibited. E-mails are susceptible to alteration. Neither SOCIETE GENERALE nor any of its subsidiaries or affiliates shall be liable for the message if altered, changed or falsified. Please visit http://sgasdisclosure.com for important information regarding SG Americas Securities, LLC (“SGAS”). Please visit http://swapdisclosure.sgcib.com for important information regarding swap transactions with SOCIETE GENERALE.
Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters
because the disks were at 100% while the snapmirror is slow and also 100% while the snapmirror finished in an acceptable time frame, does not mean you have a similar workload. there is nowhere to go from there. You cant get many iops out of 5 disks, my first though would be to look at the clients, see which one is the hog and find out what it is doing. I suppose the first thing I would do is see what protocol is doing the i/o and go from there.
sysstat -x 1
might be useful for starters.
--JMS
On Tue, Dec 13, 2016 at 11:04 AM, BERNTSEN Basil < basil.berntsen-ext@socgen.com> wrote:
Hi folks,
I'm at a loss. I have an unmaintained (fas2020) 7-mode HA pair in a site that's slated to be decommed, but for now, I need to keep the lights on. There is a qtree snapmirror that, aside from Mondays, runs in about 3-4 hours. Mondays, it takes 2-4 days. The change rate on the source doesn't appear to be any different. Also, this was working happily for years and only started acting up lately. There are only 5 disks on the system, and while they're at 100% now, they're also at 100% during a sample taken while this context wasn't updating, and while it was updating in a way that finished in 4 hours.
So far, I've checked statit during the event and while idle, and I couldn't see anything that stuck out as a massive difference, except maybe name cache hits? I have 68% name cache hits during this transfer, and 100% while idle.
I don't see anything suspicious in the wafl scan status, and reallocate status -v shows:
Reallocation scans are on
No reallocation status.
CPU is idle, network has no errors. Does anyone have any suggestions? What kinds of things should I be looking at?
Thanks,
Basil
This message and any attachments (the "message") are confidential, intended solely for the addressee(s), and may contain legally privileged information. Any unauthorized use or dissemination is prohibited. E-mails are susceptible to alteration. Neither SOCIETE GENERALE nor any of its subsidiaries or affiliates shall be liable for the message if altered, changed or falsified. Please visit http://sgasdisclosure.com for important information regarding SG Americas Securities, LLC (“SGAS”). Please visit http://swapdisclosure.sgcib.com for important information regarding swap transactions with SOCIETE GENERALE.
Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters
-
BERNTSEN Basil -
Joel Krajden -
jordan slingerland