toasters

toasters@lists.teaparty.net

1 participants
13515 discussions

NFS client tuning, problem and resolution.
by Nick Christenson 14 Sep '99

14 Sep '99

Recently, we encountered a serious problem in a high performance NFS environment using large Solaris servers as NFS clients. Due to the fantastic assistance of several people who have elected to not be named, we have identified the problem and a resolution for it. Since others on this list are likely to use Solaris servers as NFS clients in high performance environments, it was thought that this information might be generally useful. Situation and Symptoms: We were running on E class Sun's using Solaris 2.6 with what we thought were well considered kernel tuning parameters (cranked up nfs:nfs_max_threads, maxusers, rlim_fd_{max,curr}). The disks were using Sun's Gigabit Ethernet interfaces connecting to NetApp F760s using a high quality Gb Enet switch. When we turned the application on, we got stunningly low throughput numbers, and couldn't figure out why. Client CPU load was very good, most of the processing power was idle (and not in WAIT state), memory utilization was low, the filers were running at low utilization, there were no network or I/O bottlenecks in evidence. We were running two applications that do reads, writes, appends, creates, and deletes on relatively small files over NFS. One of these applications, and both have fairly similar characteristics and perform computations on the same file sets, was running just fine. The other was running well at light loads, but had horrible problems as the load went up. It appeared that some fixed resource was being consumed, and once we went over a certain load threshold, the number of processes grew exponentially while the amount of work being done remained constant. Eventually, these extra processes exhausted main memory and the machine began to thrash. Solution: Since we don't have access to Sun source, we can't be 100% certain of what was happening, but this is the best information we have. We think what we describe here is exactly what's going on, but there might be some minor variations. If someone here knows Sun internals, maybe they can fill in the gaps. Basically, it seems that we were having problems with the Directory Name Lookup Cache (DNLC) in Solaris. It seems that between Solaris 2.5.1 and Solaris 2.6 the math was changed on how this was calculated. Here's the math, as best I know it: 2.5.1 DNLC size = (max_nprocs + 16 + maxusers) + 64 2.6 DNLC size = 4 * (max_nprocs + maxusers) + 320 According to Cockroft's Sun Performance Tuning book, max_nprocs = 10 + 16 * maxusers We don't know if this calculation has changed between 2.5.1 and 2.6 Wanting large numbers of processes and large buffer sizes, we set maxusers=2048. This means that for Solaris 2.5.1, the DNLC size would be 34906 (or so) and for Solaris 2.6, the same maxusers variable would yield a size of 139624. Now, as we understand it, this is a LINEAR lookup table. Further, when deleting a file, an entry in this table must be looked up and locked. This was the distinction between our two processes, one does a single delete before completion, the other does three deletes. With a table this size, there seems to be a finite number of deletes/second one can perform over NFS, and we hit that limit. We put "set ncsize = 8192" in /etc/system, rebooted, and the problem went away. We played with sizes ranging from 4096 to 32768 and saw no huge performance difference, (8192 SEEMED best, 32768 SEEMED worst, but that was a VERY subjective evaluation), and we saw no significant difference to our DNLC cache hit rate as measured by "vmstat -s". Additional Information: One way you can look to see if you're experiencing this problem is to run "/usr/ucb/ps -uaxl" and look at the WCHAN column. To the best of our knowledge, Sun doesn't publish a translation guide to these event names, but we have it on good authority that if you see one called "nc_rele_" (which is a truncation of "nc_rele_lock") the process is waiting for DNLC entries to become unlocked. Note: On healthy machines processes will sometimes show up in this state, but if a significant percentage of them are in this state, that may indicate this problem. No, I can't accurately define "significant state". I doubt 1% is a problem, I know that 30% or higher is a problem. Also, Sun has a BugId for this problem, 4212925. As of August 2, they have a patch for 2.6, numbered 105720-08. It seems to do a lot of things, but the explanation of it isn't as revealing as we'd like, so we're a little leery of putting it into production without extensive testing. We're playing with it, but can't comment on it's efficacy at this time. It might hash the DNLC table (That would be the right solution, and word is this will happen in 2.8), but for all I know it may just revert to 2.5.1's DNLC math. Summary: If you're running Solaris 2.6 or 2.7 in a high performance environment where a lot of files are being deleted over NFS, make sure your DNLC is not too large or you'll have HUGE problems. Trust me. Hope this helps someone. -- Nick Christenson npc(a)sendmail.com

5 4

RE: Best way to copy data to new filer ??
by Melvin, Grant 14 Sep '99

14 Sep '99

Hi David, Check out the "Migrating Data Between NetApp Filers" technical report (#3018) at: http://www.netapp.com/tech_library/3018.html Hope this helps. Cheers, Grant -----Original Message----- From: David H. Brierley [mailto:dhb@ssd.ray.com] Sent: Tuesday, September 14, 1999 9:19 AM To: toasters(a)mathworks.com Subject: Best way to copy data to new filer ?? I am getting ready to convert from an F220 and an F230 to an F740. Does anyone have any recomendations on how to copy all of the data from the two old filers to the new one? I know I can use dump and restore but I was wondering if maybe "vol copy" would be more efficient. Can I use "vol copy" to copy the entire contents of two existing filers to a single new one? If so, can it merge the contents into a single volume or should I create multiple volumes? -- David H. Brierley Raytheon Systems Company - Portsmouth RI Facility Work: dhb(a)ssd.ray.com Home: dave(a)galaxia.com

1 0

Upgrades
by Jay Orr 13 Sep '99

13 Sep '99

I had recently done an upgrade from 4.2a to 5.3.2r1, and now when it is backing up, Budtool reports : 09/13/99 09:31:46> DUMP: creating "/vol/vol0/../snapshot_for_backup.7" snapshot. 09/13/99 09:32:08> ------------------------------------------------------------------ 09/13/99 09:32:08> ------ W A R N I N G ------- 09/13/99 09:32:08> Unexpected log entries 09/13/99 09:32:08> DUMP: Using subtree dump 09/13/99 09:32:08> ------------------------------------------------------------------ 09/13/99 09:32:09> DUMP: Date of this level 1 dump: Mon Sep 13 09:32:07 1999. 09/13/99 09:32:09> DUMP: Date of last level 0 dump: the epoch. 09/13/99 09:32:09> DUMP: Dumping /vol/vol0/ to standard output Last backup was a level 2 and it did the same (redid level 0). Can anyone tell me what's going on here? ----------- Fujitsu - Nexion, St. Louis, MO Jay Orr (314) 579-6517

2 2

NetApp versus Dell
by Colm Buckley 13 Sep '99

13 Sep '99

Quickie from someone in the last stages of speccing and sourcing filer solutions... has anyone got any experience of the Dell PowerVault NAS systems? They look very much like re-badged NetApp F720-F760s, with one or two small differences. Anyone have any points either pro or con these systems, versus the NetApp equivalents? Colm -- Colm Buckley B.A. B.F. # colm(a)tuatha.org colm.buckley(a)cs.tcd.ie Department of Computer Science # +353 87 2469146 # whois cb3765 Trinity College, Dublin 2, Ireland. # http://www.tuatha.org/~colm/ Don't put all your hypes in one home page.

2 1

Q on disk firmware level
by George Kahler 13 Sep '99

13 Sep '99

Could some kind soul who is running ST39173W disk drives (probably on a F520) tell me what the disk's firmware level is? Much appreciated, George ------------------------------------------------------------------------------- George Kahler e-mail: george(a)yorku.ca Sr. Systems Administrator humans: (416) 736-5257 x.22699 Computing Services, York University machines: (416) 736-5830 Ontario, Canada, M3J-1P3

1 0

Re: NetApp versus Dell
by sirbruce＠ix.netcom.com 13 Sep '99

13 Sep '99

On 09/13/99 14:48:27 you wrote: >At present they are re-badged systems with different shelves. They were >much cheaper here than NetApp's though I don't feel that Dell have >their routines completely sorted out yet so there can be some logistic >hassles with your purchase. IIRC their European plant is in Ireland so >you may have an advantage there. Dell seem best at delivering shrink >wrapped solutions. I have heard of problems with Dell sales & support as well. For instance, one customer who specifically said they wanted to use a Dell system to support their Microsoft SQL database wound up going with SAN and another vendor because they were not informed of Microsoft allowing NAS support for their SQL database on Netapp filers specifically. Bruce

1 0

Re: NetApp versus Dell
by Michael Salmon 13 Sep '99

13 Sep '99

+----- On Mon, 13 Sep 1999 13:11:16 BST, Colm Buckley writes: | | Quickie from someone in the last stages of speccing and sourcing filer | solutions... has anyone got any experience of the Dell PowerVault NAS | systems? They look very much like re-badged NetApp F720-F760s, with one | or two small differences. | | Anyone have any points either pro or con these systems, versus the | NetApp equivalents? At present they are re-badged systems with different shelves. They were much cheaper here than NetApp's though I don't feel that Dell have their routines completely sorted out yet so there can be some logistic hassles with your purchase. IIRC their European plant is in Ireland so you may have an advantage there. Dell seem best at delivering shrink wrapped solutions. Pro: much cheaper. Con: Dell is an uncertain quantity as yet. /Michael

1 0

Re: maxfiles increase - good rule of thumb?
by sirbruce＠ix.netcom.com 13 Sep '99

13 Sep '99

On 09/12/99 21:09:53 you wrote: > >Greetings, > >We're about to hit 100% in inode usage again and this time, I want to >increase `maxfiles' to a more reasonable level than the current 1 per 32KB >allocation. > >The average file size on our 216GB volume is ~4KB. But I think increasing >maxfiles by 8 times might be a bit risky. Or maybe not. Anyone ever >increase maxfiles from 1 per 32KB to 1 per 4KB? > >Just to play it safe, I'm thinking of doing it in 4KB increments... first >increase to 1 per 28KB and then work my way up if I need it. > >Suggestions or testimonials? I don't think this is the best approach to use. Rather, consider how full your filesystem currently is. If you're using 50% now, then a simply extrapolation is to simply double the current maxfiles number (don't think in terms of inodes/KB) and you should be fine. Or, if you don't want the hassle of monitoring your maxfiles, consider the max expansion for your filer and adjust upward based on current usage. The space/performance penalty for having a maxfiles a few times larger than you actually use is minimal. Personally, whenever I've had a filer go over 90% inode usage, I would generally just bump maxfiles up by 50% (total) if the disk is also close to getting full, or double it if I have a new filer that I am planning to add a lot more to in the near future. Bruce

1 0

maxfiles increase - good rule of thumb?
by Kevin Ying 12 Sep '99

12 Sep '99

Greetings, We're about to hit 100% in inode usage again and this time, I want to increase `maxfiles' to a more reasonable level than the current 1 per 32KB allocation. The average file size on our 216GB volume is ~4KB. But I think increasing maxfiles by 8 times might be a bit risky. Or maybe not. Anyone ever increase maxfiles from 1 per 32KB to 1 per 4KB? Just to play it safe, I'm thinking of doing it in 4KB increments... first increase to 1 per 28KB and then work my way up if I need it. Suggestions or testimonials? ~~~ Kevin Ying / Sysadmin Team Leader interQ, Inc. kevin(a)interq.or.jp Infoss Tower 10/F (vox) +81-3-5456-2555 20-1 Sakuragaoka-cho (fax) +81-3-5456-2556 Shibuya-ku, Tokyo 150-0031, JP

2 1

Re: local vs. network backup for F740's
by sirbruce＠ix.netcom.com 11 Sep '99

11 Sep '99

On 09/10/99 10:50:46 you wrote: > >Hi all. > >We have several F740's with about 400GB each, and we are reexcamining >our backup strategy. I would prefer a centralized, network backup over >a dedicated backup network. I don't understand the difference between the two. I mean you can have both. I suppose by the former you mean doing it over your regular network instead of a new one, which is fine, so long as you have the bandwidth for it. >We've heard rumours that the only >reasonable way to backup filers is using a locally attached DLT drive, >and that we must buy an expensive ($700) NetApp SCSI controller card >before we attempt to do a local backup. AFAIK, the 740 should have come with a tape backup card that supports something like 2 tape drives. Just make sure you have the right kind of SCSI for it. >Does this make sense to you >all? What is the conventional wisdom on the ideal way to backup >multiple filers? Personally, I prefer the local backup, depending on your backup scheme. You can back up everything at once without drive or bandwidth contention and the cost is not outrageous. But if you want to use network backup, I think that is fine too. However I would be wary of using NDMP, which historically has had problems. Then again, if you are using a centralized tape robotic and want all the features and stuff, NDMP may be your only choice. Bruce

1 0

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

1999

1998

1997

toasters