On Wed 23 Feb, 2000, "Mohler, Jeff" <jeff.mohler(a)netapp.com> wrote:
> I dont have any solid data on data/stripe/drivesize performance handy, but it is
> a wise move to not span raid-groups over multiple FCAL controllers. There will
> be write degredation involved with that that I have seen. With two F760s
> rebuilding a manually failed drive while doing nothing else, the machine without
> a stripe split across FCAL cards completed about 25% before the machine that did
> span FCAL cards for a raid stripe.
>
> mkfile tests from Solaris confirmed what Id heard..and saw in the reconstruct as
> part of my test.
This strikes me as very interesting. Until now I'd have reasoned that
having two times the bandwidth to the spindles, and the lower latency
that implies as well, (having only a proportion of io's in flight on each
channel getting in the way of subsequent io's), would have benefitted
RAID rebuilding - which is the nastiest case where all stripes must be
read, XOR'd-over and blocks written to one disk.
Can anyone explain why this would be so?
>-- End of excerpt from "Mohler, Jeff"
On Wed 23 Feb, 2000, "Bruce Sterling Woodcock" <sirbruce(a)ix.netcom.com> wrote:
> Probably not until late this year or next.
Fair enough. I imagine they might appear with new product, as it's been a
while.. *speculates wildly*
> The complete model depends entirely on your filer model, the OS version,
> exactly what disks you have, how much memory, what your op mix is, etc.
Well, yes, but that's not impossible to plug into an analytical model.
I'd also hope that these low-level functions were pretty finely tuned
in DOT and not as amenable to radical change..
> The make very little difference. On the order of 10% for most people.
> As to WHY, it's because Netapp finely balances their filers for maximum
> performance at typical op mixes. They are not like a competitor, which
> throws a bunch of hardware at a problem and you are expected to juggle
> it to figure out what works. Netapp wants to deliver a simple solution.
Yeees. However, there's a touch of faith creeping in here. I don't
think it's misplaced, certainly, I think NetApp's are my favourite
fileservers for these reasons. I guess I'm just getting more cycnical
and evidence-biased as I get older. 8)
> There are some performance papers on Netapp's web site that address
> some of your questions, and people are free to post their personal
> experiences here, but what works for one environment may not work
> for another. I would be reluctant to tell you to go out and buy more
> drives you don't need if it won't help you.
Indeed, me too. But how can we quantitatively be confident ahead of time?
> Given that those tables constantly change with the variables (drives,
> memory, OS, etc.) rules of thumb is probably the best you can do, unless
> you have a few million to spend on testing every possible configuration
> and reporting the results. :)
Well I know only one place and one group of people that could undertake
this, and they sell the boxes. I'd be very susprised if the creators,
the engineers, haven't got *something* going on in this line.
> There is no best. There is only what is best for your environment.
Yes, I'd like to think my 'best' would be considered as predicated on
the requirements. I'm not an absolutist, or at least I try not to be.
> I would guess slightly better, since the 18 gig drives are probably faster
> and you have 2 loops to share the load. However, since the bottleneck
> will be the NVRAM, I doubt there will be a major noticeable difference.
Ah, I was thinking more about this - the nvram emptying after a cp takes
time and that is a key determinant in whether you'll be taking a cp from
a cp or not, in turn being a key determinant in user-visible performance.
This nvram emptying is surely determined in turn by how rapidly the
blocks can be pushed to disk, in which case the speed of the disks and
the utility of multiple loops may have some impact, surely? Or is the
WAFL so good at distributing writes, and the bandwidth and latency of
a single loop such a good match for the nvram that this is a null-term?
> I can't consider them if you don't say what they are.
I meant, consider whether they're a big factor or not. I was thinking
about the two scenarios using the same 10-spindle RAID set size.
In the average case the DOT algorithm has to read a few blocks from
each of a few disks from a given RAID set, perform the parity XOR over
those blocks and the blocks it intends to write, generate the parity,
and then write all those new blocks to the disks that it has elected to
write to during a cp flush, always including the parity disk. Having
fewer spindles means more load per spindle, having fewer loops means
more load per loop. How much does the cpu have to do, how much can be
DMA'd or otherwise fobbed-off. How little time can you whittle the
nvram empty down to?
I mean, to be reliable you have to await the completion notice from each
disk for each io sent, and I don't believe 18GB disks to be capable of 2x
the io's of 9GB disks. Serialising 4 RAID sets on one loop though, even
with more disk-ios/second, will probably increase the final-response time
to do a full nvram flush by a goodly chunk over the scenario with 2 loops.
Given a pathalogical filer setup I could see that increasing the nvram
on a write-burdened filer might actually make the problem worse rather
than better. How does an admin, especially one not so very interested
and aware of these things, approach a filer that is struggling but they
don't know why or how to fix it?
> You can't know if you got it right unless you have an exact simulation
> of your particular ops mix and traffic patterns. Since you don't have
> that, then basically you have to use rules of thumb and guess and adjust
> when needed. Your environment changes over time, too, so what works
> one day may not work another.
Mmm, all of which makes me lean toward an analytical model, as it should
be more powerful in dealing with these imponderables: plug in the numbers
and see how it comes out. Real testing and simulation would make useful
checks against such a model, natch.
> Maybe it's just me, but while I love having numbers and data, I've always
> found such tuning tasks to be far more intuitive and situational.
Well, yes, I guess this is because we're human and quite good at dealing
with vague and wooly things that nevertheless have patterns that can be
extracted with experience.
> I mean, if
> I have a chart that says such-and-such filer with such-and-such configuration
> can handle 200 users, and the filer in my environment overloads at 100
> users, I'm not gonna keep adding users. Conversely, if it runs at 50% and
> high cache age with 200 users, I'm not going to worry about adding more.
True. Computers aren't as deterministic as they once were and we have to
deal with what's really there, however I think there is scope for better
models than none-at-all, or 'ask Bruce or Mark, they're good with filers'.
I also wouldn't advocate *such* a simple model, at least not alone.
> If a filer is overloaded with reads, and I haven't maxxed the RAM yet, I'm
> probably going to max the RAM first. Then I'll worry about adding more
> disks, and if that doesn't help, I'll know I need to reduce traffic and get
> another filer. There are some minor variations to this, of course, but I'm
> not going to waste a lot of time beforehand trying to predict exactly how
> many disks and how much memory I need when the reality can quickly
> change. Estimate, yes, but I won't follow a strict chart.
Okey, so perhaps I've overemphasised the model and chart thing, or
perhaps you're playing devil's advocate to my credulous believer. As a
tool in the salesman's, field engineer's and admin's box'o'tricks though,
a configurator other than than the SPEC result-sheets would be a very
neat thing to have.
And as for wasting a lot of time beforehand - that's rather my point. You
wouldn't have to waste time, or a lot of it.
So long as the results of a model were within a few tens of percent then
your putative admin has gotten close enough to start tuning, rather
than transhipping for the larger machine they thought they might need
but chickened out of. Better, they find they're peaking just below
the capability of the box they ordered and didn't cough up for the
big-expensive beast they'd otherwise have bought "just in case".
>-- End of excerpt from "Bruce Sterling Woodcock"
--
-Mark ... an Englishman in London ...