The answer to the question posed in the subject line is
There is no support for multicast in ONTAP releases prior to 6.0. 6.0 has multicast code to handle, for example, RIPv2; however, some network interface drivers, such as the Alteon driver, didn't have multicast support added to them. (I don't know which drivers do; you should probably ask NetApp support that question.)
The only ways to make that configuration work would, as far as I know, either be to
1) install 6.0 on the filers
or
2) find some way of doing the failover checks that doesn't involve sending multicast packets to the filer (which may mean not using the Solaris failover stuff).