XServe RAID and catalyst

[samsc]

Hugh and myself made some rather surprising discoveries this week, about the performance of a networked XServe RAID connecting multiple Macs using a fibre channel switch:

STILL TO BE CONFIRMED --- BUT

we configured one channel ( one half of xserve raid ) - to drive 3 catalysts.
the one half was set up as 3 separate 2 drive raids.

3 catalyst systems running on dual g5's were then connected to the xserve raid via the Qlogic 5200 fibre switch - everything was connected to a fibre channel switch.

each raid volume called 'Raid 1', 'Raid 2', 'Raid 3' mounted on all the macs! through the switch - without any configuration!
meaning volumes are totally shareable - WITHOUT any extra configuration of anything.
XSAN was not required.

We started off by playing back 4 layers on each catalyst - all worked fine.

-----

i think we might be able to drive up to 6 ( or even 8 ) catalyst media servers - each with 4- 6 layers from a single xserve raid.

Ultimate performance will depend on the data saturation characteristics of the fibre channel switch, and the end user content requirements.

we have not finished testing yet.
im still not yet sure whether this will work so well with long clips, which cannot be completely loaded in the cache.

the key to this is to divide the xserve raid up into smaller drive sets- 2 or 3 drives /raid set.
as reagan initially reported the performance at standard definition is not better with 7 drives than with 2 - a 7 drive raid set can playback 7-8 layers - but so can a 2 volume raid set.

this is because playing back 6-7-8 layers the bottleneck problem becomes a processor limitation.


So dividing up 14 drives in the biggest xserve raid will give you 3 raid sets/channel, each with 2 or 3 drives.

Each Catalyst mac media server runs from its own RAID set - so the data access for each server does not collide across raid volumes.
But all volumes are readable on all macs networked through the fibre channel card.

The fibre channel system means that you have actually directly networked all your volumes together.

This has an enormous potential to help in the content management of large shows, as any mac connected to the fibre channel switch can see all RAID volumes.

[samsc]

Xserve RAID

http://www.apple.com/xserve/raid/

Fibre channel switch

http://www.qlogic.com/products/fc_san_switchs.asp

other companies make these - we tested the 5200

[samsc]

More Xserve RAID info:
http://images.apple.com/xserve/raid/pdf/20041104_XserveRAID_techoverview.pdf

http://images.apple.com/xserve/raid/pdf/20041104XserveRAID_datasheet.pdf

more fibre switches:

http://www.sandirect.com/brocadepage.htm
http://www.sandirect.com/emcswitches.htm

fibre channel info
http://www.storagesearch.com/fcswitches.html

[samsc]

I drew a diagram of the proposed network layout.

Each mac should ideally access files only on its own RAID set- to maximise performance, but all volumes will mount on desktops of all macs.


----

We dont have multiple XServe RAIDS to test but we believe that multiple units can also be plugged directly into the fibre channel switch, and shared between computers.

[samsc]

Is the XServe RAID expensive?

If you consider that each single XServe RAID - configured as 6 RAID sets, you get 4 volumes with 800GB and 2 volumes with 1200GB.
You are getting a LOT of storage capacity in one unit.5.6TB


The retail price of the largest XServeRAID is $13000.
Using this to drive 6 Catalysts is then ~$2000/server.
Add 8 port Fibre switch. $2000
Fibre channel cards are about the same cost as scsi cards

A SCSI disc array with 800GB capacity using 140GB drives at $600/drive is 5*600= $3000

The XServe RAID is 50% cheaper per RAID set for the same drive size than using SCSI.
And you can network this between multiple macs, you cannot do this with SCSI.

[Jan Opseth]

Hi,

Which Raid Levels did you run the Xserv on?

Regards,

Jan Opseth

[samsc]

Hi,

Which Raid Levels did you run the Xserv on?

Regards,

Jan Opseth
RAID 0.

No redundancy

[jasonrudolph]

Another possible nice thing with this, is that you can rent Xserve RAID systems from a number of vendors....

[samsc]

Another possible nice thing with this, is that you can rent Xserve RAID systems from a number of vendors....

if you do rent-
Make sure you have enough time to configure the raid. it takes a day or so to change the raid setup!


I like it because really it was almost plugnplay.
Fibre channel cards didnt need configuring, fibre switch didnt need configuring.
Maybe we got lucky.

[Jan Opseth]

RAID 0.

No redundancy


I know you folks have planty to but do you have the time/"wanto" too
test raid 5? to see any performance drops/or rices?

Can u change stripe sizes on an Xserv Raid?


Regards,

Jan O

[samsc]

I know you folks have planty to but do you have the time/"wanto" too
test raid 5? to see any performance drops/or rices?

Can u change stripe sizes on an Xserv Raid?


Regards,

Jan O

No change in performance with raid5 - the demo unit arrived as RAID 5- i got reagan to reformat at raid0 - no difference noted.
Only tested at standard definition.
raid setup doesnt matter that much when you arent trying to get 200MB/s out of the XServe RAID.

i think you can change stripe size - it takes days to reinitialise - so we didnt change too much.


Playback at hi-def is processor limited at the moment - ( but might not be if apple released quad computers )

[marty@cwp.net]

Richard,

I spoke with Hugh yesterday. We were talking about another issue, but this is extremely interesting to me. Please keep me informed on this, as I have an immediate need for this. I am looking to do something along this line in Mid May. IF this works, this could be the answer.

Regards,

Marty

[samsc]

Im trying to get one on a big show right now- i have a show with lots of servers that can saturate this thing-

i can check out performance under real conditions.

Ill keep you informed of developments.

[tylerr]

sounds like a great solution for an install. But for road machines I like to keep the unit self contained in its case. The only wires that come out are for power, monitors, keyboard, and video out. The thought of a crew member, LD, or Catalyst Op accidently stomping on a fiber lead scares me.

But I am a chicken!

Tyler

[samsc]

sounds like a great solution for an install. But for road machines I like to keep the unit self contained in its case. The only wires that come out are for power, monitors, keyboard, and video out. The thought of a crew member, LD, or Catalyst Op accidently stomping on a fiber lead scares me.

But I am a chicken!

Tyler

its not fibre - its copper.

its seems a lot easier to configure than scsi.

[samsc]

Fibre channel cables:
http://www.cselex.com/fibre-channel-cables.htm

Some fibre channel info.

http://www.seagate.com/support/kb/disc/fibre_channel.html


Fibre Channel key features include...

Hot-pluggability ? Fibre Channel drives can be installed or removed while the host system is operational, which is crucial in high-end and heavy-use server systems where there is little or no downtime.

ANSI standard compliance for serial port interface ? Fibre Channel does not require special adapters, which can be expensive.

Speed ? In its intended environment, Fibre Channel is the fastest option available.

Cost effectiveness ? Relative to other high-end solutions, Fibre Channel is inexpensive because it does not require special adapters.

Loop resiliency ? Fibre channel provides high data integrity in multiple-drive systems, including Fibre Channel RAID.

Longer cable lengths ? Relative to LVD, Fibre Channel can maintain data integrity through significantly longer cables. This makes configuring multiple devices easier.

fc switches:
http://www.storagesearch.com/fcswitches.html

[samsc]

Apple xserve cables:
http://www.cselex.com/apple-xserve-cables.htm

You can use fibre optical cables with these devices for long distances - but you dont have to,
the SFP plugs can take optical transceivers.

http://www.apple.com/xserve/raid/fibre_channel.html


For connecting over longer distances Apple?s card accepts long and short haul SFP (Small Form Factor Pluggable) optical transceivers for use with fiber-optic cables that can extend over 500 meters. This is particularly useful for organizations using Xserve RAID to back up or archive critical data.

[tharding]

How does this all relate to the transactions/second argument?

I am looking for disk solution at the moment beyond my single Raptor content drives and am a little confused now as to what is best. I have gone back through the forums and looked at your drive/codec comparison tables as well as the previous discussions about SATA v. SCSI. This fibre channel discovery seems to conflict with that.

Do I spend the money on internal SCSI drives for my 2 machines or am I better off investing more in something expandable such as xRAID?

I would like to have the ability to play back Hi Def but the maximum number of layers is also important.

Cheers

Toby

[samsc]

i have a couple of people testing xserve raids.
and going through the issues.

they are quite good, because they are expandable, but the cost is a bit higher.

i hope to get one on a tour this month.

[tharding]

Okay,

Given that the xserve raid is probably a little out of my price range at the moment I am considering 2 choices that will cost about the same.

Option 1 - a SATA array using the new rackmount enclosure from Macgurus and putting the 8 channel SATA card in each of my G5's. This gives me the option to run a 4 drive raid for each machine or an 8 drive array for one machine. I would use raptors as the drives.

Option 2 - Buy a the Atto scsi card for each machine and put an Atlas V 146 gig drive in each. This option giving me the chance to buy a Huge systems array further down the track.

Please let me know what you think is the best option.

Cheers

Toby

[samsc]

i never tested a sata array. the last generation were crappy.
the connectors were crappy, and christian used them, and always seemed to have issues.

if you start doing multi-disc scsi, the price soon starts creeping up to xserve levels.

it just isnt possible to acquire and test all the storage options, and evaluate their performance with catalyst.
---

the main advantage of xserve RAID is networkability- you can share the storage at very high speed.
Using a fibre switch and XSAN you can create multiple disc volumes in the same unit and share them between different servers - thats the theory.

With the cheapest xserve raid which ships with 4 discs you would divide the thing up into 2 two disc raid sets, with the most expensive it might be possible to drive 6 computers at the same time.

I have a 14 disc xserve raid on a show at the moment, and im waiting for results.

[samsc]

Apple fibre channel and xserve raid support page:

http://www.apple.com/support/xserve/raid/

[samsc]

http://www.apple.com/support/downloads/fibrechannelutility20.html

[jasonrudolph]

What about RAID setups that connect via gigabit ethernet? Would these work well with catalyst? If you are using machines with the CIB on them, and don't need to network them for syncing them? Or could you add another Gigabit card......

[samsc]

What about RAID setups that connect via gigabit ethernet? Would these work well with catalyst? If you are using machines with the CIB on them, and don't need to network them for syncing them? Or could you add another Gigabit card......

ethernet is not optimised for any kindof real time networking.
it has a good headline data rate, but terrible latency.
And apples file sharing isnt optimised to do this either.

Ethernet based networks for use with catalyst- IMHO - has to be about the worse possible thing.

Fibre channel and the hardware to support it -is designed to do this kindof thing, in real world video networks.

[jasonrudolph]

I have a 14 disc xserve raid on a show at the moment, and im waiting for results.


So, how did it perform???

[samsc]

eurovision never got around to using it.

they had too many other issues.

its coming back to me soon. hopefully this week.
im trying to get it onto another gig.

initial tests showed that 6-7 layers dv layers were achievable on a 2 drive raid set.
this is due to agressive caching rather than drive speed.
xserve raid caching worked better than other raid technologies so far

[tharding]

Hi Richard

Just wondering what the latest is on using FC switches. Has anyone used the large scale 14 drive system you mentioned?

I notice Qlogic have the 1400 now which is quite cheap.

http://www.qlogic.com/products/sanbox/sanbox_1400.asp

Any thoughts on if this would work compared to the 5200?

Or are there any other cheaper units you are aware of that are worth a look?

Cheers

Toby

[samsc]

i dont know.