AMD Processors
Decrease font size
Increase font size
Topic Title: Dual dual-core Opteron rig
Topic Summary:
Created On: 12/22/2005 06:34 PM
Status: Read Only
Linear : Threading : Single : Branch
Search Topic Search Topic
Topic Tools Topic Tools
View similar topics View similar topics
View topic in raw text format. Print this topic.
 12/22/2005 06:34 PM
User is offline View Users Profile Print this message

Author Icon
CFD2
Junior Member

Posts: 3
Joined: 12/22/2005

Hi all

I'm going to use this rig purely as a compute server for CFD simulations http://en.wikipedia.org/wiki/Comput..._fluid_dynamics' ">http://en.wikipedia.org/wiki/Comput..._fluid_dynamics. This means the processes run fully in RAM and typically run for hours to days @100% CPU. CFD codes are are heavily parrallelised so each extra core is gold in reducing compute time and is fully utilised.

Basically there is very little IO except for loading the runs and post-processing (viewing results, making videos) which is pretty IO and GFX intensive as results files can run into many gigs etc.

After digging through these forums this is what I'm proposing to get:

MB: TYAN K8WE with SCSI option:
I am rather nervous about all the reported BIOS issues ...

CPUs: 2x Opteron 265's (may go for 270s):
I'm not going to OC much (stability is very NB in CFD) but may try to bump them slightly to, say 2000 MHZ.

RAM: 4x Corsair 1 gb CM72SD1024RLP-3200/S 3-3-3-8
One in each channel per CPU hopefully giving me dual channel speeds + NUMA Somone please confirm this is good to go for dual channel bandwidth + NUMA (OS = Red hat enterprise WS 4)??? Memory bandwidth is the second most NB thing for CFD after raw core speed Do I need to buy RAM in matched pairs to get dual channel running properly??? I will be popping some heat spreaders on these naked sticks.

Chip coolers: 2x Thermalright XP-90Cs + 2x Panasonic Panaflo 90 mm fans

HDD: 2x Seagate Cheetah 15K.4 36GB U320 Non raided using the onboard SCSI controller.
1st HDD: OS + apps, 2nd HDD: swap + working folders. I will also be adding a SATA 2 RAID later (probably using the onboard RAID 5 capability). The SATA 2 RAID will only be accessed very infrequently and basically just be there for data storage. I'm not worried about RAIDing the two SCSI drives as I can re-image them quickly if they **** out ...

GFX: 1x Gigabyte Geforce 7800 GT GV-NX78T256D-ZK
I currently use a geforce 4600(! and even this has more than enough oomph to do all the post-processing which is all OGL based. So a 7800 GT seems more than adaquate. Quadros are a complete waste of $ to me

PS: Silverstone ST65ZF 650 W

Chassis: LIAN LI PC-V2000 with 2x 120 mm stock fans.
Very expensive but tons of room for expansion Depending on how things turn out I may add extra/different fans.

None of this is set in stone so I welcome feedback / sugestions / flames /ph34r.gif' border='0' style='vertical-align:middle' alt='ph34r.gif' />

PS: Thanks to AMD for making the 2xx Opteron series ... pity they are so expensive /wacko.gif' border='0' style='vertical-align:middle' alt='wacko.gif' />
 12/22/2005 09:22 PM
User is offline View Users Profile Print this message

Author Icon
Stickem
Member

Posts: 135
Joined: 12/18/2005

It appears to me that you have done your homework VERY thoroughly!

X-Bit Labs just tested a K8WE workstation today: http://www.xbitlabs.com/' ">http://www.xbitlabs.com/ against a Zeon.
It should make pretty good reading for you.

Anandtech also reviewed one a while back in their "IT" section. (I think?)

As to critiquing your rig, the only thing that "may" be a touch undersized would be the power supply.
Silverstone make serious equipment, but your video card choice is also pretty power-hungry! You may wish to double check the numbers on that. (It should be fine)

There is a school of thought that says that RAM heat spreaders actually trap heat inside.

Everything else is VERY well thought out.

None of your BIOS or NUMA issues will present anything more than trivial problems.

I am depressingly envious.
 12/22/2005 09:42 PM
User is offline View Users Profile Print this message

Author Icon
mckennma
Senior Member

Posts: 2989
Joined: 06/01/2005

QUOTE(CFD2 @ Dec 22 2005, 07:34 PM)Hi all

I'm going to use this rig purely as a compute server for CFD simulations http://en.wikipedia.org/wiki/Comput..._fluid_dynamics' ">http://en.wikipedia.org/wiki/Comput..._fluid_dynamics. This means the processes run fully in RAM and typically run for hours to days @100% CPU. CFD codes are are heavily parrallelised so each extra core is gold in reducing compute time and is fully utilised.

Basically there is very little IO except for loading the runs and post-processing (viewing results, making videos) which is pretty IO and GFX intensive as results files can run into many gigs etc.

After digging through these forums this is what I'm proposing to get:

MB: TYAN K8WE with SCSI option:
I am rather nervous about all the reported BIOS issues ...

CPUs: 2x Opteron 265's (may go for 270s):
I'm not going to OC much (stability is very NB in CFD) but may try to bump them slightly to, say 2000 MHZ.

RAM: 4x Corsair 1 gb CM72SD1024RLP-3200/S 3-3-3-8
One in each channel per CPU hopefully giving me dual channel speeds + NUMA Somone please confirm this is good to go for dual channel bandwidth + NUMA (OS = Red hat enterprise WS 4)??? Memory bandwidth is the second most NB thing for CFD after raw core speed Do I need to buy RAM in matched pairs to get dual channel running properly??? I will be popping some heat spreaders on these naked sticks.

Chip coolers: 2x Thermalright XP-90Cs + 2x Panasonic Panaflo 90 mm fans

HDD: 2x Seagate Cheetah 15K.4 36GB U320 Non raided using the onboard SCSI controller.
1st HDD: OS + apps, 2nd HDD: swap + working folders. I will also be adding a SATA 2 RAID later (probably using the onboard RAID 5 capability). The SATA 2 RAID will only be accessed very infrequently and basically just be there for data storage. I'm not worried about RAIDing the two SCSI drives as I can re-image them quickly if they **** out ...

GFX: 1x Gigabyte Geforce 7800 GT GV-NX78T256D-ZK
I currently use a geforce 4600(! and even this has more than enough oomph to do all the post-processing which is all OGL based. So a 7800 GT seems more than adaquate. Quadros are a complete waste of $ to me

PS: Silverstone ST65ZF 650 W

Chassis: LIAN LI PC-V2000 with 2x 120 mm stock fans.
Very expensive but tons of room for expansion Depending on how things turn out I may add extra/different fans.

None of this is set in stone so I welcome feedback / sugestions / flames  /ph34r.gif' border='0' style='vertical-align:middle' alt='ph34r.gif' />

PS: Thanks to AMD for making the 2xx Opteron series ... pity they are so expensive  /wacko.gif' border='0' style='vertical-align:middle' alt='wacko.gif' />
[right][snapback]572541[/snapback][/right]



My K8W benchmarks' ">http://wowway.com/~mckennma/Benchmark/index.htm

My K8W specs' ">http://wowway.com/~mckennma/Benchmark/workstation.txt

The K8WE is a nice board. Lacks 5.1 or 7.1 on-board sound. I have a PCI-X SCSI RAID 320-2X with 512MB cache on mine. It is worth looking at to feed this board. Check out the Cooler Master Stacker 810 and 830. You will find them to be excellent chassis. Check the Supermicro CSE-M35S SCA SCSI U320 enclosures for your SCSI drives.

-------------------------
Tyan Thunder K8WE
Dual AMD 280 Opterons
8GB NUMA enabled DDR333 2.5-3-3-7 RAM
PCI-X SCSI RAID
http://69.14.190.80
 12/23/2005 01:03 AM
User is offline View Users Profile Print this message

Author Icon
Bitey
Elite

Posts: 1492
Joined: 10/07/2003

Why go with scsi?. You have pointed out your app will almost entirely memory bound, you HDD's could probably sleep most of the time in which case a nice pair of SATA's should do the job or even 4 of them for RAID 5.

Anyways as is it seems like a good setup for what you are doing.

Supermicro makes some nice dual capable boards, with few if any bios issues.
 12/23/2005 08:21 AM
User is offline View Users Profile Print this message

Author Icon
CFD2
Junior Member

Posts: 3
Joined: 12/22/2005

Thanks for all the responses I'm busy reading and researching them now.

Also I should point out that I have taken a fresh look at my budget have realised, like Bitey says, that I am definately overspending on SCSI. Ive decided to drop SCSI entirely in favour of a single 74 GB WD raptor and will go for a less extravagent case (stacker?). The $ saved there will enable me to get 2x 275s with a stock speedup of ~22% over the 265s. Which is pretty much equivalent to running 5 265s instead of 4 /biggrin.gif' border='0' style='vertical-align:middle' alt='biggrin.gif' /> /biggrin.gif' border='0' style='vertical-align:middle' alt='biggrin.gif' /> /biggrin.gif' border='0' style='vertical-align:middle' alt='biggrin.gif' />

SCSI wont make a jot of difference to simulations running for 24 hours but a 22% speed boost sure will /cool.gif' border='0' style='vertical-align:middle' alt='cool.gif' /> i.e. 5.3 hours quicker ... /wub.gif' border='0' style='vertical-align:middle' alt='wub.gif' /> /wub.gif' border='0' style='vertical-align:middle' alt='wub.gif' /> /wub.gif' border='0' style='vertical-align:middle' alt='wub.gif' />

I'll just have settle for slightly slower post processing which isnt a big deal in the greater scheme of things.
 12/23/2005 08:27 AM
User is offline View Users Profile Print this message

Author Icon
CFD2
Junior Member

Posts: 3
Joined: 12/22/2005

QUOTE(Stickem @ Dec 22 2005, 08:22 PM)
As to critiquing your rig, the only thing that "may" be a touch undersized would be the power supply. 
Silverstone make serious equipment, but your video card choice is also pretty power-hungry!  You may wish to double check the numbers on that.  (It should be fine)

This whole power supply business is rather touchy. Some people say they have run a similar rig to this with SLI 7800 GTXs or Quadro's without any issues on a 550W PS. I should be fine on 650 with a single 7800 GT particularly considering I wont have many HDDs initially.

QUOTEThere is a school of thought that says that RAM heat spreaders actually trap heat inside.

Interesting. I should do some flow calcs to investigate this. These heat spreaders should ideally be finned ... low profile of course /ph34r.gif' border='0' style='vertical-align:middle' alt='ph34r.gif' />

 12/23/2005 08:39 AM
User is offline View Users Profile Print this message

Author Icon
mckennma
Senior Member

Posts: 2989
Joined: 06/01/2005

QUOTE(Bitey @ Dec 23 2005, 02:03 AM)Why go with scsi?. You  have pointed out your app will almost entirely memory bound, you HDD's could probably sleep most of the time in which case a nice pair of SATA's should do the job or even 4 of them for RAID 5.

Anyways as is it seems like a good setup for what you are doing.

Supermicro makes some nice dual capable boards, with few if any bios issues.
[right][snapback]572827[/snapback][/right]


SCSI drives are more reliable and are much faster. You also get a 100% duty cycle. The SATA are 33% or 8 hours per day. Since its only two drives, 15K are worth the cost. Just add extra cooling or get a Supermicro CSE-M35S enclosure for them. I would not do RAID 5 unless you have a higher end RAID controller. Write performance can be low. I found SCSI RAID 10 has excellent write perforamnce and decent read performance. You still get the minimal sustained read performance + the boost from the SCSI RAID cache. It is worth looking at a 320-1x or 320-2x from LSI Logic.

SATA is nice. SCSI is best.

-------------------------
Tyan Thunder K8WE
Dual AMD 280 Opterons
8GB NUMA enabled DDR333 2.5-3-3-7 RAM
PCI-X SCSI RAID
http://69.14.190.80
Statistics
112018 users are registered to the AMD Processors forum.
There are currently 0 users logged in.

FuseTalk Hosting Executive Plan v3.2 - © 1999-2014 FuseTalk Inc. All rights reserved.



Contact AMD Terms and Conditions ©2007 Advanced Micro Devices, Inc. Privacy Trademark information