AMD Processors
Decrease font size
Increase font size
Topic Title: more memory!?
Topic Summary:
Created On: 03/10/2005 07:36 AM
Status: Read Only
Linear : Threading : Single : Branch
<< 1 2 Previous Last unread
Search Topic Search Topic
Topic Tools Topic Tools
View similar topics View similar topics
View topic in raw text format. Print this topic.
 04/29/2005 08:43 AM
User is offline View Users Profile Print this message

Author Icon
jes
Senior Member

Posts: 1134
Joined: 10/22/2003

QUOTE (highlandsun)That makes little sense. Just as with RAID, when you use a set of machines instead of a single machine, your MTBF for the total system is inversely proportional to the number of machines. Not to mention the increased administration overhead from managing the larger number of servers, and the painstaking effort to apportion the data to each server and keep the raw data volumes (as well as query volume) balanced as the data grows over time.


MTBF is a number, pure and simple, it's not a guarantee...it's not a precision science. It's affected by many *MANY* different criteria.

I don't see your analogy with RAID, RAID has *many* different types, Raid-0 is a very different animal to Raid-1. They serve very different purposes.

If I buy 4 identical HD's and put 2 into a Raid-0 configuration and 2 into a Raid-1 configuration, have their MTBF's changed? No....what *has* changed is my chances of recovering from a disk going down. Similarly if I run a master LDAP server and two slaves, have their individualMTBF's changed? No....but my time to recover from a "disaster" has drastically changed.

Administration overhead? If you have an LDAP server with a 1TB footprint, you're going to have administration overheads fullstop. I'll take administration
overheads over recovery overheads any day of the week. You can easily hook a number of LDAP slaves up to a master to take over in the event
that the master dies....if you're relying on a single server to perform 1TB of lookups for you then where's your DR plan?

QUOTE (highlandsun)
There will be 10-20 of these servers located at geographically distributed sites. Certainly not just one. The sysadmin overhead will be bad enough at that. If each site is a farm of smaller servers, the sysadmin cost will skyrocket.


So you will have multiple machines then as I mentioned...

Had you said you have an LDAP server with 1Gb of data I'd agree with you....1TB is another league altogether, that's a big server.....that data is either critical or it isn't...if it isn't critical then you can certainly "cut corners" (as it were), if it's critical then they have to pay the money to keep it critical....

Are those servers serving sub-groups? Do they all carry the same data or are they serving topographically different data?

QUOTE (highlandsun)
Excuse me? How does a filesystem buffer cache allow you to have consistent reads when it operates in writethru mode? This is such a basic function of any cache, that's not even a worthwhile question.


It's a completely valid question, you gave no information in your original post about exactly what the "transactional backend" was, whether it could be modified by any other processes, whether any jobs run against that backend, etc etc.

This isn't a personal attack against your system, please don't take it as such. I've just seen many systems where people try to build the most powerful machine they can when two or more less powerful systems would actually be cheaper and more effective.

-------------------------
The opinions expressed above do not represent those of Advanced Micro Devices or any of their affiliates.
http://www.shellprompt.net
Unix & Oracle Web Hosting Provider powered by AMD Opterons
 04/29/2005 12:58 PM
User is offline View Users Profile Print this message

Author Icon
highlandsun
B1FF

Posts: 111
Joined: 03/10/2005

QUOTE Had you said you have an LDAP server with 1Gb of data I'd agree with you....1TB is another league altogether, that's a big server.....that data is either critical or it isn't...if it isn't critical then you can certainly "cut corners" (as it were), if it's critical then they have to pay the money to keep it critical....

Are those servers serving sub-groups? Do they all carry the same data or are they serving topographically different data?

Yes, I completely agree with you here. Ultimately these guys will pay what it costs to keep their uptime intact. But some costs are more justified than others.

Each server will carry a complete copy of the data. In general the queries at each site will only hit a subset of the data, but it is a requirement that any server be capable of answering from any portion of the data at full speed.
QUOTE It's a completely valid question, you gave no information in your original post about exactly what the "transactional backend" was, whether it could be modified by any other processes, whether any jobs run against that backend, etc etc.
Sorry if that response was rude. I assumed that "transactional" was self-evident before, but now I hope I have explained it clearly.

As for whether other processes can be run against it - the answer is yes; the underlying database uses shared memory for its own cache so multiple processes can attach to the database environment and operate on it concurrently. The locking subsystem maintains the data consistency here just as it does in a single-process situation. That's not a real concern here; in general there will only be the one server process active and nothing else accessing the database.
QUOTE
This isn't a personal attack against your system, please don't take it as such. I've just seen many systems where people try to build the most powerful machine they can when two or more less powerful systems would actually be cheaper and more effective.
Yes, I understand that. There are merits to horizontal scaling of course, but it always brings with it a new set of management problems that are very different from what you have to deal with in vertical scaling. Some of those are problems we simply would rather not have to solve.

-------------------------
Asus A8V Deluxe, Opteron 185, 4xSamsung 1GB Reg/ECC DDR400, AC Freezer64 Pro, Antec Sonata case w/TruePower480
 04/30/2005 12:46 AM
User is offline View Users Profile Print this message

Author Icon
functional-pc
Senior Member

Posts: 1558
Joined: 01/10/2005

so...do you have it figured out?

that 'tera-ram' system is about 1.7 million for 1 terabyte of ram...

-------------------------
Latest workstation:
Lian li PC-A05B, KN3-SLi, 4400 Brisbane, Freezer 64 Pro, 2 gigs Super-Talent T800UX2GC4, 36 raptor system drive, Caviar RE 320 storage drive, Hiper 580, Ati 1300 Pro, running Vista Home Premium 64-bit with Symantec 10.2.224
 04/30/2005 01:47 AM
User is offline View Users Profile Print this message

Author Icon
highlandsun
B1FF

Posts: 111
Joined: 03/10/2005

Thanks for pointing out what I already mentioned in the first post on this thread...

-------------------------
Asus A8V Deluxe, Opteron 185, 4xSamsung 1GB Reg/ECC DDR400, AC Freezer64 Pro, Antec Sonata case w/TruePower480
 05/19/2005 01:26 AM
User is offline View Users Profile Print this message

Author Icon
sound_man_dan
Junior Member

Posts: 1
Joined: 05/19/2005

Back to horizontal scaling -- the MySQL Cluster might be worth checking out. I am considering it myself, though not for anything near 1TB RAM.

http://www.mysql.com/products/cluster/faq.html' ">http://www.mysql.com/products/cluster/faq.html

Sub-millisecond response is possible under certain conditions when using SCI sockets... (go to link and scroll down)

http://dev.mysql.com/doc/mysql...rformance-figures.html' ">http://dev.mysql.com/doc/mysql...rformance-figures.html

The system allows for data to be written from memory to disk periodically, and transaction logging could allow for recovery, when needed.

Under normal conditions, doubling the memory would increase the reliability, since data could be replicated in memory, as well as on disk, but I do not know what side-effects there might be when going from 1TB RAM to 2TB RAM.

Another thought -- from the application point of view, given that query response time is such a bottle-neck, if multiple queries could be issued in parallel, effective latency would be decreased.

Good luck! (And I would love to know the solution you go with.)

Daniel Kauffman

-------------------------
--
Daniel Kauffman
 05/20/2005 05:29 AM
User is offline View Users Profile Print this message

Author Icon
highlandsun
B1FF

Posts: 111
Joined: 03/10/2005

QUOTE(sound_man_dan @ May 18 2005, 09:26 PM)Back to horizontal scaling -- the MySQL Cluster might be worth checking out. I am considering it myself, though not for anything near 1TB RAM.


We already use the same underlying technology that MySQL uses.

Yes, it looks like we may have to abandon the vertical approach. Parallelizing across a farm of smaller machines seems to get us faster results. The big problem here is segmenting the data across all of the machines and keeping their data sizes relatively balanced, as well as balancing the query traffic.


-------------------------
Asus A8V Deluxe, Opteron 185, 4xSamsung 1GB Reg/ECC DDR400, AC Freezer64 Pro, Antec Sonata case w/TruePower480
 08/03/2005 07:12 PM
User is offline View Users Profile Print this message

Author Icon
cessna172k
Junior Member

Posts: 1
Joined: 05/18/2004

QUOTE(highlandsun @ May 20 2005, 01:29 AM)We already use the same underlying technology that MySQL uses.

Yes, it looks like we may have to abandon the vertical approach. Parallelizing across a farm of smaller machines seems to get us faster results. The big problem here is segmenting the data across all of the machines and keeping their data sizes relatively balanced, as well as balancing the query traffic.
[right][snapback]421090[/snapback][/right]



Heres just a thought. What about a medium sized server with a Raid drive tray or several. This allows you to have a single db spread across multiple files and multiple spidles which yeilds some very good performace. More spidles for a given rpm is usually better. For example we have a set of USP in MSSQL that used to run for 4 days and now runs in 6-8 hours when we switched. Our process is like yours in that the processors are not the bottle neck <30%. As was a querys go I can't say each query can vary. We Were initally going to go with a DB completely in RAM also but due to the small incremental increase in speed and the fact that we too were pushing 1TB the hard disks were a no brainer.

You stated that it needed to be off the shelf.
We used a HP585 for the main server and then 2 of the MSA30 storage rack. This totals 4111.4GB ( 28 146.8GB per drive) could be 8400GB with 28 300GB. Slower than all ram however its scaleable to fit your needs. Need more room? buy more trays and Controllers. Up to a point of course. There are only so many controllers that will fit in one machine. LOL /biggrin.gif' border='0' style='vertical-align:middle' alt='biggrin.gif' />

Maybe this applies to you maybe not. Just an option but it does deliver SCSI speed.

IBACFII
 08/09/2005 06:27 AM
User is offline View Users Profile Print this message

Author Icon
GNUS inc
Member

Posts: 136
Joined: 10/07/2003

QUOTE(highlandsun @ May 20 2005, 01:29 PM)We already use the same underlying technology that MySQL uses.

Yes, it looks like we may have to abandon the vertical approach. Parallelizing across a farm of smaller machines seems to get us faster results. The big problem here is segmenting the data across all of the machines and keeping their data sizes relatively balanced, as well as balancing the query traffic.
[right][snapback]421090[/snapback][/right]

http://www.iwill.net/product_2.asp?p_id=102&sp=Y3' ">http://www.iwill.net/product_2.asp?p_id=102&sp=Y3 and InfiniBand could help you /smile.gif' border='0' style='vertical-align:middle' alt='smile.gif' />
A lot of onboard memory + low latency 10 Gbits interconnect /cool.gif' border='0' style='vertical-align:middle' alt='cool.gif' />

-------------------------
AMD rulezzzzz!
www.etegro.com
 08/16/2005 04:49 AM
User is offline View Users Profile Print this message

Author Icon
highlandsun
B1FF

Posts: 111
Joined: 03/10/2005

I've pretty much decided to go with a farm of smaller servers accessing a single shared DB file. The DB design will allocate records on filesystem page boundaries and each node in the server farm will be responsible for a non-overlapping subset of the the records in the DB. By guaranteeing that each node accesses non-overlapping sets of records, all of the nodes can hit the shared DB without requiring the use of remote locking protocols. Since they will be doing page-sized I/Os, the filesystem caches can just move whole pages in and out without any coherency issues.

There will still need to be a locking mechanism in place for Add operations that require overflow pages (when a record doesn't fit within the first page), but we'll choose a large enough page size to make this a rare occurrence. It will waste a fair amount of space on disk if all of the records are much smaller than the FS page size, but in this case the performance gain is worth it.

That solves one of our big problems, and this approach will scale to as many nodes as we can attach to the shared filesystem, and will easily accomodate adding and removing nodes from the server farm. The remaining problem is our auxiliary indexing DBs, which map from field values to primary DB keys. These are also monolithic DBs, and I still haven't figured out a good way to break these up and distribute them in a scalable fashion. Typically they'll sum up to only 10-15% of the volume of the primary DB, but at 1TB for the primary DB that's still a lot of data for a single node to manage.

By the way, thanks GNUS Inc for the pointer, that looks like a very capable board.
I'm looking into ways to keep our index working set below 32GB, in which case systems like this would give us a lot of breathing room.

-------------------------
Asus A8V Deluxe, Opteron 185, 4xSamsung 1GB Reg/ECC DDR400, AC Freezer64 Pro, Antec Sonata case w/TruePower480
 08/23/2007 04:41 PM
User is offline View Users Profile Print this message

Author Icon
highlandsun
B1FF

Posts: 111
Joined: 03/10/2005

Look at that, 2 and a half years later, AMD decides maybe a hypertransport-connected memory fanout device isn't such a bad idea after all. G3MX couldn't get here soon enough, as far as I'm concerned...

-------------------------
Asus A8V Deluxe, Opteron 185, 4xSamsung 1GB Reg/ECC DDR400, AC Freezer64 Pro, Antec Sonata case w/TruePower480
Statistics
112018 users are registered to the AMD Processors forum.
There are currently 0 users logged in.

FuseTalk Hosting Executive Plan v3.2 - © 1999-2014 FuseTalk Inc. All rights reserved.



Contact AMD Terms and Conditions ©2007 Advanced Micro Devices, Inc. Privacy Trademark information