Re: Opteron vs Xeon (Was: What to do with 6 disks?)

Good question. If the SCSI system was moving the head from track 1 to 10, and a request then came in for track 5, could the system make the head stop at track 5 on its way to track 10? That is something that only the controller could do. However, I have no idea if SCSI does that.

|| SCSI, AFAIK, does NOT do this. What SCSI can do is allow "next" request insertion into head
of request queue (queue-jumping), and/or defer request ordering to done by drive per se (queue
re-ordering). I have looked, in vain, for evidence that SCSI somehow magically "stops in the
middle of request to pick up data" (my words, not yours)

The only part I am pretty sure about is that real-world experience shows SCSI is better for a mixed I/O environment. Not sure why, exactly, but the command queueing obviously helps, and I am not sure what else does.

|| TCQ is the secret sauce, no doubt. I think NCQ (the SATA version of per se drive request reordering)
should go a looong way (but not all the way) toward making SATA 'enterprise acceptable'. Multiple
initiators (e.g. more than one host being able to talk to a drive) is a biggie, too. AFAIK only SCSI
drives/controllers do that for now.

--
Bruce Momjian | http://candle.pha.pa.us
pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 359-1001
+ If your life is a hard drive, | 13 Roberts Road
+ Christ can be your backup. | Newtown Square, Pennsylvania 19073

---------------------------(end of broadcast)---------------------------
TIP 9: the planner will ignore your desire to choose an index scan if your
joining column's datatypes do not match

Mohan, Ross wrote:
> The only part I am pretty sure about is that real-world experience shows SCSI is better for a mixed I/O environment. Not sure why, exactly, but the command queueing obviously helps, and I am not sure what else does.
>
> || TCQ is the secret sauce, no doubt. I think NCQ (the SATA version of per se drive request reordering)
> should go a looong way (but not all the way) toward making SATA 'enterprise acceptable'. Multiple
> initiators (e.g. more than one host being able to talk to a drive) is a biggie, too. AFAIK only SCSI
> drives/controllers do that for now.

What is 'multiple initiators' used for in the real world?

--
Bruce Momjian | http://candle.pha.pa.us
pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 359-1001
+ If your life is a hard drive, | 13 Roberts Road
+ Christ can be your backup. | Newtown Square, Pennsylvania 19073

pgsql-performance-owner(at)postgresql(dot)org wrote on 04/19/2005 11:10:22 AM:
>
> What is 'multiple initiators' used for in the real world?

I asked this same question and got an answer off list: Somebody said their
SAN hardware used multiple initiators. I would try to check the archives
for you, but this thread is becoming more of a rope.

Multiple initiators means multiple sources on the bus issuing I/O
instructions to the drives. In theory you can have two computers on the
same SCSI bus issuing I/O requests to the same drive, or to anything else
on the bus, but I've never seen this implemented. Others have noted this
feature as being a big deal, so somebody is benefiting from it.

Rick
>
> --
> Bruce Momjian | http://candle.pha.pa.us
> pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 359-1001
> + If your life is a hard drive, | 13 Roberts Road
> + Christ can be your backup. | Newtown Square, Pennsylvania
19073
>
> ---------------------------(end of broadcast)---------------------------
> TIP 7: don't forget to increase your free space map settings

On Tue, Apr 19, 2005 at 11:22:17AM -0500, Richard_D_Levine(at)raytheon(dot)com wrote:
>
>
> pgsql-performance-owner(at)postgresql(dot)org wrote on 04/19/2005 11:10:22 AM:
> >
> > What is 'multiple initiators' used for in the real world?
>
> I asked this same question and got an answer off list: Somebody said their
> SAN hardware used multiple initiators. I would try to check the archives
> for you, but this thread is becoming more of a rope.
>
> Multiple initiators means multiple sources on the bus issuing I/O
> instructions to the drives. In theory you can have two computers on the
> same SCSI bus issuing I/O requests to the same drive, or to anything else
> on the bus, but I've never seen this implemented. Others have noted this
> feature as being a big deal, so somebody is benefiting from it.

It's a big deal for Oracle clustering, which relies on shared drives. Of
course most people doing Oracle clustering are probably using a SAN and
not raw SCSI...
--
Jim C. Nasby, Database Consultant decibel(at)decibel(dot)org
Give your computer some brain candy! www.distributed.net Team #1828

Windows: "Where do you want to go today?"
Linux: "Where do you want to go tomorrow?"
FreeBSD: "Are you guys coming, or what?"

Now that we've hashed out which drives are quicker and more money equals
faster...

Let's say you had a server with 6 separate 15k RPM SCSI disks, what raid
option would you use for a standalone postgres server?

a) 3xRAID1 - 1 for data, 1 for xlog, 1 for os?
b) 1xRAID1 for OS/xlog, 1xRAID5 for data
c) 1xRAID10 for OS/xlong/data
d) 1xRAID1 for OS, 1xRAID10 for data
e) .....

I was initially leaning towards b, but after talking to Josh a bit, I suspect
that with only 4 disks the raid5 might be a performance detriment vs 3 raid 1s
or some sort of split raid10 setup.

--
Jeff Frost, Owner <jeff(at)frostconsultingllc(dot)com>
Frost Consulting, LLC http://www.frostconsultingllc.com/
Phone: 650-780-7908 FAX: 650-649-1954

http://stats.distributed.net is setup with the OS, WAL, and temp on a
RAID1 and the database on a RAID10. The drives are 200G SATA with a
3ware raid card. I don't think the controller has battery-backed cache,
but I'm not sure. In any case, it's almost never disk-bound on the
mirror; when it's disk-bound it's usually the RAID10. But this is a
read-mostly database. If it was write-heavy, that might not be the case.

Also, in general, I see very little disk activity from the OS itself, so
I don't think there's a large disadvantage to having it on the same
drives as part of your database. I would recommend different filesystems
for each, though. (ie: not one giant / partition)

On Tue, Apr 19, 2005 at 06:00:42PM -0700, Jeff Frost wrote:
> Now that we've hashed out which drives are quicker and more money equals
> faster...
>
> Let's say you had a server with 6 separate 15k RPM SCSI disks, what raid
> option would you use for a standalone postgres server?
>
> a) 3xRAID1 - 1 for data, 1 for xlog, 1 for os?
> b) 1xRAID1 for OS/xlog, 1xRAID5 for data
> c) 1xRAID10 for OS/xlong/data
> d) 1xRAID1 for OS, 1xRAID10 for data
> e) .....
>
> I was initially leaning towards b, but after talking to Josh a bit, I
> suspect that with only 4 disks the raid5 might be a performance detriment
> vs 3 raid 1s or some sort of split raid10 setup.
>
> --
> Jeff Frost, Owner <jeff(at)frostconsultingllc(dot)com>
> Frost Consulting, LLC http://www.frostconsultingllc.com/
> Phone: 650-780-7908 FAX: 650-649-1954
>
> ---------------------------(end of broadcast)---------------------------
> TIP 5: Have you checked our extensive FAQ?
>
> http://www.postgresql.org/docs/faq
>

--
Jim C. Nasby, Database Consultant decibel(at)decibel(dot)org
Give your computer some brain candy! www.distributed.net Team #1828

Windows: "Where do you want to go today?"
Linux: "Where do you want to go tomorrow?"
FreeBSD: "Are you guys coming, or what?"

My experience:

1xRAID10 for postgres
1xRAID1 for OS + WAL

Jeff Frost wrote:
> Now that we've hashed out which drives are quicker and more money equals
> faster...
>
> Let's say you had a server with 6 separate 15k RPM SCSI disks, what raid
> option would you use for a standalone postgres server?
>
> a) 3xRAID1 - 1 for data, 1 for xlog, 1 for os?
> b) 1xRAID1 for OS/xlog, 1xRAID5 for data
> c) 1xRAID10 for OS/xlong/data
> d) 1xRAID1 for OS, 1xRAID10 for data
> e) .....
>
> I was initially leaning towards b, but after talking to Josh a bit, I
> suspect that with only 4 disks the raid5 might be a performance
> detriment vs 3 raid 1s or some sort of split raid10 setup.
>

Jeff,

> Let's say you had a server with 6 separate 15k RPM SCSI disks, what raid
> option would you use for a standalone postgres server?
>
> a) 3xRAID1 - 1 for data, 1 for xlog, 1 for os?
> b) 1xRAID1 for OS/xlog, 1xRAID5 for data
> c) 1xRAID10 for OS/xlong/data
> d) 1xRAID1 for OS, 1xRAID10 for data
> e) .....
>
> I was initially leaning towards b, but after talking to Josh a bit, I
> suspect that with only 4 disks the raid5 might be a performance detriment
> vs 3 raid 1s or some sort of split raid10 setup.

Knowing that your installation is read-heavy, I'd recommend (d), with the WAL
on the same disk as the OS, i.e.

RAID1 2 disks OS, pg_xlog
RAID 1+0 4 disks pgdata

--
--Josh

Josh Berkus
Aglio Database Solutions
San Francisco

> RAID1 2 disks OS, pg_xlog
> RAID 1+0 4 disks pgdata

Looks like the consensus is RAID 1 for OS, pg_xlog and RAID10 for pgdata. Now
here's another performance related question:

I've seen quite a few folks touting the Opteron as 2.5x faster with postgres
than a Xeon box. What makes the Opteron so quick? Is it that Postgres
really prefers to run in 64-bit mode?

When I look at AMD's TPC-C scores where they are showing off the Opteron
http://www.amd.com/us-en/Processors/ProductInformation/0,,30_118_8796_8800~96125,00.html
It doesn't appear 2.5x as fast as the Xeon systems, though I have heard from a
few Postgres folks that a dual Opteron is 2.5x as fast as a dual Xeon. I
would think that AMD would be all over that press if they could show it, so
what am I missing? Is it a bus speed thing? Better south bridge on the
boards?

--
Jeff Frost, Owner <jeff(at)frostconsultingllc(dot)com>
Frost Consulting, LLC http://www.frostconsultingllc.com/
Phone: 650-780-7908 FAX: 650-649-1954

>I've seen quite a few folks touting the Opteron as 2.5x
>faster with postgres than a Xeon box. What makes the
>Opteron so quick? Is it that Postgres really prefers to
>run in 64-bit mode?

I don't know about 2.5x faster (perhaps on specific types
of loads), but the reason Opterons rock for database
applications is their insanely good memory bandwidth and
latency that scales much better than the Xeon. Opterons
also have a ccNUMA-esque I/O fabric and two dedicated
on-die memory channels *per processor* -- no shared bus
there, closer to real UNIX server iron than a glorified
PC.

We run a large Postgres database on a dual Opteron in
32-bit mode that crushes Xeons running at higher clock
speeds. It has little to do with bitness or theoretical
instruction dispatch, and everything to do with the
superior memory controller and I/O fabric. Databases are
all about moving chunks of data around and the Opteron
systems were engineered to do this very well and in a very
scalable fashion. For the money, it is hard to argue with
the price/performance of Opteron based servers. We
started with one dual Opteron postgres server just over a
year ago (with an equivalent uptime) and have considered
nothing but Opterons for database servers since. Opterons
really are clearly superior to Xeons for this application.
I don't work for AMD, just a satisfied customer. :-)

re: 6 disks. Unless you are tight on disk space, a hot
spare might be nice as well depending on your needs.

Cheers,

J. Andrew Rogers

On Tue, 19 Apr 2005, J. Andrew Rogers wrote:

> I don't know about 2.5x faster (perhaps on specific types of loads), but the
> reason Opterons rock for database applications is their insanely good memory
> bandwidth and latency that scales much better than the Xeon. Opterons also
> have a ccNUMA-esque I/O fabric and two dedicated on-die memory channels *per
> processor* -- no shared bus there, closer to real UNIX server iron than a
> glorified PC.

Thanks J! That's exactly what I was suspecting it might be. Actually, I
found an anandtech benchmark that shows the Opteron coming in at close to 2.0x
performance:

http://www.anandtech.com/linux/showdoc.aspx?i=2163&p=2

It's an Opteron 150 (2.4ghz) vs. Xeon 3.6ghz from August. I wonder if the
differences are more pronounced with the newer Opterons.

-Jeff

I posted this link a few months ago and there was some surprise over the
difference in postgresql compared to other DBs. (Not much surprise in
Opteron stomping on Xeon in pgsql as most people here have had that
experience -- the surprise was in how much smaller the difference was in
other DBs.) If it was across the board +100% in MS-SQL, MySQL, etc --
you can chalk in up to overall better CPU architecture. Most of the time
though, the numbers I've seen show +0-30% for [insert DB here] and a
huge whopping +++++ for pgsql. Why the pronounced preference for
postgresql, I'm not sure if it was explained fully.

BTW, the Anandtech test compares single CPU systems w/ 1GB of RAM. Go to
dual/quad and SMP Xeon will suffer even more since it has to share a
fixed amount of FSB/memory bandwidth amongst all CPUs. Xeons also seem
to suffer more from context-switch storms. Go > 4GB of RAM and the Xeon
suffers another hit due to the lack of a 64-bit IOMMU. Devices cannot
map to addresses > 4GB which means the OS has to do extra work in
copying data from/to > 4GB anytime you have IO. (Although this penalty
might exist all the time in 64-bit mode for Xeon if Linux/Windows took
the expedient and less-buggy route of using a single method versus
checking whether target addresses are > or < 4GB.)

Jeff Frost wrote:
> On Tue, 19 Apr 2005, J. Andrew Rogers wrote:
>
>> I don't know about 2.5x faster (perhaps on specific types of loads),
>> but the reason Opterons rock for database applications is their
>> insanely good memory bandwidth and latency that scales much better
>> than the Xeon. Opterons also have a ccNUMA-esque I/O fabric and two
>> dedicated on-die memory channels *per processor* -- no shared bus
>> there, closer to real UNIX server iron than a glorified PC.
>
>
> Thanks J! That's exactly what I was suspecting it might be. Actually,
> I found an anandtech benchmark that shows the Opteron coming in at close
> to 2.0x performance:
>
> http://www.anandtech.com/linux/showdoc.aspx?i=2163&p=2
>
> It's an Opteron 150 (2.4ghz) vs. Xeon 3.6ghz from August. I wonder if
> the differences are more pronounced with the newer Opterons.
>
> -Jeff
>
> ---------------------------(end of broadcast)---------------------------
> TIP 9: the planner will ignore your desire to choose an index scan if your
> joining column's datatypes do not match
>

On Apr 19, 2005, at 11:07 PM, Josh Berkus wrote:

> RAID1 2 disks OS, pg_xlog
> RAID 1+0 4 disks pgdata
>

This is my preferred setup, but I do it with 6 disks on RAID10 for
data, and since I have craploads of disk space I set checkpoint
segments to 256 (and checkpoint timeout to 5 minutes)

Vivek Khera, Ph.D.
+1-301-869-4449 x806

On Apr 20, 2005, at 12:40 AM, Jeff Frost wrote:

> I've seen quite a few folks touting the Opteron as 2.5x faster with
> postgres than a Xeon box. What makes the Opteron so quick? Is it
> that Postgres really prefers to run in 64-bit mode?
>

The I/O path on the opterons seems to be much faster, and having 64-bit
all the way to the disk controller helps... just be sure to run a
64-bit version of your OS.

Vivek Khera, Ph.D.
+1-301-869-4449 x806