Re: understanding postgres issues/bottlenecks

Ok, here some information:

OS: Centos 5.x (Linux 2.6.18-53.1.21.el5 #1 SMP Tue May 20 09:34:18 EDT
2008 i686 i686 i386 GNU/Linux)
RAID: it's a hardware RAID controller
The disks are 9600rpm SATA drives

(6 disk 1+0 RAID array and 2 separate disks for the OS).


About iostat (on sdb I have pg_xlog, on sdc I have data)

 > iostat -k
Linux 2.6.18-53.1.21.el5 (*******)       01/07/2009

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          17.27    0.00    5.13   45.08    0.00   32.52

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sda              30.42        38.50       170.48  182600516  808546589
sdb              46.16         0.23        52.10    1096693  247075617
sdc             269.26       351.51       451.00 1667112043 2138954833



 > iostat -x -k -d 2 5
Linux 2.6.18-53.1.21.el5 (*******)       01/07/2009

Device:         rrqm/s   wrqm/s   r/s   w/s    rkB/s    wkB/s avgrq-sz
avgqu-sz   await  svctm  %util
sda               0.17    12.68  0.47 29.95    38.51   170.51
13.74     0.03    0.86   0.19   0.57
sdb               0.01    80.11  0.05 46.11     0.23    52.01
2.26     0.01    0.22   0.22   1.01
sdc               7.50    64.57 222.55 46.69   350.91   450.98
5.96     0.57    2.05   3.13  84.41

Device:         rrqm/s   wrqm/s   r/s   w/s    rkB/s    wkB/s avgrq-sz
avgqu-sz   await  svctm  %util
sda               0.00     0.00  0.00  0.00     0.00     0.00
0.00     0.00    0.00   0.00   0.00
sdb               0.00   196.00  1.00 117.00     4.00  1252.00
21.29     0.02    0.19   0.19   2.30
sdc               1.50    66.00 277.00 66.50  3100.00   832.00
22.89    50.84  242.30   2.91 100.10

Device:         rrqm/s   wrqm/s   r/s   w/s    rkB/s    wkB/s avgrq-sz
avgqu-sz   await  svctm  %util
sda               0.00     0.00  0.00  0.00     0.00     0.00
0.00     0.00    0.00   0.00   0.00
sdb               0.00   264.50  0.00 176.50     0.00  1764.00
19.99     0.04    0.21   0.21   3.70
sdc               3.50   108.50 291.50 76.00  3228.00   752.00
21.66    89.42  239.39   2.72 100.05

Device:         rrqm/s   wrqm/s   r/s   w/s    rkB/s    wkB/s avgrq-sz
avgqu-sz   await  svctm  %util
sda               0.00     4.98  0.00  1.00     0.00    23.88
48.00     0.00    0.00   0.00   0.00
sdb               0.00    23.88  0.00  9.45     0.00   133.33
28.21     0.00    0.21   0.21   0.20
sdc               1.00   105.97 274.13 53.73  3297.51   612.94
23.85    67.99  184.58   3.04  99.55

Device:         rrqm/s   wrqm/s   r/s   w/s    rkB/s    wkB/s avgrq-sz
avgqu-sz   await  svctm  %util
sda               0.00     0.00  0.00  0.00     0.00     0.00
0.00     0.00    0.00   0.00   0.00
sdb               0.00    79.00  0.00 46.00     0.00   500.00
21.74     0.01    0.25   0.25   1.15
sdc               2.50   141.00 294.00 43.50  3482.00   528.00
23.76    51.33  170.46   2.96 100.05


vmstat in the same time:
 > vmstat 2
procs -----------memory---------- ---swap-- -----io---- --system--
-----cpu------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy
id wa st
 0 27     80 126380  27304 3253016    0    0    98    55    0    1 17  5
33 45  0
 0 26     80 124516  27300 3255456    0    0  3438  1724 2745 4011 11
2  8 78  0
 1 25     80 124148  27276 3252548    0    0  3262  2806 3572 7007 33
11  3 53  0
 1 28     80 128272  27244 3248516    0    0  2816  1006 2926 5624 12  3
12 73  0


I will run pgbench in the next days.


>
> Aside from all the advice here about system tuning, as a system admin I'd also
> ask is the box doing the job you need? And are you looking at the Postgres
> log (with logging of slow queries) to see that queries perform in a sensible
> time? I'd assume with the current performance figure there is an issue
> somewhere, but I've been to places where it was as simple as adding one
> index, or even modifying an index so it does what the application developer
> intended instead of what they ask for ;)
>


I already checked postgres log and resolved index/slow queries issues.
Actually I have queries that sometime are really fast, and sometime go
in timeout.
But all the required indexes are there. For sure, there are space to
improve performances also in that way, but I would like also to
investigate issue from other point of views (in order to understand also
how to monitor the server).


Cheers and thanks a lot.
ste