pgperffarm.git
6 years agoadd results filed in TestDataSetDetailSerializer
Hongyuan Ma [Thu, 21 Jun 2018 10:14:47 +0000 (18:14 +0800)]
add results filed in TestDataSetDetailSerializer

6 years agoadapt get_dataset_info in TestRecordDetailSerializer
Hongyuan Ma [Thu, 21 Jun 2018 04:54:51 +0000 (12:54 +0800)]
adapt get_dataset_info in TestRecordDetailSerializer

6 years agoadd uuid field
Hongyuan Ma [Wed, 20 Jun 2018 13:25:11 +0000 (21:25 +0800)]
add uuid field

6 years agoadd hash field form TestRecord model;add TestRecordDetailViewSet;
Hongyuan Ma [Wed, 20 Jun 2018 12:32:13 +0000 (20:32 +0800)]
add hash field form TestRecord model;add TestRecordDetailViewSet;

6 years agoupdate basic-table
Hongyuan Ma [Mon, 18 Jun 2018 03:39:42 +0000 (11:39 +0800)]
update basic-table

6 years agoupdate TestRecordListSerializer;add trend field;
Hongyuan Ma [Mon, 18 Jun 2018 02:54:57 +0000 (10:54 +0800)]
update TestRecordListSerializer;add trend field;

6 years agoupdate the receiver
Hongyuan Ma [Sun, 17 Jun 2018 11:30:54 +0000 (19:30 +0800)]
update the receiver

6 years agoadd calc_status receiver
Hongyuan Ma [Sun, 17 Jun 2018 10:56:33 +0000 (18:56 +0800)]
add calc_status receiver

6 years agoadd TestDataUploadError
Hongyuan Ma [Sat, 16 Jun 2018 06:36:25 +0000 (14:36 +0800)]
add TestDataUploadError

6 years agoadd result parse in upload
Hongyuan Ma [Fri, 15 Jun 2018 15:36:30 +0000 (23:36 +0800)]
add result parse in upload

6 years agoadd branch_data;add cate_sn field;
Hongyuan Ma [Fri, 15 Jun 2018 15:23:01 +0000 (23:23 +0800)]
add branch_data;add cate_sn field;

6 years agoAdd temporary data import
Hongyuan Ma [Fri, 15 Jun 2018 12:38:48 +0000 (20:38 +0800)]
Add temporary data import

6 years agoadd alias dbtool
Hongyuan Ma [Fri, 15 Jun 2018 11:56:18 +0000 (19:56 +0800)]
add alias dbtool

6 years agoadd CreateTestDateSetSerializer
Hongyuan Ma [Thu, 14 Jun 2018 15:00:52 +0000 (23:00 +0800)]
add CreateTestDateSetSerializer

6 years agoadd CreateTestRecordSerializer
Hongyuan Ma [Thu, 14 Jun 2018 14:28:13 +0000 (22:28 +0800)]
add CreateTestRecordSerializer

6 years agoadd TestDataSet model class
Hongyuan Ma [Wed, 13 Jun 2018 12:18:47 +0000 (20:18 +0800)]
add TestDataSet model class

6 years agoOverwrite the table component
Hongyuan Ma [Sat, 9 Jun 2018 05:18:14 +0000 (13:18 +0800)]
Overwrite the table component

6 years agoadd machine serialler;render test list data;
Hongyuan Ma [Fri, 8 Jun 2018 18:14:43 +0000 (02:14 +0800)]
add machine serialler;render test list data;

6 years agofix CORS problem;add detailInfo function
Hongyuan Ma [Thu, 7 Jun 2018 14:17:34 +0000 (22:17 +0800)]
fix CORS problem;add detailInfo function

6 years agoadd data upload interface
Hongyuan Ma [Wed, 6 Jun 2018 14:28:37 +0000 (22:28 +0800)]
add data upload interface

6 years agoadd testDetail viewSet
Hongyuan Ma [Mon, 4 Jun 2018 11:45:57 +0000 (19:45 +0800)]
add testDetail viewSet

6 years agofix url problem
Hongyuan Ma [Sun, 3 Jun 2018 12:49:50 +0000 (20:49 +0800)]
fix url problem

6 years agoadd token auth
MaleicAcid [Sun, 3 Jun 2018 03:57:00 +0000 (11:57 +0800)]
add token auth

6 years agoadjust detail page layout
MaleicAcid [Sat, 2 Jun 2018 15:42:29 +0000 (23:42 +0800)]
adjust detail page layout

6 years agoadd related serializer
MaleicAcid [Sat, 2 Jun 2018 09:44:15 +0000 (17:44 +0800)]
add related serializer

6 years agoadd TestRecord serializer
MaleicAcid [Sat, 2 Jun 2018 09:29:23 +0000 (17:29 +0800)]
add TestRecord serializer

6 years agoadjust model class
MaleicAcid [Sat, 2 Jun 2018 08:34:59 +0000 (16:34 +0800)]
adjust model class

6 years agoadd result card component
MaleicAcid [Fri, 1 Jun 2018 15:43:41 +0000 (23:43 +0800)]
add result card component

6 years agoadd result.json sample
Hongyuan Ma [Fri, 1 Jun 2018 03:58:57 +0000 (11:58 +0800)]
add result.json sample

6 years agoadd detailInfo page
Hongyuan Ma [Fri, 1 Jun 2018 03:57:42 +0000 (11:57 +0800)]
add detailInfo page

6 years agobeautify table-list
Hongyuan Ma [Thu, 31 May 2018 15:46:30 +0000 (23:46 +0800)]
beautify table-list

7 years agoupdate .gitignore
Hongyuan Ma [Mon, 28 May 2018 14:10:08 +0000 (22:10 +0800)]
update .gitignore

7 years agoadd case sensitive in .gitignore
Hongyuan Ma [Mon, 28 May 2018 14:08:19 +0000 (22:08 +0800)]
add case sensitive in .gitignore

7 years agoadjust rate-bar color
Hongyuan Ma [Mon, 28 May 2018 02:14:34 +0000 (10:14 +0800)]
adjust rate-bar color

7 years agoadd status-service
Hongyuan Ma [Mon, 28 May 2018 01:38:01 +0000 (09:38 +0800)]
add status-service

7 years agoadd wating state for applyBtn
Hongyuan Ma [Mon, 28 May 2018 01:24:42 +0000 (09:24 +0800)]
add wating state for applyBtn

7 years agoaccomplist table-list
Hongyuan Ma [Sun, 27 May 2018 12:41:14 +0000 (20:41 +0800)]
accomplist table-list

7 years agoaccompilsh selectItem function
Hongyuan Ma [Sat, 26 May 2018 04:19:57 +0000 (12:19 +0800)]
accompilsh selectItem function

7 years agoupdate rate-bar
Hongyuan Ma [Fri, 25 May 2018 16:26:36 +0000 (00:26 +0800)]
update rate-bar

7 years agoadd RateBat component
Hongyuan Ma [Thu, 24 May 2018 15:52:52 +0000 (23:52 +0800)]
add RateBat component

7 years agoadd pg font icon
Hongyuan Ma [Thu, 24 May 2018 12:33:04 +0000 (20:33 +0800)]
add pg font icon

7 years agoupdate clearBtn function
Hongyuan Ma [Wed, 23 May 2018 14:12:46 +0000 (22:12 +0800)]
update clearBtn function

7 years agoadd table util
Hongyuan Ma [Wed, 23 May 2018 13:54:10 +0000 (21:54 +0800)]
add table util

7 years agoadd ppolicy page
Hongyuan Ma [Wed, 23 May 2018 13:40:03 +0000 (21:40 +0800)]
add ppolicy page

7 years agoadd deleteItem func;
Hongyuan Ma [Tue, 22 May 2018 13:40:56 +0000 (21:40 +0800)]
add deleteItem func;

7 years agoadd selected items css
Hongyuan Ma [Tue, 22 May 2018 13:03:14 +0000 (21:03 +0800)]
add selected items css

7 years agoadd selectItemClick function;
Hongyuan Ma [Tue, 22 May 2018 12:46:40 +0000 (20:46 +0800)]
add selectItemClick function;

7 years agoadd general pagination
Hongyuan Ma [Mon, 21 May 2018 08:33:02 +0000 (16:33 +0800)]
add general pagination

7 years agofix map issue
Hongyuan Ma [Sun, 20 May 2018 14:52:09 +0000 (22:52 +0800)]
fix map issue

7 years agoSeparate filter conditions
Hongyuan Ma [Sun, 20 May 2018 14:27:58 +0000 (22:27 +0800)]
Separate filter conditions

7 years agoadd hover effect
Hongyuan Ma [Fri, 18 May 2018 15:06:24 +0000 (23:06 +0800)]
add hover effect

7 years agoadd loggedin dom
Hongyuan Ma [Fri, 18 May 2018 14:47:07 +0000 (22:47 +0800)]
add loggedin dom

7 years agoadd util.js
Hongyuan Ma [Thu, 17 May 2018 14:30:23 +0000 (22:30 +0800)]
add util.js

7 years agoadd login page
Hongyuan Ma [Thu, 17 May 2018 14:14:12 +0000 (22:14 +0800)]
add login page

7 years agoupdate readme.rst
Hongyuan Ma [Thu, 17 May 2018 13:21:28 +0000 (21:21 +0800)]
update readme.rst

7 years agoFix reference issues
Hongyuan Ma [Thu, 17 May 2018 13:15:58 +0000 (21:15 +0800)]
Fix reference issues

7 years agomodify filter component
Hongyuan Ma [Wed, 16 May 2018 10:53:56 +0000 (18:53 +0800)]
modify filter component

7 years agomodify settings.py
Hongyuan Ma [Tue, 15 May 2018 15:13:27 +0000 (23:13 +0800)]
modify settings.py

7 years agoadd apps,db_tools,extra_apps folders
Hongyuan Ma [Tue, 15 May 2018 14:49:46 +0000 (22:49 +0800)]
add apps,db_tools,extra_apps folders

7 years agoonline adapter
Hongyuan Ma [Sun, 13 May 2018 14:24:04 +0000 (22:24 +0800)]
online adapter

7 years agoonline adapter
Hongyuan Ma [Sun, 13 May 2018 14:21:19 +0000 (22:21 +0800)]
online adapter

7 years agoonline adapter
Hongyuan Ma [Sun, 13 May 2018 14:17:14 +0000 (22:17 +0800)]
online adapter

7 years agoadd front-end code
Hongyuan Ma [Sun, 13 May 2018 12:28:27 +0000 (20:28 +0800)]
add front-end code

7 years agoRefactor pgbench results output
Mark Wong [Sat, 10 Feb 2018 01:59:44 +0000 (17:59 -0800)]
Refactor pgbench results output

The idea is that one results document represents a test of one
particular commit.  Then it contains the results of all the tests run
against that commit.  I think it makes more sense to restructure the
pgbench results to be like:

{
    "pgbench": {
        "ro": {
            "results at a scale factor": {
                "number of clients": {
                    "std": 4389.727756305762,
                    "metric": 41390.590287,
                    "median": 43137.716637,
                    "results": [
                        {
                            ... pgbench results of a test ...
                        }
                        ...
                    ]
                },
                ....
        },
        "rw": {
            ... read write test results ...
    },
    ... other tests results
}

7 years agoUpdate default pgbench and postgresql settings
Mark Wong [Tue, 6 Feb 2018 19:19:09 +0000 (11:19 -0800)]
Update default pgbench and postgresql settings

Make the default settings what we want to be tested.  The idea is to
minimize setup on the clients.

7 years agoUse collectd for system and postgres statistics
Mark Wong [Wed, 13 Dec 2017 19:54:43 +0000 (11:54 -0800)]
Use collectd for system and postgres statistics

7 years agoMove results directory check to start of client
Mark Wong [Sat, 3 Feb 2018 22:15:19 +0000 (14:15 -0800)]
Move results directory check to start of client

Between the benchmark and collector modules, it's not clear who might
have created the output directory first.  Leave a warning in the
benchmark modules for now, otherwise don't let it prevent the tests from
running.

7 years agoUse https instead of git protocol
Mark Wong [Thu, 1 Feb 2018 19:17:16 +0000 (11:17 -0800)]
Use https instead of git protocol

This is for ease of automated testing.  Using git:// can require manual
confirmation about updating the ssh known hosts file.

7 years agoAdd support for FreeBSD
Mark Wong [Wed, 31 Jan 2018 22:13:31 +0000 (14:13 -0800)]
Add support for FreeBSD

The memory calculations for pgbench will need to be platform specific.
Also the system data collection may have to be platform specific.  Thus
far, just added conditions so that things will run on FreeBSD now.

7 years agoSet default directories to /tmp
Mark Wong [Wed, 31 Jan 2018 00:04:25 +0000 (16:04 -0800)]
Set default directories to /tmp

Set the default directories in the client settings.py file to use /tmp.
This will make it easier to automate testing by reducing the number of
settings to configure.

7 years agoSet cwd when running pgbench
Mark Wong [Tue, 9 Jan 2018 17:15:08 +0000 (09:15 -0800)]
Set cwd when running pgbench

The output directory is used for the cwd when executing pgbench so that
the output files are saved in the results directory.

7 years agoUpdate ignore file for ctags
Mark Wong [Wed, 13 Dec 2017 19:51:43 +0000 (11:51 -0800)]
Update ignore file for ctags

7 years agoAdd sbin locations to PATH for sysctl
Mark Wong [Wed, 13 Dec 2017 19:31:16 +0000 (11:31 -0800)]
Add sbin locations to PATH for sysctl

The location of sysctl may change depending on the distribution.  Try to
add all possible paths to the environment.

7 years agoAdd requirements.txt for the client
Mark Wong [Wed, 27 Sep 2017 21:00:10 +0000 (14:00 -0700)]
Add requirements.txt for the client

7 years agoForce csv.DictWriter to create unix file format
Mark Wong [Mon, 7 Aug 2017 23:10:18 +0000 (16:10 -0700)]
Force csv.DictWriter to create unix file format

7 years agoAlways write results of a test to results.json
Mark Wong [Sun, 30 Jul 2017 23:19:35 +0000 (16:19 -0700)]
Always write results of a test to results.json

The idea is to keep things simple by having a directory of results per
test, and to have each directory with a consistently named results
files.

7 years agoCapture uname information
Mark Wong [Sat, 29 Jul 2017 00:51:05 +0000 (17:51 -0700)]
Capture uname information

It has some things that are not in the sysctl information.  Like
hardware architecture.

7 years agoStop storing log files as json
Mark Wong [Fri, 28 Jul 2017 00:06:06 +0000 (17:06 -0700)]
Stop storing log files as json

Keep the json results document small by saving log information
externally.  This include:

* postgres log
* pgbench output
* sar output

7 years agoFix use of env in call()
Mark Wong [Thu, 27 Jul 2017 00:51:59 +0000 (17:51 -0700)]
Fix use of env in call()

The env parameter to call() was clearing the PATH environment to only
have the postgres binary install path.  Modify only the PATH
environment.

7 years agoInstall pgbench from contrib for older versions
Mark Wong [Mon, 24 Jul 2017 04:50:14 +0000 (21:50 -0700)]
Install pgbench from contrib for older versions

Prior to 9.5, pgbench was in contrib.

7 years agoAlter postgres options on the command line
Mark Wong [Mon, 24 Jul 2017 01:45:06 +0000 (18:45 -0700)]
Alter postgres options on the command line

This allows testing of older versions of postgres.  Otherwise only
versions of postgres supporting ALTER SYSTEM (9.4+) will work with this
client.

7 years agoSaved timestamp from test start
Mark Wong [Fri, 28 Jul 2017 16:51:34 +0000 (09:51 -0700)]
Saved timestamp from test start

7 years agoUse python2.7 from environment settings
Mark Wong [Fri, 28 Jul 2017 16:50:14 +0000 (09:50 -0700)]
Use python2.7 from environment settings

7 years agoPythonic way to check for None
Mark Wong [Tue, 18 Jul 2017 20:55:51 +0000 (13:55 -0700)]
Pythonic way to check for None

Also check to make sure files exist before trying to collect them.

7 years agoFix check for csv flag
Mark Wong [Tue, 18 Jul 2017 19:44:14 +0000 (12:44 -0700)]
Fix check for csv flag

7 years agoFix handling of csv option
Mark Wong [Tue, 18 Jul 2017 18:00:24 +0000 (11:00 -0700)]
Fix handling of csv option

7 years agopep8 coding style
Mark Wong [Thu, 20 Jul 2017 16:38:15 +0000 (09:38 -0700)]
pep8 coding style

8 years agoalso add CSV output to pgbench
Tomas Vondra [Thu, 13 Oct 2016 13:39:10 +0000 (15:39 +0200)]
also add CSV output to pgbench

8 years agoadd support for collecting results into simple CSV file
Tomas Vondra [Thu, 13 Oct 2016 12:48:03 +0000 (14:48 +0200)]
add support for collecting results into simple CSV file

Originally the results were collected into a JSON format, which is
not very convenient when running ad-hoc benchmarks.

8 years agoadd shebang explicitly invoking Python 2.7
Tomas Vondra [Thu, 13 Oct 2016 12:27:25 +0000 (14:27 +0200)]
add shebang explicitly invoking Python 2.7

The code is not fully compatible with Python 3.x (although it should
not be a big deal to make that happen).

8 years agoexplicitly flush stdout in log()
Tomas Vondra [Thu, 13 Oct 2016 11:47:33 +0000 (13:47 +0200)]
explicitly flush stdout in log()

Otherwise the messages may get buffered for a quite long time, which
is annoying, particularly in multi-process system.

8 years agofix lockups in PostgreSQL collector
Tomas Vondra [Thu, 13 Oct 2016 11:41:43 +0000 (13:41 +0200)]
fix lockups in PostgreSQL collector

When the workers produce results and return them through a queue, we need
to first drain the queue and then wait for the thread to join(). Otherwise
it seems to lock up - the join() call with wait as long as there are items
in the queue. This also happens because we do both things (reading from
queue and waiting for the thread) in the same thread.

8 years agoopen connections in a way resilient to temporary failures
Tomas Vondra [Thu, 13 Oct 2016 11:22:49 +0000 (13:22 +0200)]
open connections in a way resilient to temporary failures

In case of connection failure (e.g. a backend getting terminated by
OOM killer and taking down the whole server), the workers failed
with an exception without putting a result into the queue, leading
to hangs.

This fixes that by making the workers resilient to connection issues
by catching the exceptions and terminating nicely. Also, we retry
connection several times, to continue with benchmarking even after
OOM hits us unexpectedly, etc.

8 years agoCheck configuration before running any benchmarks
Tomas Vondra [Thu, 11 Aug 2016 00:00:32 +0000 (02:00 +0200)]
Check configuration before running any benchmarks

Before doing any heavy-lifting, check existence of directories,
binaries and some other sanity checks.

8 years agoFix CWD when running 'git pull', fix parameters.
Tomas Vondra [Thu, 11 Aug 2016 00:08:06 +0000 (02:08 +0200)]
Fix CWD when running 'git pull', fix parameters.

When refreshing a reporitory clone, running 'git pull $dir' does
not do the trick - it assumes the CWD is the clone, and imports
data from repository at $dir. So we need to set CWD properly and
do just 'git pull'.

Also, prefix class members with 'self'.

8 years agoAdd .lock to .gitignore
Tomas Vondra [Wed, 10 Aug 2016 22:58:43 +0000 (00:58 +0200)]
Add .lock to .gitignore

8 years agoRework the perffarm client configuration.
Tomas Vondra [Wed, 10 Aug 2016 22:46:26 +0000 (00:46 +0200)]
Rework the perffarm client configuration.

Instead of configuration hardcoded into the perffrarm-client.py
script, move it into a separate settings.py file, and allow
override using settings_local.py (not required).

Also somewhat improve the benchmark configurations by allowing
specifying number of runs, durations etc. in the configuration
file (instead of using default parameter values).

8 years agoHandle missing sar stats gracefully
Tomas Vondra [Wed, 10 Aug 2016 22:06:11 +0000 (00:06 +0200)]
Handle missing sar stats gracefully

8 years agoImport initial version of the client
Tomas Vondra [Wed, 10 Aug 2016 21:23:55 +0000 (23:23 +0200)]
Import initial version of the client

9 years agoUpdate requirements.txt
Dave Page [Tue, 15 Mar 2016 11:36:06 +0000 (11:36 +0000)]
Update requirements.txt