skytools.git
11 years agodebian/changelog: updated
martinko [Tue, 3 Dec 2013 14:46:52 +0000 (15:46 +0100)]
debian/changelog: updated

11 years agolondiste.handlers.dispatch: fixed "ignore events aiming at obsolete partitions" bug
martinko [Tue, 3 Dec 2013 14:46:09 +0000 (15:46 +0100)]
londiste.handlers.dispatch: fixed "ignore events aiming at obsolete partitions" bug

but really :-)

11 years agosql/pgq: tag pgq.batch_id_seq as dumpable (extension)
Marko Kreen [Wed, 30 Oct 2013 23:35:05 +0000 (01:35 +0200)]
sql/pgq: tag pgq.batch_id_seq as dumpable (extension)

That case was broken in old 9.1 releases, fixed since 9.1.7.

The sequence will be owned by extension and without tagging
it will not be dumped in data dump.

11 years agopython/pgq: fixes (event & walker)
martinko [Mon, 11 Nov 2013 14:22:53 +0000 (15:22 +0100)]
python/pgq: fixes (event & walker)

11 years agoMerge branch 'master' of skype-git:/git/dba/skytools-3
martinko [Fri, 20 Sep 2013 14:09:06 +0000 (16:09 +0200)]
Merge branch 'master' of skype-git:/git/dba/skytools-3

11 years agoMerge branch 'hotfix/3.1.5.3' 3.1.5.3
martinko [Fri, 20 Sep 2013 12:40:41 +0000 (14:40 +0200)]
Merge branch 'hotfix/3.1.5.3'

11 years agodebian/changelog: updated
martinko [Fri, 20 Sep 2013 12:39:32 +0000 (14:39 +0200)]
debian/changelog: updated

11 years agolondiste.handlers.dispatch: fixed "ignore events aiming at obsolete partitions" bug
martinko [Thu, 19 Sep 2013 15:30:18 +0000 (17:30 +0200)]
londiste.handlers.dispatch: fixed "ignore events aiming at obsolete partitions" bug

11 years agodeb: include data_maintainer3 in package
Marko Kreen [Wed, 18 Sep 2013 12:41:16 +0000 (15:41 +0300)]
deb: include data_maintainer3 in package

11 years agosql/pgq: fix retry-queue vs. newgrants
Marko Kreen [Sun, 18 Aug 2013 15:27:33 +0000 (18:27 +0300)]
sql/pgq: fix retry-queue vs. newgrants

11 years agoAdd timeout for add-table operation to avoid possible locking issues.
Petr Jelinek [Wed, 4 Sep 2013 12:55:29 +0000 (14:55 +0200)]
Add timeout for add-table operation to avoid possible locking issues.

11 years agoMerge remote-tracking branch 'gamato/master'
Marko Kreen [Fri, 9 Aug 2013 14:51:02 +0000 (17:51 +0300)]
Merge remote-tracking branch 'gamato/master'

11 years agoMerge branch 'release/skytools_3_1_5' skytools_3_1_5
martinko [Fri, 9 Aug 2013 14:03:34 +0000 (16:03 +0200)]
Merge branch 'release/skytools_3_1_5'

11 years agov3.1.5
martinko [Fri, 9 Aug 2013 14:01:16 +0000 (16:01 +0200)]
v3.1.5

11 years agoMerge remote-tracking branch 'gamato/develop'
Marko Kreen [Wed, 31 Jul 2013 09:36:53 +0000 (12:36 +0300)]
Merge remote-tracking branch 'gamato/develop'

11 years agoMerge branch 'develop' into release/skytools_3_1_5
martinko [Wed, 31 Jul 2013 09:26:01 +0000 (11:26 +0200)]
Merge branch 'develop' into release/skytools_3_1_5

11 years agolondiste.handlers.shard: fixed incomplete docstring for part handler
martinko [Wed, 31 Jul 2013 09:21:51 +0000 (11:21 +0200)]
londiste.handlers.shard: fixed incomplete docstring for part handler

11 years agopreparation for upcoming release (updating files)
martinko [Fri, 26 Jul 2013 12:09:51 +0000 (14:09 +0200)]
preparation for upcoming release (updating files)

11 years agopgq_node.get_node_info: document node_attrs
Marko Kreen [Fri, 26 Jul 2013 11:54:01 +0000 (14:54 +0300)]
pgq_node.get_node_info: document node_attrs

11 years agolondiste/repair: made less verbose
martinko [Thu, 18 Jul 2013 16:13:54 +0000 (18:13 +0200)]
londiste/repair: made less verbose

(in response to "londiste3 repair command floods the console with the diff mesages")

11 years agoMerge branch 'master' of https://github.com/markokr/skytools into develop
martinko [Wed, 10 Jul 2013 10:14:37 +0000 (12:14 +0200)]
Merge branch 'master' of https://github.com/markokr/skytools into develop

11 years agoMerge branch 'feature/skylog_tnetstrings_via_udp' into develop
martinko [Wed, 10 Jul 2013 10:13:29 +0000 (12:13 +0200)]
Merge branch 'feature/skylog_tnetstrings_via_udp' into develop

11 years agoMerge remote-tracking branch 'tarvip/walmgr-createslave'
Marko Kreen [Tue, 9 Jul 2013 10:35:10 +0000 (13:35 +0300)]
Merge remote-tracking branch 'tarvip/walmgr-createslave'

11 years agowalmgr: new command createslave and new option --synch-standby
Tarvi Pillessaar [Mon, 10 Jun 2013 06:20:35 +0000 (09:20 +0300)]
walmgr: new command createslave and new option --synch-standby

11 years agoMerge remote-tracking branch 'petr/master'
Marko Kreen [Mon, 8 Jul 2013 10:24:14 +0000 (13:24 +0300)]
Merge remote-tracking branch 'petr/master'

11 years agopgq: move RetriableEvent to consumer.py
Marko Kreen [Sun, 7 Jul 2013 14:50:07 +0000 (17:50 +0300)]
pgq: move RetriableEvent to consumer.py

It's hard to get overview if the retry-related code
is laying around in different places.

As only Consumer class deals with retry, move all crap
into same file.

11 years agopgq.Consumer: non-lazy loading should use RetriableEvent
Marko Kreen [Sun, 7 Jul 2013 14:45:10 +0000 (17:45 +0300)]
pgq.Consumer: non-lazy loading should use RetriableEvent

11 years agopgq.logutriga: don't quote '-'
Marko Kreen [Tue, 2 Jul 2013 13:13:45 +0000 (16:13 +0300)]
pgq.logutriga: don't quote '-'

Not needed and now logutriga() behavior matches
skytools.db_urlencode() and urllib.quote_plus().

11 years agoAdd consumer_filter option to simple_local_consumer.
Petr Jelinek [Tue, 2 Jul 2013 09:53:59 +0000 (11:53 +0200)]
Add consumer_filter option to simple_local_consumer.

11 years agoset_tcp_keepalive: accept socket object directly
Marko Kreen [Fri, 21 Jun 2013 09:24:46 +0000 (12:24 +0300)]
set_tcp_keepalive: accept socket object directly

11 years agoMerge branch 'master' of https://github.com/markokr/skytools into develop
martinko [Thu, 20 Jun 2013 19:49:32 +0000 (21:49 +0200)]
Merge branch 'master' of https://github.com/markokr/skytools into develop

11 years agolondiste: resync now checks if table is available on provider
martinko [Thu, 20 Jun 2013 19:42:23 +0000 (21:42 +0200)]
londiste: resync now checks if table is available on provider

11 years agopgq: Consumer class also need _make_event, for _load_batch_events_old()
Marko Kreen [Thu, 20 Jun 2013 17:39:26 +0000 (20:39 +0300)]
pgq: Consumer class also need _make_event, for _load_batch_events_old()

11 years agolondiste: fix multi -v passing
Marko Kreen [Wed, 19 Jun 2013 21:17:04 +0000 (00:17 +0300)]
londiste: fix multi -v passing

11 years agoMerge remote-tracking branch 'gamato/develop'
Marko Kreen [Wed, 19 Jun 2013 21:17:26 +0000 (00:17 +0300)]
Merge remote-tracking branch 'gamato/develop'

11 years agonews: fix date
Marko Kreen [Wed, 19 Jun 2013 20:22:17 +0000 (23:22 +0300)]
news: fix date

11 years agolondiste --wait-sync: show progress in absolute numbers
Marko Kreen [Wed, 19 Jun 2013 20:20:08 +0000 (23:20 +0300)]
londiste --wait-sync: show progress in absolute numbers

This gives more information and is less random.

12 years agolondiste: add-table supports --find-copy-node working with --create now
martinko [Mon, 27 May 2013 18:12:50 +0000 (20:12 +0200)]
londiste: add-table supports --find-copy-node working with --create now

12 years agoquerybuilder: show list of missing arguments on KeyError
Marko Kreen [Sun, 26 May 2013 19:55:38 +0000 (22:55 +0300)]
querybuilder: show list of missing arguments on KeyError

12 years agolondiste: resync supports --find-copy-node and --copy-node options now
martinko [Mon, 20 May 2013 11:23:25 +0000 (13:23 +0200)]
londiste: resync supports --find-copy-node and --copy-node options now

12 years agolondiste.playback: support multiple -v options for workers
martinko [Fri, 17 May 2013 13:10:31 +0000 (15:10 +0200)]
londiste.playback: support multiple -v options for workers

12 years agolondiste.handlers: renamed handler 'part' to 'shard'
martinko [Thu, 16 May 2013 14:52:09 +0000 (16:52 +0200)]
londiste.handlers: renamed handler 'part' to 'shard'

It was getting more and more confusing, especially since 'dispatch' handler mixed both partitioning types (table and database) in one class.

12 years agolondiste.handlers.dispatch: added switch to ignore events aiming at obsolete (dropped...
martinko [Thu, 16 May 2013 12:59:23 +0000 (14:59 +0200)]
londiste.handlers.dispatch: added switch to ignore events aiming at obsolete (dropped) partitions

12 years agolondiste.handlers: fixes
martinko [Wed, 15 May 2013 14:43:56 +0000 (16:43 +0200)]
londiste.handlers: fixes

12 years agoMerge branch 'feature/dispatch_handler_with_sharding' into develop
martinko [Wed, 15 May 2013 12:32:06 +0000 (14:32 +0200)]
Merge branch 'feature/dispatch_handler_with_sharding' into develop

12 years agolondiste add-table: --skip-non-existing ignores tables not on provider
Marko Kreen [Tue, 14 May 2013 11:47:16 +0000 (14:47 +0300)]
londiste add-table: --skip-non-existing ignores tables not on provider

12 years agolondiste: moved hash_key logic from dispatch to part handler
martinko [Thu, 2 May 2013 09:46:53 +0000 (11:46 +0200)]
londiste: moved hash_key logic from dispatch to part handler

12 years agolondiste: changed handler argument 'key' to 'hash_key'
martinko [Thu, 2 May 2013 09:19:42 +0000 (11:19 +0200)]
londiste: changed handler argument 'key' to 'hash_key'

12 years agoskytools.timeutil: make tests more robust
Marko Kreen [Tue, 30 Apr 2013 10:57:14 +0000 (13:57 +0300)]
skytools.timeutil: make tests more robust

microsec -> float secs conversion can lose some precision

12 years agoskytools.skylog: added more fields to UdpTNetStringsHandler log message
martinko [Tue, 30 Apr 2013 09:49:17 +0000 (11:49 +0200)]
skytools.skylog: added more fields to UdpTNetStringsHandler log message

12 years agoskytools.skylog: moved hostaddr resolving to module init
martinko [Tue, 30 Apr 2013 09:38:00 +0000 (11:38 +0200)]
skytools.skylog: moved hostaddr resolving to module init

No need to do it per log call as hostname is constant currently.

12 years agonews: fix typos
Marko Kreen [Thu, 25 Apr 2013 13:45:49 +0000 (16:45 +0300)]
news: fix typos

12 years agoskytools.scripting: more safety in run_func_safely()
martinko [Wed, 24 Apr 2013 12:13:58 +0000 (15:13 +0300)]
skytools.scripting: more safety in run_func_safely()

Scenario resolved:
startup() fails and some variables are never initialised.  those are used in overridden send_stats() which is called from generic exception handler in run_func_safely().  thus a new exception is raised that would obscure the original exception raised in startup().

12 years agolondiste: initial implementation of shard-ed Dispatch handler
martinko [Thu, 18 Apr 2013 14:06:56 +0000 (16:06 +0200)]
londiste: initial implementation of shard-ed Dispatch handler

12 years agoscripts/data_maintainer.py: added suffix "3" to service name
martinko [Thu, 18 Apr 2013 13:55:38 +0000 (15:55 +0200)]
scripts/data_maintainer.py: added suffix "3" to service name

12 years agoskytools.skylog: initial implementation of UdpTNetStringsHandler
martinko [Thu, 18 Apr 2013 13:30:35 +0000 (15:30 +0200)]
skytools.skylog: initial implementation of UdpTNetStringsHandler

12 years agolondiste copy: fix data filtering when parallel_copies > 1
Marko Kreen [Thu, 18 Apr 2013 11:55:25 +0000 (14:55 +0300)]
londiste copy: fix data filtering when parallel_copies > 1

The problem appears in 'catching-up' phase were code checked
table status only and not whether the table is owned by
current process.  Then events can be replayed by several copy
processes when they are all in catching-up phase.

The problem does not appear when local_only=1.

Also remove unused function.

12 years agolondiste status: make node name easier to select
Marko Kreen [Thu, 18 Apr 2013 11:49:07 +0000 (14:49 +0300)]
londiste status: make node name easier to select

12 years agograntfu: 2-pass processing
Marko Kreen [Sat, 30 Mar 2013 18:28:21 +0000 (20:28 +0200)]
grantfu: 2-pass processing

First revokes, then grants.

This supports one-object-in-several-sections situation.

Although that is bad style and better avoided, current
behavour results in unobvious breakage.

12 years agolondiste add: --skip-non-existing option
Marko Kreen [Sat, 30 Mar 2013 18:27:26 +0000 (20:27 +0200)]
londiste add: --skip-non-existing option

12 years agoparse_pgarray: allow None
Marko Kreen [Fri, 29 Mar 2013 11:33:41 +0000 (13:33 +0200)]
parse_pgarray: allow None

12 years agov3.1.4 skytools_3_1_4
Marko Kreen [Wed, 17 Apr 2013 08:30:38 +0000 (11:30 +0300)]
v3.1.4

12 years agodoc: mention config option in create-* commands
Marko Kreen [Wed, 17 Apr 2013 08:28:41 +0000 (11:28 +0300)]
doc: mention config option in create-* commands

12 years agotodo: remove few done items
Marko Kreen [Tue, 16 Apr 2013 19:24:59 +0000 (22:24 +0300)]
todo: remove few done items

12 years agopython: text fixes
martinko [Mon, 15 Apr 2013 15:54:32 +0000 (17:54 +0200)]
python: text fixes

12 years agolondiste: inform about no tables passed on cmd line (thus nothing to do)
martinko [Mon, 15 Apr 2013 15:54:04 +0000 (17:54 +0200)]
londiste: inform about no tables passed on cmd line (thus nothing to do)

12 years agoMerge branch 'master' of git://github.com/markokr/skytools
Marko Kreen [Tue, 2 Apr 2013 19:36:56 +0000 (22:36 +0300)]
Merge branch 'master' of git://github.com/markokr/skytools

12 years agocascadeadmin: downgrade missing host= error.
Marko Kreen [Tue, 2 Apr 2013 19:34:56 +0000 (22:34 +0300)]
cascadeadmin: downgrade missing host= error.

It's not 100% certain that user does not want such
connect strings, so erroring out is not right.

12 years agoMerge pull request #27 from kitsemets/master
Marko Kreen [Tue, 2 Apr 2013 11:21:56 +0000 (04:21 -0700)]
Merge pull request #27 from kitsemets/master

libtool added as build dependency into installation instructions

12 years agolibtool added as build dependency
kaarel [Mon, 1 Apr 2013 17:00:55 +0000 (10:00 -0700)]
libtool added as build dependency

12 years agolondiste: bump version to 3.1.4
Marko Kreen [Tue, 26 Mar 2013 09:53:11 +0000 (11:53 +0200)]
londiste: bump version to 3.1.4

12 years agotimeutil.datetime_to_timestamp: new function
Marko Kreen [Sat, 23 Mar 2013 12:13:51 +0000 (14:13 +0200)]
timeutil.datetime_to_timestamp: new function

converts datetime() to posix timestamp.

12 years agoUpdate AUTHORS
Marko Kreen [Sat, 23 Mar 2013 12:09:57 +0000 (14:09 +0200)]
Update AUTHORS

12 years agoDBScript.add_connect_string_profile: single place for profile logic
Marko Kreen [Fri, 22 Mar 2013 16:00:55 +0000 (18:00 +0200)]
DBScript.add_connect_string_profile: single place for profile logic

12 years agostatus: forgot to drop cstr_extra argument
Marko Kreen [Fri, 22 Mar 2013 15:43:47 +0000 (17:43 +0200)]
status: forgot to drop cstr_extra argument

12 years agocascadeadmin: cleanup of profile hack in status command
Marko Kreen [Fri, 22 Mar 2013 15:25:26 +0000 (17:25 +0200)]
cascadeadmin: cleanup of profile hack in status command

12 years agolondiste: connection profiles
Marko Kreen [Thu, 21 Mar 2013 21:16:07 +0000 (23:16 +0200)]
londiste: connection profiles

Way to add extra connect string parameters for non-config
connection strings (public node locations).

12 years agotests/londiste/regen.sh: test various combination of options
Marko Kreen [Thu, 21 Mar 2013 21:39:04 +0000 (23:39 +0200)]
tests/londiste/regen.sh: test various combination of options

12 years agocascadeadmin: forgot to make node_name optional everywhere
Marko Kreen [Thu, 21 Mar 2013 21:20:24 +0000 (23:20 +0200)]
cascadeadmin: forgot to make node_name optional everywhere

12 years agoMerge remote-tracking branch 'gamato/master'
Marko Kreen [Thu, 21 Mar 2013 16:59:14 +0000 (18:59 +0200)]
Merge remote-tracking branch 'gamato/master'

12 years agocascadeadmin: avoid hardwired db name
Marko Kreen [Wed, 20 Mar 2013 17:37:06 +0000 (19:37 +0200)]
cascadeadmin: avoid hardwired db name

12 years agolondiste create: load node name and initial provider also from config
Marko Kreen [Wed, 20 Mar 2013 15:54:39 +0000 (17:54 +0200)]
londiste create: load node name and initial provider also from config

12 years agoadminscripts: make info commands not use pidfile
Marko Kreen [Wed, 20 Mar 2013 15:17:15 +0000 (17:17 +0200)]
adminscripts: make info commands not use pidfile

otherwise they conflict with change operations.

12 years agolondiste.create_trigger: added check for after-trigger being overshadowed by skip...
martinko [Tue, 19 Mar 2013 15:04:49 +0000 (16:04 +0100)]
londiste.create_trigger: added check for after-trigger being overshadowed by skip-trigger

12 years agolondiste.create_trigger: fixed compatibility with postgres 9+
martinko [Tue, 19 Mar 2013 14:43:12 +0000 (15:43 +0100)]
londiste.create_trigger: fixed compatibility with postgres 9+

12 years agopython: text fixes
martinko [Wed, 6 Mar 2013 16:47:36 +0000 (17:47 +0100)]
python: text fixes

12 years agoMerge branch 'master' of https://github.com/markokr/skytools
Tarvi Pillessaar [Tue, 5 Mar 2013 14:02:46 +0000 (16:02 +0200)]
Merge branch 'master' of https://github.com/markokr/skytools

12 years agoMerge remote-tracking branch 'gamato/master'
Marko Kreen [Tue, 5 Mar 2013 13:34:49 +0000 (15:34 +0200)]
Merge remote-tracking branch 'gamato/master'

12 years agonewgrants/pgq_node: another fix
Marko Kreen [Tue, 5 Mar 2013 13:31:35 +0000 (15:31 +0200)]
newgrants/pgq_node: another fix

12 years agoMerge branch 'master' of https://github.com/markokr/skytools
martinko [Tue, 5 Mar 2013 13:26:14 +0000 (14:26 +0100)]
Merge branch 'master' of https://github.com/markokr/skytools

12 years agodocs
martinko [Tue, 5 Mar 2013 13:25:50 +0000 (14:25 +0100)]
docs

12 years agowalmgr: small loglevel changes
Tarvi Pillessaar [Tue, 5 Mar 2013 07:44:08 +0000 (09:44 +0200)]
walmgr: small loglevel changes

12 years agodata_maintainer: better way to handle .looping and .loop_delay
Marko Kreen [Mon, 4 Mar 2013 13:37:13 +0000 (15:37 +0200)]
data_maintainer: better way to handle .looping and .loop_delay

12 years agoloop_delay: let class variable set default
Marko Kreen [Mon, 4 Mar 2013 12:55:36 +0000 (14:55 +0200)]
loop_delay: let class variable set default

This way it's easier to change default in subclass.

12 years agoMerge branch 'master' of https://github.com/markokr/skytools
martinko [Mon, 4 Mar 2013 09:55:03 +0000 (10:55 +0100)]
Merge branch 'master' of https://github.com/markokr/skytools

12 years agoscripts/data_maintainer.py: fixes
martinko [Mon, 4 Mar 2013 09:48:02 +0000 (10:48 +0100)]
scripts/data_maintainer.py: fixes

12 years agoMerge remote-tracking branch 'petr/master'
Marko Kreen [Mon, 4 Mar 2013 09:35:06 +0000 (11:35 +0200)]
Merge remote-tracking branch 'petr/master'

12 years agoRevert "setup_skytools: install grant files under /usr/share/skytools"
Marko Kreen [Mon, 4 Mar 2013 09:33:09 +0000 (11:33 +0200)]
Revert "setup_skytools: install grant files under /usr/share/skytools"

This reverts commit c63236f11753c06a10dee125f2dc1cd241e92714.

setup_skytools.py wants ability to regenerate the fixes in pure-Python,
which is pain for grant files.  Will look into it later.

12 years agoMore newgrants fixes..
Marko Kreen [Mon, 4 Mar 2013 09:23:29 +0000 (11:23 +0200)]
More newgrants fixes..

12 years agolondiste: actually execute the ENABLE/DISABLE RULE query in londiste.create_partition
Petr Jelinek [Sun, 3 Mar 2013 23:58:55 +0000 (00:58 +0100)]
londiste: actually execute the ENABLE/DISABLE RULE query in londiste.create_partition