sqlite dumpiter dumps invalid script when virtual tables are used

[RonnyPfannschmidt]

BPO	20463
Nosy	@bitdancer, @RonnyPfannschmidt, @palaviv, @tirkarthi
Files	20463.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2014-01-31.19:57:36.445>
labels = ['extension-modules', 'type-bug']
title = 'sqlite dumpiter dumps invalid script when virtual tables are used'
updated_at = <Date 2018-09-23.16:16:06.541>
user = 'https://github.com/RonnyPfannschmidt'

bugs.python.org fields:

activity = <Date 2018-09-23.16:16:06.541>
actor = 'xtreak'
assignee = 'ghaering'
closed = False
closed_date = None
closer = None
components = ['Extension Modules']
creation = <Date 2014-01-31.19:57:36.445>
creator = 'Ronny.Pfannschmidt'
dependencies = []
files = ['45049']
hgrepos = []
issue_num = 20463
keywords = ['patch']
message_count = 3.0
messages = ['209825', '248823', '278437']
nosy_count = 5.0
nosy_names = ['ghaering', 'r.david.murray', 'Ronny.Pfannschmidt', 'palaviv', 'xtreak']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue20463'
versions = ['Python 2.7']

Linked PRs

• gh-64662: Add virtual table support to sqlite3.Connection.iterdump #108340
• [3.12] gh-64662: Fix virtual table support in sqlite3.Connection.iterdump (#108340) #108563
• [3.11] gh-64662: Fix virtual table support in sqlite3.Connection.iterdump (#108340) #108564
• gh-64662: Fix NEWS entry and remove What's New entry #108565

[RonnyPfannschmidt]

when using virtual tables, dumpiter generates a broken db script
virtual table entries must be created as master table entries
the sqlite tools dump does that correctly
however pythons iterdump seems to do that rather different and wrong

sqlite3 test.db "create virtual table test using fts4(example);"
-------------------------------
sqlite dump
PRAGMA foreign_keys=OFF;
BEGIN TRANSACTION;
PRAGMA writable_schema=ON;
INSERT INTO sqlite_master(type,name,tbl_name,rootpage,sql)VALUES('table','test','test',0,'CREATE VIRTUAL TABLE test using fts4(example)');
CREATE TABLE 'test_content'(docid INTEGER PRIMARY KEY, 'c0example');
CREATE TABLE 'test_segments'(blockid INTEGER PRIMARY KEY, block BLOB);
CREATE TABLE 'test_segdir'(level INTEGER,idx INTEGER,start_block INTEGER,leaves_end_block INTEGER,end_block INTEGER,root BLOB,PRIMARY KEY(level, idx));
CREATE TABLE 'test_docsize'(docid INTEGER PRIMARY KEY, size BLOB);
CREATE TABLE 'test_stat'(id INTEGER PRIMARY KEY, value BLOB);
PRAGMA writable_schema=OFF;
COMMIT;
------------------------------
python iterdump "import sqlite3;
c=sqlite3.connect("test.db");
print("\n".join(c.iterdump()))"
BEGIN TRANSACTION;
CREATE VIRTUAL TABLE test using fts4(example);
CREATE TABLE 'test_content'(docid INTEGER PRIMARY KEY, 'c0example');
CREATE TABLE 'test_docsize'(docid INTEGER PRIMARY KEY, size BLOB);
CREATE TABLE 'test_segdir'(level INTEGER,idx INTEGER,start_block INTEGER,leaves_end_block INTEGER,end_block INTEGER,root BLOB,PRIMARY KEY(level, idx));
CREATE TABLE 'test_segments'(blockid INTEGER PRIMARY KEY, block BLOB);
CREATE TABLE 'test_stat'(id INTEGER PRIMARY KEY, value BLOB);
COMMIT;

[ghaering]

apsw contains code that handles the issues with dumping SQLite databases very well. I plan to integrate this code into pysqlite. We can then later port the fix to the sqlite3 module.

See ghaering/pysqlite#10 for the tasks and
https://github.com/rogerbinns/apsw/blob/master/tools/shell.py#L1012 for the apsw code.

[palaviv]

Attached is patch with a fix for this issue. This is not as comprehensive as the solution in apsw but I think this should cover most of the cases. The result for Ronny dump after this fix is:

BEGIN TRANSACTION;
PRAGMA writable_schema=ON;
INSERT INTO sqlite_master(type,name,tbl_name,rootpage,sql)VALUES('table','test','test',0,'CREATE VIRTUAL TABLE test using fts4(example)');
CREATE TABLE 'test_content'(docid INTEGER PRIMARY KEY, 'c0example');
CREATE TABLE 'test_docsize'(docid INTEGER PRIMARY KEY, size BLOB);
CREATE TABLE 'test_segdir'(level INTEGER,idx INTEGER,start_block INTEGER,leaves_end_block INTEGER,end_block INTEGER,root BLOB,PRIMARY KEY(level, idx));
CREATE TABLE 'test_segments'(blockid INTEGER PRIMARY KEY, block BLOB);
CREATE TABLE 'test_stat'(id INTEGER PRIMARY KEY, value BLOB);
PRAGMA writable_schema=OFF;
COMMIT;

[erlend-aasland]

My plan for iterdump is to align it with the implementation in the SQLite shell, but I haven't gotten around to it yet. I think @palaviv's patch is a step in the right direction, and I don't think we should block that patch, just because iterdump also has other problems.