xml.dom.minidom does not escape CR, LF and TAB characters within attribute values

[Tomalak]

BPO	5752
Nosy	@amauryfa, @devdanzin, @dvarrazzo, @ezio-melotti, @wking, @mitar, @moriyoshi
Files	minidom_test.py: A small test script that outlines the problem test_toxml.py: New test script to demonstrate that the patch proposed works test_multiline_roundtrip.diff: Tomalak's minidom_test.py as unittest, patch for test_minidom minidom.patch: Proposed patch to minidom.py minidom.diff: Old file before new file in patch. minidom.py: monkey patch module applying minidom.diff

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2009-04-14.11:13:06.464>
labels = ['expert-XML', 'easy', 'type-bug']
title = 'xml.dom.minidom does not escape CR, LF and TAB characters within attribute values'
updated_at = <Date 2019-06-22.20:28:40.662>
user = 'https://bugs.python.org/Tomalak'

bugs.python.org fields:

activity = <Date 2019-06-22.20:28:40.662>
actor = 'mitar'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['XML']
creation = <Date 2009-04-14.11:13:06.464>
creator = 'Tomalak'
dependencies = []
files = ['13921', '13960', '13966', '13977', '18184', '18325']
hgrepos = []
issue_num = 5752
keywords = ['patch', 'easy']
message_count = 33.0
messages = ['85964', '87178', '87332', '87333', '87351', '87429', '87431', '87432', '87526', '87527', '87528', '87564', '87586', '87590', '87604', '87609', '87676', '87679', '87681', '90548', '90676', '110999', '111493', '111642', '112497', '112570', '119728', '142964', '185398', '185408', '185419', '185423', '185576']
nosy_count = 13.0
nosy_names = ['effbot', 'amaury.forgeotdarc', 'ajaksu2', 'piro', 'ezio.melotti', 'sayap', 'Tomalak', 'devon', 'labrat', 'mitar', 'moriyoshi', 'gauges', 'Jens.Grivolla']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue5752'
versions = ['Python 2.6', 'Python 3.1', 'Python 2.7', 'Python 3.2']

Linked PRs

• gh-50002: xml.dom.minidom now preserves whitespaces in attributes #107947

[Tomalak]

Current behavior upon toxml() is:

<foo attribute="multiline
value" />

Upon reading the document again, the new line is normalized and
collapsed into a space (according to the XML spec, section 3.3.3), which
means that it is lost.

Better behavior would be something like this (within attribute values only):

<foo attribute="multiline value" />

[sechifrancesco]

Ok, I've tried to solve this problem, but I think that the keyword
'easy' is not suitable for this kind of task, because it is necessary to
modify the expat module that is very complex.

[Tomalak]

@francesco Sechi: Would it not just require a minimal change to the
_write_data() method? Something along the lines of (sorry, no Python
expert, maybe I am way off):

def _write_data(writer, data, is_attrib=False):
    "Writes datachars to writer."
    if is_attrib: 
        data = data.replace("\r", "
").replace("\n", "
")
    data = data.replace("&", "&amp;").replace("<", "&lt;")
    data = data.replace("\"", "&quot;").replace(">", "&gt;")
    writer.write(data)

and in Element.writexml():

    #[...]
    for a_name in a_names:
        writer.write(" %s=\"" % a_name)
        _write_data(writer, attrs[a_name].value, True)
    #[...]

[Tomalak]

Of course it should be:

def _write_data(writer, data, is_attrib=False):
    "Writes datachars to writer."
    data = data.replace("&", "&amp;").replace("<", "&lt;")
    data = data.replace("\"", "&quot;").replace(">", "&gt;")
    if is_attrib: 
        data = data.replace("\r", "&#13;").replace("\n", "&#10;")
    writer.write(data)

[sechifrancesco]

Don't worry, I'm a newer too.
No, your solution does not work, because the method you refer
(_write_data) is called by the toxml() function, but the newline is
replaced with a whitespace by the parsestring() function. The
parsestring function, as I already said, refers to the 'expat' module,
that is very complex (for me).

[Tomalak]

Hmm... I thought toxml() is the part that needs to be fixed, not the
parsing/reading. I mentioned the reading only to outline the data loss
that occurs eventually.

My point is: The toxml() (i.e. _write_data) *actually writes* the
newline to the output. And within parameters, it just shouldn't.

[Tomalak]

Attaching a patch that fixes the problem.

[Tomalak]

Attaching a test file that outlines the problem. Output on my system
(Windows / Python 3.0) is:

Without the patch:
C:\Python30>python.exe c:\minidom_test.py
False
1 -->"multiline
value"
2 -->"multiline value"

With the patch:
C:\Python30>python.exe c:\minidom_test.py
True
1 -->"multiline
value"
2 -->"multiline
value"

[sechifrancesco]

I think that the problem is: the xmldoc1 has the newline or not? If it
hasn't your patch works only in the particular case you pass a toxml
return value to 'parsestring'. If I pass an XML string with newlines it
doesn't work. So your solution is not generic and cannot be considered a
patch for the issue you proposed.

[sechifrancesco]

I try to explain better what is my opinion:

• If you add the attribute by using setAttribute the newlines are kept
  and the toxml works well

• If you add the attribute by using the parsestring, passing it an XML
  string the newlines are replaced

• Your patch works only if you act on a well-constructed (i.e.with
  newlines kept in internal data structures) xml.dom.minidom.Document
  object only, so...

• If you try to execute your patched toxml method of a
  xml.dom.minidom.Document constructed using parsestring passing it a
  string with newline it does not work.

So your patch works only in a specific case: you are trying to fix a
problem in parsestring, acting on its actual parameter.

[Tomalak]

Francesco, I think you are missing the point. :-) The problem has two sides.

If I create an XML document using the DOM (not by parsing it from a
string!), then I can put newline characters into attribute value. This
is allowed and conforms to the XML spec.

However, *literal* newlines in an attribute value (i.e. when the
document is parsed from a string) have no meaning. The parser treats
them as if they were insignificant whitespace -- they are converted to a
single space. This is also valid and conforms to the XML spec.

The catch: This leads to an actual data loss if I *wanted* to store
newline characters in an attribute -- unless the newline characters are
properly encoded. Encoding the newline characters is also valid and
conforms to the spec, so the DOM implementation should do it.

In other words - the parsing process you refer to is actually working
fine. If an attribute contains a literal newline, it is indeed okay to
collapse it into a space. It's only the document serializing that is broken.

Minidom is clearly missing functionality here, and it does not conform
to the XML spec. If I store a string of data in an XML document, it must
be ensured that upon reading the document again, I get the *same* data
back. This is what I check with my test script.