Add 'noError' argument to encoding conversion functions.
authorHeikki Linnakangas <heikki.linnakangas@iki.fi>
Thu, 1 Apr 2021 08:45:22 +0000 (11:45 +0300)
committerHeikki Linnakangas <heikki.linnakangas@iki.fi>
Thu, 1 Apr 2021 08:45:22 +0000 (11:45 +0300)
commitea1b99a6619cd9dcfd46b82ac0d926b0b80e0ae9
tree1a2c73601043edd7d6512fba51859cc860215e15
parente2639a767bfa1afebaf1877515a1187feb393443
Add 'noError' argument to encoding conversion functions.

With the 'noError' argument, you can try to convert a buffer without
knowing the character boundaries beforehand. The functions now need to
return the number of input bytes successfully converted.

This is is a backwards-incompatible change, if you have created a custom
encoding conversion with CREATE CONVERSION. This adds a check to
pg_upgrade for that, refusing the upgrade if there are any user-defined
encoding conversions. Custom conversions are very rare, there are no
commonly used extensions that I know of that uses that feature. No other
objects can depend on conversions, so if you do have one, you can fairly
easily drop it before upgrading, and recreate it after the upgrade with
an updated version.

Add regression tests for built-in encoding conversions. This doesn't cover
every conversion, but it covers all the internal functions in conv.c that
are used to implement the conversions.

Reviewed-by: John Naylor
Discussion: https://www.postgresql.org/message-id/e7861509-3960-538a-9025-b75a61188e01%40iki.fi
40 files changed:
doc/src/sgml/ref/create_conversion.sgml
src/backend/commands/conversioncmds.c
src/backend/utils/error/elog.c
src/backend/utils/mb/conv.c
src/backend/utils/mb/conversion_procs/cyrillic_and_mic/cyrillic_and_mic.c
src/backend/utils/mb/conversion_procs/euc2004_sjis2004/euc2004_sjis2004.c
src/backend/utils/mb/conversion_procs/euc_cn_and_mic/euc_cn_and_mic.c
src/backend/utils/mb/conversion_procs/euc_jp_and_sjis/euc_jp_and_sjis.c
src/backend/utils/mb/conversion_procs/euc_kr_and_mic/euc_kr_and_mic.c
src/backend/utils/mb/conversion_procs/euc_tw_and_big5/euc_tw_and_big5.c
src/backend/utils/mb/conversion_procs/latin2_and_win1250/latin2_and_win1250.c
src/backend/utils/mb/conversion_procs/latin_and_mic/latin_and_mic.c
src/backend/utils/mb/conversion_procs/utf8_and_big5/utf8_and_big5.c
src/backend/utils/mb/conversion_procs/utf8_and_cyrillic/utf8_and_cyrillic.c
src/backend/utils/mb/conversion_procs/utf8_and_euc2004/utf8_and_euc2004.c
src/backend/utils/mb/conversion_procs/utf8_and_euc_cn/utf8_and_euc_cn.c
src/backend/utils/mb/conversion_procs/utf8_and_euc_jp/utf8_and_euc_jp.c
src/backend/utils/mb/conversion_procs/utf8_and_euc_kr/utf8_and_euc_kr.c
src/backend/utils/mb/conversion_procs/utf8_and_euc_tw/utf8_and_euc_tw.c
src/backend/utils/mb/conversion_procs/utf8_and_gb18030/utf8_and_gb18030.c
src/backend/utils/mb/conversion_procs/utf8_and_gbk/utf8_and_gbk.c
src/backend/utils/mb/conversion_procs/utf8_and_iso8859/utf8_and_iso8859.c
src/backend/utils/mb/conversion_procs/utf8_and_iso8859_1/utf8_and_iso8859_1.c
src/backend/utils/mb/conversion_procs/utf8_and_johab/utf8_and_johab.c
src/backend/utils/mb/conversion_procs/utf8_and_sjis/utf8_and_sjis.c
src/backend/utils/mb/conversion_procs/utf8_and_sjis2004/utf8_and_sjis2004.c
src/backend/utils/mb/conversion_procs/utf8_and_uhc/utf8_and_uhc.c
src/backend/utils/mb/conversion_procs/utf8_and_win/utf8_and_win.c
src/backend/utils/mb/mbutils.c
src/bin/pg_upgrade/check.c
src/include/catalog/catversion.h
src/include/catalog/pg_proc.dat
src/include/mb/pg_wchar.h
src/test/regress/expected/conversion.out
src/test/regress/expected/opr_sanity.out
src/test/regress/input/create_function_1.source
src/test/regress/output/create_function_1.source
src/test/regress/regress.c
src/test/regress/sql/conversion.sql
src/test/regress/sql/opr_sanity.sql