Rework refactoring of hex and encoding routines

This commit addresses some issues with c3826f83 that moved the hex
decoding routine to src/common/:
- The decoding function lacked overflow checks, so when used for
security-related features it was an open door to out-of-bound writes if
not carefully used that could remain undetected.  Like the base64
routines already in src/common/ used by SCRAM, this routine is reworked
to check for overflows by having the size of the destination buffer
passed as argument, with overflows checked before doing any writes.
- The encoding routine was missing.  This is moved to src/common/ and
it gains the same overflow checks as the decoding part.

On failure, the hex routines of src/common/ issue an error as per the
discussion done to make them usable by frontend tools, but not by shared
libraries.  Note that this is why ECPG is left out of this commit, and
it still includes a duplicated logic doing hex encoding and decoding.

While on it, this commit uses better variable names for the source and
destination buffers in the existing escape and base64 routines in
encode.c and it makes them more robust to overflow detection.  The
previous core code issued a FATAL after doing out-of-bound writes if
going through the SQL functions, which would be enough to detect
problems when working on changes that impacted this area of the
code.  Instead, an error is issued before doing an out-of-bound write.
The hex routines were being directly called for bytea conversions and
backup manifests without such sanity checks.  The current calls happen
to not have any problems, but careless uses of such APIs could easily
lead to CVE-class bugs.

Author: Bruce Momjian, Michael Paquier
Reviewed-by: Sehrope Sarkuni
Discussion: https://postgr.es/m/20201231003557.GB22199@momjian.us

2 files changed, 25 insertions, 16 deletions

diff --git a/src/include/common/hex.h b/src/include/common/hex.h
new file mode 100644
index 00000000000..3c3c956bb66
--- /dev/null
+++ b/src/include/common/hex.h
@@ -0,0 +1,25 @@
+/*------------------------------------------------------------------------
+ *
+ *	hex.h
+ *	  Encoding and decoding routines for hex strings.
+ *
+ *	Portions Copyright (c) 1996-2020, PostgreSQL Global Development Group
+ *	Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ *		  src/include/common/hex.h
+ *
+ *------------------------------------------------------------------------
+ */
+
+#ifndef COMMON_HEX_H
+#define COMMON_HEX_H
+
+extern uint64 pg_hex_decode(const char *src, size_t srclen,
+							char *dst, size_t dstlen);
+extern uint64 pg_hex_encode(const char *src, size_t srclen,
+							char *dst, size_t dstlen);
+extern uint64 pg_hex_enc_len(size_t srclen);
+extern uint64 pg_hex_dec_len(size_t srclen);
+
+#endif							/* COMMON_HEX_H */
diff --git a/src/include/common/hex_decode.h b/src/include/common/hex_decode.h
deleted file mode 100644
index 29ab2484585..00000000000
--- a/src/include/common/hex_decode.h
+++ /dev/null
@@ -1,16 +0,0 @@
-/*
- *	hex_decode.h
- *		hex decoding
- *
- *	Portions Copyright (c) 1996-2021, PostgreSQL Global Development Group
- *	Portions Copyright (c) 1994, Regents of the University of California
- *
- *	src/include/common/hex_decode.h
- */
-#ifndef COMMON_HEX_DECODE_H
-#define COMMON_HEX_DECODE_H
-
-extern uint64 hex_decode(const char *src, size_t len, char *dst);
-
-
-#endif							/* COMMON_HEX_DECODE_H */