Non-ASCII char support for Windows terminals

[Eskibear]

When launching in integrated/external terminal, the debugger passes commands to vscode, and vscode assembles and sends it to the terminal. But if there's non-ascii char in code/path, the java process will have unexpected behavior, because of bad Unicode support in Windows.

Either VS Code or the extension should do something to cover these cases.

Example

Folder structure:

中文目录
├── Hello.class
└── Hello.java
0 directories, 2 files

Content of file 中文目录/Hello.java:

public class Hello {
  public static void main(String[] args) {
    System.out.println("中文");
  }
}

a) on Mac OSX

Works as expected

$> java -cp 中文目录 Hello
中文

b) on Windows

Non-ASCII char in classpath

Classpath is not correctly recognized by Java, if it contains non-ascii char.

D:\Test>java -cp 中文目录 Hello
Error: Could not find or load main class Hello

Non-ASCII char in stdout

cd into the folder, the classpath is correctly recognized now, but the terminal fails to print non-ascii chars.

D:\Test>cd 中文目录

D:\Test\中文目录>java -cp . Hello
ä??æ??

To display non-ascii char in Windows terminals, you have to modify the code page, see https://ss64.com/nt/chcp.html

Workaround: set code page to 65001 (utf-8)

D:\Test\中文目录>chcp 65001
Active code page: 65001

D:\Test\中文目录>java -cp . Hello
中文

[Eskibear]

Currently in v0.21.0 we use a launcher.bat to set code page to 65001 before launching the java process.
But it's reported to have issues:

• If IT Policy forbid to run .bat files, debugging is totally blocked. Launching BAT for UTF-8 fix is incompatible with Security Policies disallowing BAT execution. #646
• [Fixed] It doesn't work well with Git Bash for Windows for some users. Debugger just broken with git bash as default shell on windows. #642
• [Fixed] For MacOS, commands sent to integrated terminal sometimes is incomplete, thus debuggee process is not started, leading to debugger timeout. incomplete command for launching debuggee in integrated terminal #614 Failed to launch debuggee in terminal. Reason: Failed to launch debuggee in terminal. Reason: java.util.concurrent.TimeoutException: timeout #651

[Eskibear]

Non-ASCII char in classpath
Classpath is not correctly recognized by Java, if it contains non-ascii char.

D:\Test>java -cp 中文目录 Hello
Error: Could not find or load main class Hello

Tried in 1.51.1, above issue has been fixed, but non-ascii stdout is still not working well. see below.

[testforstephen]

Yan Zhang (@Eskibear) Have a loot at the screenshot above, Hello.java file is using UTF-8, but the javac and java command is using the default platform encoding, which is Chinese encoding. That's why Chinese character is not printed well.

[Eskibear]

Jinbo Wang (@testforstephen) Yes, as shown in my screenshot, it's cp936 (Chinese GB2312 encoding). The solutions can be either:

• change file encoding to GB2312 (cp936).
• change code page of the terminal to 65001 (UTF-8).

So I don't think it's an issue of Java debugger extension. It's due to inconsistent encoding between .java file and terminal, which also results in bad output even outside VS Code. I think we can close this one now.

[NinelieZ]

In my situation, stdout works well, but it works wrong when i trying to input a Chinese string with Java Extension Pack.

[imba-tjd]

I'd like to discuss this issue.

First of all, according to my test, C C++ C# java can't read Chinese characters (actually including all characters beyond ASCII) under chcp 65001.

The example you showed above works for me under 936. (Let's talk about 乱码 LATER)

Do you have a good reason to use chcp 65001?

[Eskibear]

Do you have a good reason to use chcp 65001?

Use UTF8 for better compatibility. Inconsistent encoding can happen among System(Windows), terminal(PS), JVM, editor (VSCode)...etc.

If you have issue with that, you can open C:\Users\<username>\.vscode\extensions\vscjava.vscode-java-debug-<version>\scripts\launcher.bat and change code page back to 936. Let me know if that works for all your use cases, e.g.

• print Chinese chars
• input Chinese chars in terminal (CMD/PS)
• work with path containing Chinese chars

[imba-tjd]

Let me show you how the program works when using CLI. The following tests are not related to launcher.bat and vscode-java-debug. The Code Page is 936.

One way is using GBK as the file encoding.

When using UTF8 file encoding, simply use javac -encoding utf8.

But there is no way to work using java test.java directly.

The reason is that file.encoding by default is corresponding to CP 936 GBK. When not set -Dfile.encoding or set it to GBK, it tries to read source code using GBK, thus causes hard code string messy (package 目录;). But the input/output are normal:

When using -Dfile.encoding=UTF8, hard code string is normal, but runtime encoding is also set to utf8 and the CP is 936 so it can't input or output.

Ideally chcp 65001 would solve this, because both source and runtime are utf8. But due to bugs in Windows console, programs can't read things in 65001.

So my conclusion is that this extension should compile source code with -encoding utf8 and do not set -Dfile.encoding to UTF8 and do not use chcp 65001.

[Eskibear]

I met issue with your GBK example:

For utf8 case:

But there is no way to work using java test.java directly.

I don't see why you can run java <path-to-java> directly before compile it to a class file.

So my conclusion is that this extension should compile source code with -encoding utf8 and do not set -Dfile.encoding to UTF8 and do not use chcp 65001.

One thing to mention, Java files are compiled by JDT language server, which I know little about the encoding it's using. /cc Jinbo Wang (@testforstephen), please share more about JDT, and see if above proposal makes any sense?

[imba-tjd]

Please retry your first two test in a new CMD console and in chcp 936. It's yet another issue related to powershell. As for java test.java, it's because I'm using JDK16 PS能分别设定输入编码和输出编码，在PS里运行chcp只会改变输入编码，在CMD里运行就会都改。PS又能完全继承CMD的编码。你前两个测试不能正常显示，必定是你的PS一开始在chcp 65001的环境里（比如控制面板的那个设置，或者用了F5执行了本扩展的那个launcher.bat），你之后手动执行了chcp 936，只改变了输入编码。 所以在PS里不能用chcp，可以用这条命令代替： [Console]::InputEncoding = [Console]::OutputEncoding = [Text.Encoding]::GetEncoding("GBK")