Opened 6 years ago

Closed 2 years ago

#10 closed defect (fixed)

filenames are not translated to/from utf-8

Reported by: Arne Döring Owned by: Thomas Bernard
Priority: major Milestone: 2.5
Component: GrafX2 Version: 2.4
Keywords: encoding Cc: miniupnp

Description

when I loaded a file, the file browser showed a folder that had an 'ä', but the 'ä' was replace by two other random characters. This is a clear indication that an utf8 string is read by a single byte encoding. When I save a file with an ä in it's filename I get a filename where my operating system tells me that it is an invalid filename.

Attachments (7)

issue_10.png (24.9 KB) - added by miniupnp 4 years ago.
show the problem on a linux machine
Work_in_progress_issue10.patch (4.0 KB) - added by miniupnp 4 years ago.
issue10_a.patch (6.9 KB) - added by miniupnp 4 years ago.
fully working patch
issue10_with_ifdef.patch (8.4 KB) - added by miniupnp 4 years ago.
This patch is using #ifdef to enable iconv only when applicable (linux/FreeBSD/etc.)
cyrillic_chars_screenshot.png (2.1 KB) - added by Thomas Bernard 2 years ago.
Work in progress
cyrillic_chars_screenshot2.png (4.0 KB) - added by Thomas Bernard 2 years ago.
almost done !
win32_brush_factory.png (10.9 KB) - added by Thomas Bernard 2 years ago.
Factory displaying Windows Short names

Download all attachments as: .zip

Change History (21)

comment:1 Changed 5 years ago by pulkomandy

Summary: wrong file encoding on save load in linuxfilenames are not translated to/from utf-8

Changed 4 years ago by miniupnp

Attachment: issue_10.png added

show the problem on a linux machine

Changed 4 years ago by miniupnp

comment:2 Changed 4 years ago by miniupnp

OK, I've started to fix that.
with "Work_in_progress_issue10.patch​" file and directories name are displayed correctly in the "Load" dialog.

Changed 4 years ago by miniupnp

Attachment: issue10_a.patch added

fully working patch

comment:3 Changed 4 years ago by miniupnp

I just need to add a #define in order for the code to be enabled only for OS with UTF8 file names ;)

comment:4 Changed 4 years ago by pulkomandy

Hi,
Thanks for working on this!

Some comments:

  • We definitely need a #ifdef for this, because some platforms may not have libiconv. Something similar to what we do for Lua could be used.
  • This is only a partial fix: it only works if the file name can be converted to CP1252. This issue was originally opened because of lack of support for cyrillic chars in file names, and that would still be a problem.

comment:5 Changed 4 years ago by yrizoud

About displaying characters that are not in the Windows-1252 code page, I would really like to use a similar system as OpenXcom? : A single font image contains individual characters, no matter which encoding / codepage they are originally. A configuration file defines which characters are present, in which order (It can be a simple UTF-8 string of these characters, easy to edit and expand)
With 384 characters, they cover a large number of languages, without useless symbols.
https://github.com/SupSuper/OpenXcom/blob/master/bin/common/Language/FontBig.png

But I think this should wait for the SDL2 port, because in this one I increased the base resolution. Characters would be 16x16 pixels, this leaves much more room for even the latin accented characters.

comment:6 Changed 4 years ago by pulkomandy

We already have the latin accented chars (well, most of them). Fitting cyrillic should not be too hard, if someone can draw the glyphs.

Maybe we could try switching the whole GUI to SDL_ttf and some ttf fonts. Proportional fonts in some places could be interesting, and it would solve that problem.

Changed 4 years ago by miniupnp

Attachment: issue10_with_ifdef.patch added

This patch is using #ifdef to enable iconv only when applicable (linux/FreeBSD/etc.)

comment:7 Changed 4 years ago by miniupnp

Cc: miniupnp added

comment:8 Changed 4 years ago by pulkomandy

Thanks, applied in r2172. I'm leaving the issue open for people using cyrillic chars, still.

comment:9 Changed 3 years ago by Thomas Bernard

Owner: changed from Adrien Destugues to Thomas Bernard
Status: newaccepted

About cyrillic chars, can someone make us a cyrillic font in the same format as font_Classic.png etc. ?

8x8 pixel chars, with 32 chars per line, with the 256 characters from the Unicode 0400-04FF range. The resulting PNG should be 256x64
http://www.unicode.org/charts/PDF/U0400.pdf

Changed 2 years ago by Thomas Bernard

Work in progress

Changed 2 years ago by Thomas Bernard

almost done !

comment:11 Changed 2 years ago by Thomas Bernard

https://gitlab.com/GrafX2/grafX2/merge_requests/31

still some work to make it work for windows (with small filenames/long filenames)
but I'm close

comment:12 Changed 2 years ago by Thomas Bernard

I think everything is OK with the merge request now.
Only thing missing are fonts for missing characters !

comment:13 Changed 2 years ago by Thomas Bernard

The code has been merged, so we could close the ticket now,
but let take time to detect all border effects...

I just noticed the Factory displays the short names under Windows :(

Changed 2 years ago by Thomas Bernard

Attachment: win32_brush_factory.png added

Factory displaying Windows Short names

comment:14 Changed 2 years ago by Thomas Bernard

Resolution: fixed
Status: acceptedclosed

everything is fixed now. Just need more 8x8 fonts for non-latin characters ;)

Note: See TracTickets for help on using tickets.