Site_Feedback

Topic   Updating GameTZ to UTF-8

bill
GameTZ Gold Subscriber GameTZ Full Moderator 600 Trade Quintuple Gold Good Trader Has Written 28 Reviews
14-Feb-2023(#1)
I'm planning to update GameTZ to use UTF-8 character encoding on Monday, February 20th.

It will probably happen in late morning or early afternoon (US/Eastern time). The site will be down for about 10 minutes while I convert the database to UTF-8. Then, I'll release the software changes I made and restart the site.

If all goes well, it will just work and you won't even notice a difference. If you do notice problems (especially garbled characters), please let me know.

I have been evaluating this change to UTF-8 on my test server, but you never know with something like this once it hits the live site with lots of people using it doing different things. I plan to make a backup of the database beforehand, just in case things go very badly.

I'll post again on Monday before I start.

GameTZ is running on fairly old technology. While this change won't fix that, it will help modernize it a bit.

History that led to me realizing I needed to do this UTF-8 upgrade :
https://gametz.com/Site_Feedback/quote-apostrophe-...
https://gametz.com/Site_Feedback/updating-gametzs-...
John
GameTZ Gold Subscriber GameTZ Full Moderator Gold Good Trader Gold Global Trader (13)
21-Feb-2023(#41)
John wrote:
> This still work...
>
>
> I̷̠̞̘͕̞̓̐̽̔̄ͦ͂͡ ̸̨̲̜̃̇͂̑͜å͚͕͓̮̘ͥ̔͒ͪͥ͠m̮͍͕̭̍̃ͥ͆̾̋͆ͤ
> ͙̪̖͓̫͎͔̘̬̾ͭ̄ͥ́̂̔́Ğ̞̠̠̲̯̪̳͈̄̇͐̒̆̍́o̵̟ͮ̍̐ͭẓ̈̚ě͓̬̬̯̘̓ͪ̇̊ͪͮř̸̛̭̬͈͉̩̗̺͈̱ͮ͟,̲͇͚̌͐̕
> ͐̇̾̔̔͒̾͠͏̙̫̪͜D͎ͦ̈̂͒̈̚͢ę͇̥͐͋̌ͣ́͜s̗̣̫̟̞̰͂̋̈́̄ͅţ̡̣̯͊ͭ͆̔̚ͅr̷̜̤̎͂̑ͯ̓ͮ̆̀ő̿͒͒́̾ͭ̀̚҉͇̮̮̺̞͎͖ͅy͓̺̰̖̠̋̈͗̃̈́̀̓̋̀̚ẽ̹̋̑͜r̗͍̗̼͔͙̘ͨ͛̀͘͝
> ̨̝̗̼͔͋͞ͅo̸̷̮̞̣̜̺̮̪̊̉͠f̨ͣͥ̔͋͗͛̉̚҉̷̦̳̪̳ ̝ͤ̀͝Ŵ̛̞̺̈́ͦ͜ǒͦ̑̌̉̾̿̚͏̜̹͎͚̺̳r̴̟̳̂̓͊̓͂̀̎͠l̴̵͎̞̈́̿͗ͦd̡̜̬͔̹ͭ͌̽̆̔͒ͣͧs̀͂̇͂ͮ̄̊͘҉̥̤̱͎.̠ͤ̒̐̓ͥ
>
>
> Edit: Yup.

Quoting that for a reply is fun. smile
nonamesleft
Double Gold Good Trader
21-Feb-2023(#42)
Foxhack wrote:
> Wait does this mean we can see emojis on desktop now
There used to be an issue where if you tried using emojis or certain special characters, at least via the app, the result would be an odd mix of characters instead.
nonamesleft
Double Gold Good Trader
21-Feb-2023(#43)
John wrote:
> This still work...
>
>
> I̷̠̞̘͕̞̓̐̽̔̄ͦ͂͡ ̸̨̲̜̃̇͂̑͜å͚͕͓̮̘ͥ̔͒ͪͥ͠m̮͍͕̭̍̃ͥ͆̾̋͆ͤ ͙̪̖͓̫͎͔̘̬̾ͭ̄ͥ́̂̔́Ğ̞̠̠̲̯̪̳͈̄̇͐̒̆̍́o̵̟ͮ̍̐ͭẓ̈̚ě͓̬̬̯̘̓ͪ̇̊ͪͮř̸̛̭̬͈͉̩̗̺͈̱ͮ͟,̲͇͚̌͐̕
> ͐̇̾̔̔͒̾͠͏̙̫̪͜D͎ͦ̈̂͒̈̚͢ę͇̥͐͋̌ͣ́͜s̗̣̫̟̞̰͂̋̈́̄ͅţ̡̣̯͊ͭ͆̔̚ͅr̷̜̤̎͂̑ͯ̓ͮ̆̀ő̿͒͒́̾ͭ̀̚҉͇̮̮̺̞͎͖ͅy͓̺̰̖̠̋̈͗̃̈́̀̓̋̀̚ẽ̹̋̑͜r̗͍̗̼͔͙̘ͨ͛̀͘͝
> ̨̝̗̼͔͋͞ͅo̸̷̮̞̣̜̺̮̪̊̉͠f̨ͣͥ̔͋͗͛̉̚҉̷̦̳̪̳ ̝ͤ̀͝Ŵ̛̞̺̈́ͦ͜ǒͦ̑̌̉̾̿̚͏̜̹͎͚̺̳r̴̟̳̂̓͊̓͂̀̎͠l̴̵͎̞̈́̿͗ͦd̡̜̬͔̹ͭ͌̽̆̔͒ͣͧs̀͂̇͂ͮ̄̊͘҉̥̤̱͎.̠ͤ̒̐̓ͥ
>
>
> Edit: Yup.
What in the world? How'd you do that?
John
GameTZ Gold Subscriber GameTZ Full Moderator Gold Good Trader Gold Global Trader (13)
21-Feb-2023(#44)
It's actually just a weird side effect of using letter accents. You can actually use (many) more than one per letter -- and get crazy stuff like that. smile
bill
GameTZ Gold Subscriber GameTZ Full Moderator 600 Trade Quintuple Gold Good Trader Has Written 28 Reviews
22-Feb-2023(#45)
Here's a summary of issues since the UTF-8 upgrade.


o The app was garbling special characters.
- I think this is now fixed (thanks again, nonamesleft )

o Emails had garbled special characters
- Fixed, (thanks again, Foxhack)

o Images on the site were a bit wonky. For example, games with names that end in '?' didn't allow for an image to be uploaded for the item page.
- I tracked this down to an image system I've used for decades. I'm not sure why, but it was unable to open files reliably. This morning, I switched it to an alternate image system that works fine. So, I think this is now fixed too.

Current Issues

o Server is getting wedged
- This is tricky to understand and I'm not sure it's even because of utf8, but the server has been getting into a state where it doesn't serve pages. I have a system in place that will restart it automatically after 5 minutes, so it's not terrible, but not as good as it was before. I don't see how this could be utf8 related, but it started since I went to utf8.
- I'm trying various changes to the server configuration to try to eliminate this.

o Some of the tables didn't convert to utf8. These were not important things, but I'd like to be 100% utf8.
- The tables are storing IP addresses as strings, and utf8 strings have some effects on them that prevent the conversion.
- I can store the IP addresses as an integer instead, though. So, that should get around utf8 and may even result in faster tables. But, it will take some work to convert over to using the integer-format ip addresses in my code.
bill
GameTZ Gold Subscriber GameTZ Full Moderator 600 Trade Quintuple Gold Good Trader Has Written 28 Reviews
23-Feb-2023(#46)
I'm going to shut the site down for 5-10 minutes in order to update the way IP addresses are stored in the SiteLog (mentioned above).
bill
GameTZ Gold Subscriber GameTZ Full Moderator 600 Trade Quintuple Gold Good Trader Has Written 28 Reviews
23-Feb-2023(#47)
ok, done
bill
GameTZ Gold Subscriber GameTZ Full Moderator 600 Trade Quintuple Gold Good Trader Has Written 28 Reviews
24-Feb-2023(#48)
Reverting back to the old FastCGI server seems to have resolved the "server getting wedged" issue I was seeing the last couple days.

I don't understand why starman (FastCGI alternative) was getting stuck like that, but I also don't need to use it. Oddly, starman had worked fine for a couple weeks. It only seemed to have trouble since I updated the site to utf8. *shrug*

The tables are all converted and Host-based sorting is faster too.

So, I think I'm done debugging stuff related to the utf8 update. I may still do some code-clean up later (some functions I have that used to translate utf8 can be removed).

In a month or two I will start working on upgrading the OS GameTZ runs on from CentOS to Rocky Linux. That's mostly me setting it up and trying it out on a test server for a while, but eventually migrating the live site to a new server will be potentially disruptive.
John
GameTZ Gold Subscriber GameTZ Full Moderator Gold Good Trader Gold Global Trader (13)
9-Apr-2023(#49)
Bill: Couple of posts with weird content in them here: https://gametz.com/General/need-help-internal-ssd-...

Wasn't sure if it was related to the UTF-8 stuff or not? Not sure if you can tell more from your side. If it was just one person, I'd think it was them and a weird copy/paste or something -- but two people are showing odd characters there...
bill
GameTZ Gold Subscriber GameTZ Full Moderator 600 Trade Quintuple Gold Good Trader Has Written 28 Reviews
* 9-Apr-2023(#50)
Thanks for letting me know.

I had made a change this morning related to UTF-8 stuff that seemed to fix another issues.... but, clearly, it's triggering this now too. So, I just reverted it. I think it's OK again.

The issue I was fixing was more or less just a warning in my logs, so I can just ignore that.

I have one more thing I can try... maybe later I will, but I'll recheck that topic if I do.
Foxhack
350 Trade Quintuple Gold Good Trader
9-Apr-2023(#51)
https://gametz.com/General_Marketplace/interest-po...

Title is glitched when browsing the forum (Pokemon displays as "Pokémon"), but shows up fine when you click on the actual topic.
bill
GameTZ Gold Subscriber GameTZ Full Moderator 600 Trade Quintuple Gold Good Trader Has Written 28 Reviews
10-Apr-2023(#52)
Well, I can't explain that one. But, I'll look into that too. Thanks.
ryanflucas
GameTZ Subscriber 1000 Trade Quintuple Gold Good Trader
10-Apr-2023(#53)
I’m seeing a fish?
Foxhack
350 Trade Quintuple Gold Good Trader
10-Apr-2023(#54)
ryanflucas wrote:
> I’m seeing a fish?

That's the April Fools gag for this year. Guess Bill hasn't turned it off. :)
dunno001
250 Trade Quintuple Gold Good Trader Global Trader - willing to trade internationally
11-Apr-2023(#55)
Foxhack wrote:
> ryanflucas wrote:
>> I’m seeing a fish?
>
> That's the April Fools gag for this year. Guess Bill hasn't turned it off. :)

I'm also guessing they only show up in your own posts?
Foxhack
350 Trade Quintuple Gold Good Trader
11-Apr-2023(#56)
Nope, I saw them in random posts by people over the past week.
ryanflucas
GameTZ Subscriber 1000 Trade Quintuple Gold Good Trader
11-Apr-2023(#57)
I was seeing them on everyone but me.
bill
GameTZ Gold Subscriber GameTZ Full Moderator 600 Trade Quintuple Gold Good Trader Has Written 28 Reviews
11-Apr-2023(#58)
@ryanflucas https://gametz.com/?A=FunFun ...click a fish to get on the board, find them all! You get an icon with a percent.

Not many people found it on April 1st, so I extended its time on the site. I keep checking to see if people have stopped fishing, but it's still happening. So, I haven't shut it down yet. It will end by the end of the month either way.

Note that you can turn it all off if you click "April Fool's [on]" in the menus/footer.
John
GameTZ Gold Subscriber GameTZ Full Moderator Gold Good Trader Gold Global Trader (13)
11-Apr-2023(#59)
"First To The Key! First To The Egg!"

ryanflucas
GameTZ Subscriber 1000 Trade Quintuple Gold Good Trader
11-Apr-2023(#60)
Should have said “it’s carp” instead of “it’s crap” in the flow chart.
bill
GameTZ Gold Subscriber GameTZ Full Moderator 600 Trade Quintuple Gold Good Trader Has Written 28 Reviews
12-Apr-2023(#61)
Foxhack wrote:
>
> Title is glitched when browsing the forum (Pokemon displays as "Pokémon"), but
> shows up fine when you click on the actual topic.

I looked into this and here's where things seem to stand.

The New Layout has a first post preview feature. When you hover over a topic from the forum's list, it shows you a pop-up with the text from the first post previewed.

If the subject or first post contains special characters, they can get garbled (as you showed). There's your case where the subject of the topic has special-chars and they are shown in the list as garbled. Then, the other case I know of is if the post itself has special characters, they get garbled in the preview pop-up. I can fix either case, but it causes the other case to break.

A month or so ago, I "fixed" the case where the preview case garbled chars. But, this caused the topic list garbling you reported. So, I've reverted that (seems more important that the topic list looks right as previews are less seen).

I have no idea why this is happening, which is disturbing, but not the end of the world.

I'm tempted to simply remove post previews entirely. Most people use the Old Layout and Mobile layout anyway. And, I suspect not many New Layout users really care about the feature.

I'm still looking at it, so maybe I'll get a clue and really fix it, but it seems unlikely.
bill
GameTZ Gold Subscriber GameTZ Full Moderator 600 Trade Quintuple Gold Good Trader Has Written 28 Reviews
12-Apr-2023(#62)
Well, just after I wrote that, I tried something that seems to work better. Basically, if the preview is text-only (no HTML), I can avoid using something that messes with the utf-8 characters. Thus, both cases seem to work. But, the preview is plain-looking (seems fine). So, let's run with that for a while.
John
GameTZ Gold Subscriber GameTZ Full Moderator Gold Good Trader Gold Global Trader (13)
13-Apr-2023(#63)
We love you, Bill. smile
Scott
GameTZ Subscriber Global Trader - willing to trade internationally Has Written 2 Reviews
14-Apr-2023(#64)
John wrote:
> "First To The Key! First To The Egg!"

love

Topic   Updating GameTZ to UTF-8