Site_Feedback

Topic   Apostrophe Bug on Mobile Site

whitefire
GameTZ Subscriber 550 Trade Quintuple Gold Good Trader Global Trader - willing to trade internationally
16-Dec-2017(#1)
If I search with an apostrophe, this happens:

image

image


bill
GameTZ Gold Subscriber GameTZ Full Moderator 600 Trade Quintuple Gold Good Trader Gold Global Trader (15) Has Written 26 Reviews
16-Dec-2017(#2)
I am able to reproduce this on my iPad.

I assume this is related to ios 11 using "smart punctuation" where special characters for apostrophe and a few other symbols are sent using a different character set that causes problems with the app and web users (often mobile).

One known fix for this is to disable "smart punctuation" under the settings app of ios 11 (General->Keyboard->Smart Punctuation).

I tried the fix I'm using for other situations for this search case, but it didn't fix this issue. I'm unsure what else to try. It looks like the framework I'm using for the mobile site (jQuery Mobile) is getting stuck on the special apostrophe and hangs. Googling, I didn't find anyone else complaining about this. So, I'm not sure what's going on. I'll try to look into it more...
bill
GameTZ Gold Subscriber GameTZ Full Moderator 600 Trade Quintuple Gold Good Trader Gold Global Trader (15) Has Written 26 Reviews
16-Dec-2017(#3)
Another workaround is to not type the apostrophe as search will still find stuff without it.
bill
GameTZ Gold Subscriber GameTZ Full Moderator 600 Trade Quintuple Gold Good Trader Gold Global Trader (15) Has Written 26 Reviews
16-Dec-2017(#4)
I think I fixed it. For now, at least.

The general problem GTZ is facing is that this is an old website and we used an older character encoding (ISO-8859-1) which has become unpopular. Most newer technology has standardized on UTF-8 character encoding. For the most part, it's not a problem because GTZ avoids using any special characters. So, the subset of characters actually being using is the same for both of these character sets and thus it just works when I mix software component that use UTF-8 with my code.

But, iso 11 sending special apostrophes that end up in user input sent to GTZ's server tickles this problem. GTZ encodes the apostrophes with ISO-8859-1, but the framework it uses for the mobile site (jQuery Mobile) uses UTF-8.

Now, a page should never freeze like you're seeing, so this does boil down to a bug in jQuery Mobile: https://bugs.jqueryui.com/ticket/9518 -- but it's rare enough (few other sites use ISO-8859-1), so there doesn't seem to be a fix (that bug is 4 years old, and says 'fixed' but clearly isn't -- if you open a JavaScript console you can see the error deep within jQuery Mobile and I'm using the latest stable version of that).


Potentially, I could/should convert GameTZ to UTF-8 and these problems would all be fixed. But, I suspect it's not that simple. It's kind of terrifying, to be honest. There are a number of layers to the site (e.g. code is written in Perl which has an encoding, web-server itself may do something with encoding, the database system surely uses an encoding and has 20 years of text in it, the app likely has its own way of doing all this too, etc.)

Fixing bugs like this 1-by-1 by forcing each case to work isn't good either. I basically replace the special characters in many places, but just the ones I find for that case and there are likely many more places that would need it, but that's a clear sign it's the wrong way to fix it.

Anyway, I may try to take this UTF-8 conversion on and see if it's feasible. We'll see.

You probably don't understand what I'm talking about. This is as much a note to myself as anything.
whitefire
GameTZ Subscriber 550 Trade Quintuple Gold Good Trader Global Trader - willing to trade internationally
16-Dec-2017(#5)
bill wrote:
> Another workaround is to not type the apostrophe as search will still find stuff
> without it.

Yeah, obviously, I did that. I just figured you'd still want to know because maybe some people will think they can't add the game or the site is broken.

John
GameTZ Gold Subscriber GameTZ Full Moderator 400 Trade Quintuple Gold Good Trader Gold Global Trader (13) Secret Santa
17-Dec-2017(#6)
To be fair to GameTZ, Apple's shift to "smart punctuation" is breaking lots of other sites and apps too. Annoying as heck...
bill
GameTZ Gold Subscriber GameTZ Full Moderator 600 Trade Quintuple Gold Good Trader Gold Global Trader (15) Has Written 26 Reviews
17-Dec-2017(#7)
I changed the character set encoding to UTF-8 and it doesn't seem to have caused any serious issue. It may just fix stuff like this... we'll see.
whitefire
GameTZ Subscriber 550 Trade Quintuple Gold Good Trader Global Trader - willing to trade internationally
19-Dec-2017(#8)
I turned off "smart" punctuation and haven't had any problems since.

bill
GameTZ Gold Subscriber GameTZ Full Moderator 600 Trade Quintuple Gold Good Trader Gold Global Trader (15) Has Written 26 Reviews
* 19-Dec-2017(#9)
That should definitely resolve the issue.

But, I also made a change a couple days ago that I think will fix it even if you have smart punctuation on. It's on by default, so maybe the bug can be prevented for others at least.

Thanks for letting me know about it.
John
GameTZ Gold Subscriber GameTZ Full Moderator 400 Trade Quintuple Gold Good Trader Gold Global Trader (13) Secret Santa
* 21-Dec-2017(#10)
It ended up breaking things on my Secret Santa site. I'll have to adjust for next year. I didn't have time to fix it for this year, so just edited the data manually. smile
bill
GameTZ Gold Subscriber GameTZ Full Moderator 600 Trade Quintuple Gold Good Trader Gold Global Trader (15) Has Written 26 Reviews
21-Dec-2017(#11)
How did it break?
John
GameTZ Gold Subscriber GameTZ Full Moderator 400 Trade Quintuple Gold Good Trader Gold Global Trader (13) Secret Santa
21-Dec-2017(#12)
It just barfed on the character. So, the user put in their address and wishlist -- and the DB just cut it off at the first funny character.

I don't sanitize the inputs much. I'm still using mostly the old/original code that I didn't write and have just been fixing things as needed. I'm fairly lazy with the SS. smile
BucketofJustice
GameTZ Subscriber 350 Trade Quintuple Gold Good Trader Has Written 6 Reviews Secret Santa
23-Dec-2017(#13)
Is that what happened to my offer response yet?

image

Mobile version on Chrome. Latest iOS. iPhone 7Plus.
ODB
GameTZ Subscriber 250 Trade Quintuple Gold Good Trader Global Trader - willing to trade internationally
23-Dec-2017(#14)
I know I'm a trouble maker and a handful sometimes but it really means a lot to me that when people bring stuff like this up, that bill will take time to see what's wrong, what the workaround or solution is and generally fix things like this. For what it's worth, it's appreciated.
bill
GameTZ Gold Subscriber GameTZ Full Moderator 600 Trade Quintuple Gold Good Trader Gold Global Trader (15) Has Written 26 Reviews
24-Dec-2017(#15)
BucketofJustice wrote:
> Is that what happened to my offer response yet?
>
> {imgt}https://i.imgur.com/3dJ4gvW.jpg 1242 2208{imgt}
>
> Mobile version on Chrome. Latest iOS. iPhone 7Plus.

Yes, this is likely a related issue. The garbled characters are what will often happens if one character set seeing stuff encoded with another that it doesn't understand. And, it seems to be the apostrophe in your case -- that is a known special character case (aka "smart punctuation").

I was able to reproduce this with my iPad. So, I will look into it more soon.

I suspect it may be related to the way a counter-offer has a 2-phase submit process (with confirmation). For me, the confirmation page showed what I wrote properply, but after it was over, I saw the garbled characters (e.g. for apostrophe).
bill
GameTZ Gold Subscriber GameTZ Full Moderator 600 Trade Quintuple Gold Good Trader Gold Global Trader (15) Has Written 26 Reviews
27-Dec-2017(#16)
OK, I have a fix in for that offer situation.
Sid_Ceaser
GameTZ Gold Subscriber 500 Trade Quintuple Gold Good Trader Has Written 6 Reviews
* 2-Jan-2018(#17)
I'm a sucker for using the ---- , but I've noticed that it's turned into little question marks on everything.


Edit: Oh, there it is. What's with the dashes before it? Weeeeird.



bill
GameTZ Gold Subscriber GameTZ Full Moderator 600 Trade Quintuple Gold Good Trader Gold Global Trader (15) Has Written 26 Reviews
2-Jan-2018(#18)
What was the character there and how did you make it?

The dashes and question-mark in a diamond thing are usually a sign that it's a different character set and can't figure it out. These character sets use multiple bytes, so it can end up a few character (E.g. dashes are shown), I think.
Sid_Ceaser
GameTZ Gold Subscriber 500 Trade Quintuple Gold Good Trader Has Written 6 Reviews
2-Jan-2018(#19)

It was a dot. I think its called a bullet. On a mac it's made by alt-8. It had been working for years, and just recently I noticed everything was turned to question marks: both on my studio computer and my iPad. And my android phone when I use that.




John
GameTZ Gold Subscriber GameTZ Full Moderator 400 Trade Quintuple Gold Good Trader Gold Global Trader (13) Secret Santa
2-Jan-2018(#20)
To note it, the other day I copy-and-pasted a pi symbol (just went to Google, searched for pi, and selected it) into the forum here -- and got the same sort of thing...

Here it is again: --
bill
GameTZ Gold Subscriber GameTZ Full Moderator 600 Trade Quintuple Gold Good Trader Gold Global Trader (15) Has Written 26 Reviews
2-Jan-2018(#21)
--
bill
GameTZ Gold Subscriber GameTZ Full Moderator 600 Trade Quintuple Gold Good Trader Gold Global Trader (15) Has Written 26 Reviews
2-Jan-2018(#22)
I see... yeah, it's not working like I assume it should. I'll try to look into it some more.
bill
GameTZ Gold Subscriber GameTZ Full Moderator 600 Trade Quintuple Gold Good Trader Gold Global Trader (15) Has Written 26 Reviews
* 3-Jan-2018(#23)
I suspect the issue is that the database is storing text (e.g. forum posts) using "latin1_swedish_ci" character set. This is not UTF-8, which is much more common for web-related text encoding these days. So, stuff like pi or that dot Sid uses don't work when I make the web site use UTF-8 because of some translation issue that's happen when the character is stored in the db. I had been using ISO-8859-1 for the page encoding and that did work. I suspect that ISO-8859-1 and latin1_swedish_ci are related.

So... it is possible to change the database to use UTF-8 for storing and that should make everything work as it would match the page encoding. I was looking into this last week, but it seemed rather hard/tricky to get right. Basically, all the existing stored text (e.g. decades of forum posts) would have to be converted. 99.9% of the characters are normal/boring (letters/numbers/punctuation - ascii that is common to all character sets). But, that odd 0.1% could be a problem. And, in conversion, I could lose the essence of what it was to begin with (e.g. end up with stored <?> chars that can't be fixed).

I could revert the site back to using ISO-8859-1, but the original issue that whitefire reported (e.g. using special chars to search) would come back. There are other issues too.

I guess I'll try to look into and test converting text tables to utf-8 in the database to see if I can get to a point where I'm fairly confident that will fix this and won't be too much of an issue (that 0.1%). It's possible, I still don't really understand what's happening and a utf-8 db table still won't work. So, I should try that just to see if it is the solution I think it is.
bill
GameTZ Gold Subscriber GameTZ Full Moderator 600 Trade Quintuple Gold Good Trader Gold Global Trader (15) Has Written 26 Reviews
3-Jan-2018(#24)
Well, I tried changing the db table to utf-8 (on my test server), but it didn't fix this pi thing. I'm really unsure what's going on now, but I'll keep looking into it. There are a number of layers to this, so maybe it's something else I'm not yet aware of...
whitefire
GameTZ Subscriber 550 Trade Quintuple Gold Good Trader Global Trader - willing to trade internationally
4-Jan-2018(#25)
@bill Not to throw another thing onto the pile, but I also noticed an issue on the mobile site where now the text box when you're writing messages after "I Have Sent" within trades drops below the keyboard when I'm trying to type. Here's an example:

image
image

It doesn't matter what I do to try to adjust it; it keeps doing this. It may be different in widescreen mode though; I didn't check.


bill
GameTZ Gold Subscriber GameTZ Full Moderator 600 Trade Quintuple Gold Good Trader Gold Global Trader (15) Has Written 26 Reviews
5-Jan-2018(#26)
@whitefire - I see that as an issue we've had all along with mobile devices in all layouts potentially. I've seen it happen a lot. Even in the app, it happens. Sucks, but I don't think I have any way to improve it. You can often scroll to adjust it somewhat, or rotating can help too.
bill
GameTZ Gold Subscriber GameTZ Full Moderator 600 Trade Quintuple Gold Good Trader Gold Global Trader (15) Has Written 26 Reviews
5-Jan-2018(#27)
In other news, I put the site back to ISO-8859-1 encoding given people reporting alt-codes not working (also copy/paste like π [which I assume will now work]).

I don't really understand what's going wrong, and I think this re-breaks whitefire's original search case. But, that also seems rare compared to the alt-code and pasting special characters.

It looks like this breaks any special chars that were posted since I change it to UTF-8, though. dead face;
bill
GameTZ Gold Subscriber GameTZ Full Moderator 600 Trade Quintuple Gold Good Trader Gold Global Trader (15) Has Written 26 Reviews
5-Jan-2018(#28)
Actually, testing the search with apostrophe case just now seemed to show it works fine. Maybe because I put in other code to force the smart punctuation to regular characters? Anyway, that makes it seem even more that we're best off sticking with ISO-8859-1
John
GameTZ Gold Subscriber GameTZ Full Moderator 400 Trade Quintuple Gold Good Trader Gold Global Trader (13) Secret Santa
* 5-Jan-2018(#29)
π

EDIT: Neat! smile

Topic   Apostrophe Bug on Mobile Site