{"id":311,"date":"2008-11-13T22:40:34","date_gmt":"2008-11-13T22:40:34","guid":{"rendered":"http:\/\/dalelane.co.uk\/blog\/?p=311"},"modified":"2009-05-17T18:51:09","modified_gmt":"2009-05-17T18:51:09","slug":"posting-to-twitter-carefully","status":"publish","type":"post","link":"https:\/\/dalelane.co.uk\/blog\/?p=311","title":{"rendered":"Posting to Twitter… carefully"},"content":{"rendered":"

I’ve recently picked up my the code for my Windows Mobile Twitter client<\/a> again. <\/p>\n

It was originally written back in April as a hackday idea<\/a>. The code posts Twitter updates using a variation on the twitter-from-curl<\/a> approach of HTTP POSTing “status=MyTweet” to the twitter update url<\/a>. <\/p>\n

I started with the update URL, and appended the message I wanted to tweet. This is fine for a quick hackday demo, but it did mean that you could end up with a URL like:<\/p>\n

http:\/\/twitter.com\/statuses\/update.xml?status=Hello (twitter) world! Special chars = a problem?<\/a><\/p>\n

Which fails if you want to post characters such as accents or characters which have special meaning in URLs, like + ? \/ & etc. <\/p>\n

I was encouraged by a number of users to have another look at this, which I’ve done now, and hopefully version 1.1<\/a> solves the problems. <\/p>\n

A quick Google turned up that a number of other Twitter apps share at least some of the same problems<\/a> that mine had, so thought I’d share the fix here. <\/p>\n

Step 1 – Url-encode<\/strong><\/p>\n

Okay, so technically I did this on the night on HackDay (it might have been a hack, but even I know that spaces in a URL aren’t the best idea!<\/em>) but I’m including it here for completeness. <\/p>\n

My app was written in C++, so I used InternetCanonicalizeUrl<\/code><\/a> to turn the tweet-posting API into something a little safer – a percent-encoded Uri. For example, this turned spaces in the message into %20<\/code> <\/p>\n

Step 2 – Specifying a content type<\/strong><\/p>\n

A few twitter users such as @reVoid<\/a> and @michaelmcmillan<\/a> reported problems when they tried to tweet \u00c3\u2020\u00c3\u02dc\u00c3\u2026 characters.<\/p>\n

After a bit of experimentation, the answer turned out to be to add charset=utf-8<\/code> to the HTTP headers I send when I post.<\/p>\n

TCHAR header[]  = TEXT(\"Content-Type:application\/x-www-form-urlencoded;charset=utf-8\");<\/pre>\n
A quick play with accented characters like \u00c3\u00a9 and \u00c3\u00a1 seemed to suggest that this would get accents working.<\/p>\n
Step 3 – UTF-8 encoding<\/strong><\/p>\n
This appeared to fix most accented characters. But then @walti<\/a> pointed out that my code still broke when posting certain characters with German umlautes. Either the next character after each umlaut would be lost when posted, or sometimes the whole remainder of the tweet after an umlaut would be lost. <\/p>\n
Googling showed that other apps such as Twitpic<\/a>, Ping.fm<\/a> and betwittered<\/a> shared this bug. <\/p>\n
I didn’t figure this one out for myself, but the answer was given to me by a helpful user on the Twitter API Google Group<\/a>. <\/p>\n
It seems that url encoding isn’t sufficient. You also need to UTF-8 encode the characters. For example, when posting \u00c3\u00b6 – it wasn’t enough for me to url-encode it and send it as %F6<\/code>, I also need to utf8-encode it to %C3%B6<\/code>. <\/p>\n
I wrote a quick-and-dirty encoder to handle this:<\/p>\n
\/\/---------------------------------------------------\r\n\/\/ doing a little UTF-8 encoding... \r\n\/\/---------------------------------------------------\r\n\r\nLPWSTR utf8encodedTweet = new TCHAR[dwNewSz * 3];\r\nnewptr = 0;\r\nfor (int i=0; i < dwNewSz; i++)\r\n{\r\n    if (lpszEncTweetMessage[i] == '%')\r\n    {\r\n        if (lpszEncTweetMessage[i+1] == 'B')\r\n        {\r\n            utf8encodedTweet[newptr++] = '%';\r\n            utf8encodedTweet[newptr++] = 'C';\r\n            utf8encodedTweet[newptr++] = '2';\r\n        }\r\n        else if (lpszEncTweetMessage[i+1] == 'C')\r\n        {\r\n            utf8encodedTweet[newptr++] = '%';\r\n            utf8encodedTweet[newptr++] = 'C';\r\n            utf8encodedTweet[newptr++] = '3';\r\n            utf8encodedTweet[newptr++] = '%';\r\n            utf8encodedTweet[newptr++] = '8';\r\n            i += 2;\r\n        }\r\n        else if (lpszEncTweetMessage[i+1] == 'D')\r\n        {\r\n            utf8encodedTweet[newptr++] = '%';\r\n            utf8encodedTweet[newptr++] = 'C';\r\n            utf8encodedTweet[newptr++] = '3';\r\n            utf8encodedTweet[newptr++] = '%';\r\n            utf8encodedTweet[newptr++] = '9';\r\n            i += 2;\r\n        }\r\n        else if (lpszEncTweetMessage[i+1] == 'E')\r\n        {\r\n            utf8encodedTweet[newptr++] = '%';\r\n            utf8encodedTweet[newptr++] = 'C';\r\n            utf8encodedTweet[newptr++] = '3';\r\n            utf8encodedTweet[newptr++] = '%';\r\n            utf8encodedTweet[newptr++] = 'A';\r\n            i += 2;\r\n        }\r\n        else if (lpszEncTweetMessage[i+1] == 'F')\r\n        {\r\n            utf8encodedTweet[newptr++] = '%';\r\n            utf8encodedTweet[newptr++] = 'C';\r\n            utf8encodedTweet[newptr++] = '3';\r\n            utf8encodedTweet[newptr++] = '%';\r\n            utf8encodedTweet[newptr++] = 'B';\r\n            i += 2;\r\n        }\r\n    }\r\n\r\n    utf8encodedTweet[newptr++] = lpszEncTweetMessage[i];\t\t\r\n}\r\nutf8encodedTweet[newptr] = '\\0';<\/pre>\n
Is that it?<\/strong><\/p>\n
I think that covers all the bases... <\/p>\n
No doubt someone will point out otherwise before too long. :-)  <\/p>\n","protected":false},"excerpt":{"rendered":"
I’ve recently picked up my the code for my Windows Mobile Twitter client again. It was originally written back in April as a hackday idea. The code posts Twitter updates using a variation on the twitter-from-curl approach of HTTP POSTing “status=MyTweet” to the twitter update url. I started with the update URL, and appended the […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[7],"tags":[295,151,243,298,296,297,299,19,43],"class_list":["post-311","post","type-post","status-publish","format-standard","hentry","category-code","tag-internetcanonicalizeurl","tag-twitter","tag-twittoday","tag-url-encode","tag-utf-8","tag-utf8","tag-widechartomultibyte","tag-windows-mobile","tag-windowsmobile"],"_links":{"self":[{"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=\/wp\/v2\/posts\/311","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=311"}],"version-history":[{"count":0,"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=\/wp\/v2\/posts\/311\/revisions"}],"wp:attachment":[{"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=311"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=311"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=311"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}