Converting UTF-8 to the 7-bit GSM Default Alphabet

As blogged about previously, the SME Toolkit has a feature which allows integration with telephone companies to send SMS messages via the SMPP protocol. Recently, we experienced the challenge of needing to communicate SMS messages to a telco utilizing the 7-bit GSM Default Alphabet. The challenge arose from the fact that, internally, the SME Toolkit […]

Character Encoding Tricks for Vim

Manipulating large amounts of data can often be a challenge, especially when the data utilizes complex character encodings or requires a change of character encoding. Fortunately, certain text editors such as Vim are well suited to handle this type of work. Vim supports many character encodings, and provides enhanced functionality to work with some of […]

Character Encoding Fun with SMS Messages

While trying to send SMS messages via the SME Toolkit application using the SMPP protocol, I encountered some bizarre behavior. I would send a seemingly ordinary message via SMPP only to receive a somewhat altered message on my mobile devices. Specifically, only certain symbols were altered, and latin letters appeared normally. What is more, the […]

On the Importance of Character Sets & Character Encodings in MySQL

When transmitting and storing digital data, one of the most important considerations should be the character encoding. Unfortunately, this rarely seems to be on anyone’s mind when setting up a database or making a database connection. For the most part, the defaults are just expected to work and provide the best set of options. With […]

Converting to UTF-16 and UCS-2 with Iconv

Recently the SME Toolkit, a project sponsored by the International Finance Corporation (a member of the World Bank Group), was attempting to send international SMS messages. This gave everyone on the team a good lesson in character encodings. We had previously utilized UTF-16 to send our SMS messages to the telephone company which we were […]