RetrieveMessagebook fails for binary alphabet messages containing null byte

[charlie]

on milestone 5, with a Taiwainese SIM. when I try the RetrieveMessagebook? DBus call, I get :

gsmsim.RetrieveMessagebook('read') 
> DBusException: org.freedesktop.DBus.Error.NoReply: Did not receive a reply.

The frameworkd log shows (I replaced the numbers by XXXXX) :

2009.02.05 13:49:58 dbus.service ERROR    Unable to append ([(21, 'read', '+XXXXXXXX', u'Hi', {'pid': 0, 'type': 'sms-deliver', 'timestamp': 'Tue Nov 18 18:18:30 2008 +0800', 'alphabet': 'gsm_default'}), (23, 'read', '+XXXXXXXX', u'I am \u77e5\u5df1\u77e5\u5f7c\u6b64\u4e4b\u9593', {'pid': 0, 'type': 'sms-deliver', 'timestamp': 'Tue Nov 18 18:27:33 2008 +0800', 'alphabet': 'ucs2'}), (24, 'read', '+XXXXXXXXX', u'\u4e4b\u5f8c\u4f86\u5c45\u4e0a\u9801', {'pid': 0, 'type': 'sms-deliver', 'timestamp': 'Tue Nov 18 18:44:06 2008 +0800', 'alphabet': 'ucs2'}), (25, 'read', '0935120188', u'12/10\u524d\u64a5800\u63094\u8f38\u5165\u4ee3\u78bc\u4efb\u4e00\u6d41\u884c\u7b54\u9234\u514d\u8cbb.313961\u746a\u9e97\u4e9e\u51f1\u8389I Stay In Love.313963\u4e39\u5c3c\u723eFree Loop!', {'pid': 0, 'type': 'sms-deliver', 'timestamp': 'Mon Nov 24 13:12:05 2008 +0800', 'alphabet': 'ucs2'}), (27, 'read', '+886939662463', u'Hi', {'pid': 0, 'type': 'sms-deliver', 'timestamp': 'Mon Nov 24 17:28:09 2008 +0800', 'alphabet': 'gsm_default'}), (29, 'read', 'XXXXXXXXXX', u'\u5225\u5fd8\u4e86\u60a8\u7684\u5c08\u5c6c\u6b0a\u76ca\uff1a\u514d\u8cbb\u4e0b\u8f09\u9234\u8072\u684c\u5e03\u599d\u9ede\u624b\u6a5f,\u9084\u53ef\u73a9\u904a\u6232,\u770b\u65b0\u805e,\u627e\u7f8e\u98df,\u5feb\u64a59670107,\u6536\u5230\u7c21\u8a0a\u5373\u53ef\u9023\u4e0a\u5c08\u5c6c\u7db2\u9801,\u82e5\u7121\u6cd5\u4e0a\u7db2\u8acb\u64a5899', {'pid': 0, 'type': 'sms-deliver', 'timestamp': 'Tue Nov 25 10:28:21 2008 +0800', 'alphabet': 'ucs2'}), (37, 'read', '+XXXXXXXXXX', u'Hi', {'pid': 0, 'type': 'sms-deliver', 'timestamp': 'Tue Nov 25 19:20:08 2008 +0800', 'alphabet': 'gsm_default'}), (38, 'read', 'XXXXXXXXXX', 'U\x06\x01\xae\x02\x05j\x00E\xc6\x0c\x03musicwap.catch.net.tw/cm-portalwap/newsong.jsp\x00\x01\x03\xe6\xad\x8c\xe5\xa3\x87\xe6\x96\xb0\xe4\xba\xba\xe8\xbc\xa9\xe5\x87\xba\xe9\x98\xbf\xe9\x91\x8c.\xe9\xbb\x83\xe9\x9d\x96\xe5\x80\xab.\xe6\xa2\x81\xe6\x96\x87\xe9\x9f\xb3\xe6\x96\xb0\xe6\xad\x8c\xe4\xb8\x8b\xe8\xbc\x89\x00\x01\x01', {'csm_id': 124, 'timestamp': 'Wed Nov 26 15:39:09 2008 +0800', 'pid': 0, 'csm_seq': 1, 'alphabet': 'binary', 'csm_num': 1, 'type': 'sms-deliver'})],) to message with signature a(isssa{sv}): <type 'exceptions.TypeError'>: expected string without null bytes
2009.02.05 13:49:58 ogsmd.channel ERROR    unhandled exception in response callback: expected string without null bytes
Traceback (most recent call last):

I guess the last message on the list is the one that causes problem. I will try to test further.

[jluebbe]

Could you try RetrieveMessage?(id) on each message in turn?

[charlie]

Ah I am having a problem, I deleted the binary message and now I can't create a new one (DBus won't allow me to send any string with a '\x00' inside, so I can use the StoreMessage?).

Of course now I can't reproduce the error.

I am going to try modify the dbus API to accept a variant instead of a string for the content argument of the StoreMessage? method. This way I should be able to pass it a vector of bytes, and then convert it into a string in the framework.

[charlie]

I added tests for the StoreMessage? call (​4ae7746). The second test is currently failing due to the null character in the binary data.

[charlie]

I tried to modify the framework to accept array of byte as argument to the StoreMessage? function, and then convert the array back to a str, but it stills fails for unknown reason. I attach the log.

Should the client encode the binary data by itself to remove the null bytes ?

[charlie]

even if the framework accept array of byte as argument to StoreMessage? it still fails to store it

[peppertarts]

I can reproduce this by requesting an auto configuration text from my provider's website. The message they send contains null characters.

​http://getsettings.o2.co.uk/ServeSettingsWebApp/

[jluebbe]

peppertarts, could you send a ogsmd debug log with that message? That message should not be declared as a text SMS by your provider.

[peppertarts]

Binary sms conversion to string?

[peppertarts]

The message is marked as 'alphabet':'binary' but it looks like there is still an attempt to convert it to a string when RetrieveMessagebook? is called. I've attached a log that shows the message and the trace.

[Sascha]

mms service sms from t-mobile germany

[daniel]

Ah great, some binary SMS!

The problem is that DBus Strings don't allow a null byte. I'll modify ogsmd so the binary contents will not show in the user data, but as a bytearray in the properties.

[daniel]

Please see commit [ceef20a8780e2abc96d4f68e49ffb11fd208a0e7] which should fix this issue (but not if UCS2 text includes a null byte).

Binary data is now a bytearray in the properties.

[daniel]

A quick note why the text itself should not be a bytearray.

This is quite ugly for the majority of the cases where one just want's to read/send a text message. It really sucks that DBus strings do not allow null bytes...

[daniel]

I've successfully sent and received binary SMS now so I will close this.