provide a way to debug framework when it goes to a state where it does not respond to dbus

[lindi]

This is a wishlist bug about making framework easier to debug.

After some time framework sometimes goes to a state where it does not reply to any dbus calls. However, it is connected to dbus since it can send signals when an incoming call occurs. At present I don't know any good way to investigate the state of the framework process when it has hit this broken state. GDB is able to show instruction traces but I was not able to figure out what is going on there.

Maybe we could have framework listen on some UNIX socket and evaluate any python expressions that are written to this socket? Or maybe there are some GDB macros that could make python debuggable with gdb?

[daniel]

What does the frameworkd log say in this situation? I haven't encountered this state yet, how often does that happen for you?

[lindi]

gdb instruction trace of the problem

[lindi]

syslog output during problem

[lindi]

strace of ogsmd

[lindi]

strace of gsm0710muxd

[lindi]

I hit this maybe once a month. Please let me know what other information I can provide next time I hit this.

[lindi]

Just for the record, I hit this again :-) Again frameworkd did not answer to any dbus method call but was happily logging data from calypso to syslog.

[johnsu01]

This also happens to me. Using frameworkd from git today, up to b325493722f765e2c53392f1ef79dddb35d8d74d on my Debian system. I don't see any error output in the log (which was running on default verbosity). It was receiving calls and doing GPRS just fine earlier in the day, then someone told me that I had missed a call. When I investigated, frameworkd was in a state where it would not respond to dbus.

root@calvino:~# mdbus -s org.freesmartphone.frameworkd /org/freesmartphone/Framework org.freesmartphone.Framework.GetVersion?

ERROR:dbus.proxies:Introspect error on :1.62:/org/freesmartphone/Framework: dbus.exceptions.DBusException: org.freedesktop.DBus.Error.NoReply?: Did not receive a reply. Possible causes include: the remote application did not send a reply, the message bus security policy blocked the reply, the reply timeout expired, or the network connection was broken. /org/freesmartphone/Framework: GetVersion? failed: org.freedesktop.DBus.Error.NoReply?

[johnsu01]

Just to add more info -- I'm using 2.6.29-GTA02_andy-tracking-mokodev #58 PREEMPT Sat Feb 7 20:28:54 GMT 2009 armv4tl GNU/Linux and firmware moko11.

[mickey]

I have just committed a loophole implementation for the framework, which allows you to do live/post-mortem digging around:

root@om-gta02:~# telnet localhost 8822
Python 2.6.2 (r262:71600, Apr 25 2009, 15:05:31) 
[GCC 4.2.4] on linux2
Type "help", "copyright", "credits" or "license" for more information.
(NetworkInterpreterConsole)
>>> controller
<framework.controller.Controller object at 0x1a6230>
>>> 

[PaulFertser]

Small note:

One needs to add "-l" to frameworkd command line for this to be activated.