| 05:14.37 | *** join/#asterisk-bugs RiceCracker (~RiceCrack@59.152.236.158) |
| 05:58.45 | *** join/#asterisk-bugs oej (~olle@2001:16d8:cc57:1000::42:1003) |
| 07:08.47 | *** join/#asterisk-bugs infobot (~infobot@rikers.org) |
| 07:08.47 | *** topic/#asterisk-bugs is Asterisk Bugs Discussion -=- https://issues.asterisk.org/jira -=- http://xkcd.org/583/ |
| 08:43.46 | *** join/#asterisk-bugs tzafrir_laptop (~tzafrir@local.xorcom.com) |
| 08:58.18 | *** join/#asterisk-bugs irroot (~gregory@41.79.221.226) |
| 08:59.23 | *** part/#asterisk-bugs irroot (~gregory@41.79.221.226) |
| 12:16.05 | *** join/#asterisk-bugs mjordan (~mjordan@nat/digium/x-xixjghemosqmskjb) |
| 12:16.05 | *** mode/#asterisk-bugs [+o mjordan] by ChanServ |
| 13:00.16 | *** join/#asterisk-bugs serafie (~erin@nat/digium/x-gbnzebxqnevsgajp) |
| 13:02.37 | *** join/#asterisk-bugs putnopvut (~putnopvut@asterisk/master-of-queues/mmichelson) |
| 13:02.38 | *** mode/#asterisk-bugs [+o putnopvut] by ChanServ |
| 13:08.17 | *** join/#asterisk-bugs leifmadsen (~Leif@asterisk/documenteur-extraordinaire/blitzrage) |
| 13:08.17 | *** mode/#asterisk-bugs [+o leifmadsen] by ChanServ |
| 13:26.42 | *** join/#asterisk-bugs oej_ (~olle@2001:16d8:cc57:1000::42:1003) |
| 13:27.36 | *** join/#asterisk-bugs oej_ (~olle@2001:16d8:cc57:1000::42:1005) |
| 13:30.27 | *** join/#asterisk-bugs oej_ (~olle@2001:16d8:cc57:1000::42:1005) |
| 13:47.49 | *** join/#asterisk-bugs jsmith (~jsmith@fedora/jsmith) |
| 13:47.50 | *** mode/#asterisk-bugs [+o jsmith] by ChanServ |
| 13:48.08 | jsmith | putnopvut: Hey, feel like taking another stab at that connected line segfault, now that I have better backtraces? |
| 13:50.54 | *** join/#asterisk-bugs oej (~olle@2001:16d8:cc57:1000::42:1005) |
| 13:54.14 | putnopvut | jsmith: Sure. Remind me of the issue number? |
| 13:54.54 | jsmith | 20226 |
| 13:55.13 | jsmith | I'll be attaching more backtraces to the bug, if you think they'll be useful |
| 13:55.13 | putnopvut | has another look |
| 13:55.50 | jsmith | should also have some better backtraces from some deadlocks today as well |
| 13:57.13 | putnopvut | jsmith: happen to have a core dump of the crash from 20226 handy? |
| 13:57.22 | putnopvut | the latest crash, that is? |
| 13:58.32 | jsmith | putnopvut: Possibly... it may be another hour or two before I get it though, since I don't have direct access to the box in question, and the folks in Arizona aren't awake yet |
| 13:58.50 | putnopvut | Okay. |
| 13:59.38 | jsmith | If not, I'm sure we'll have another core file and backtrace to work with soon :-( |
| 13:59.53 | jsmith | (We apparently had three segfaults yesterday -- while I was on an airplane) |
| 14:00.01 | putnopvut | So the better backtrace shows the crash occurring in the timer code. It's on a line where there are a bunch of pointers, and I'm trying to see which is the bad one. |
| 14:00.03 | jsmith | For some reason, the problems always happen while I'm in the air |
| 14:00.13 | jsmith | Interesting.... |
| 14:00.30 | leifmadsen | which timing module? |
| 14:00.36 | leifmadsen | out of curiousity |
| 14:01.13 | putnopvut | res_timing_pthread, it appears |
| 14:01.18 | jsmith | So, I'll admit my Asterisk skills are a bit rusty in that area -- I didn't keep up with all the timing changes |
| 14:01.18 | leifmadsen | eep |
| 14:01.26 | jsmith | Is that the prefered timing module? |
| 14:01.33 | jsmith | Should I be using something else? |
| 14:01.40 | leifmadsen | jsmith: order of timing module preference: res_timing_dahdi, res_timing_timerfd, res_timing_pthread |
| 14:01.47 | jsmith | Can someone give me the 30-second tour? |
| 14:01.51 | leifmadsen | jsmith: install dahdi, and use res_timing_dahdi if you can |
| 14:01.54 | leifmadsen | jsmith: see above :) |
| 14:01.58 | jsmith | Right -- and we have DAHDI cards in the box |
| 14:02.03 | putnopvut | I could be wrong about what I found. It may just be that res_timing_pthread was loaded and running but not actually used. |
| 14:02.05 | leifmadsen | oh ya, totally be using res_timing_dahdi then :) |
| 14:02.20 | jsmith | Next stupid question -- how to I tell it which timing module to use? |
| 14:02.25 | leifmadsen | putnopvut: ya, but I always recommend res_timing_dahdi be used if at all possible to eliminate pthread timing issues |
| 14:02.31 | jsmith | Force the module in modules.conf and not load the others? |
| 14:02.41 | jsmith | feels stupid for having to ask |
| 14:02.47 | leifmadsen | jsmith: dahdi should really be the preferred if the module exists |
| 14:02.57 | leifmadsen | ls /usr/lib/modules/asterisk/res_timing* |
| 14:03.03 | jsmith | leifmadsen: Ah, gotcha |
| 14:03.24 | jsmith | But that means I have to have a DAHDI card in the machine that I compile Asterisk on, correct? |
| 14:03.28 | leifmadsen | maybe a ./configure ; make menuselect needs to be run to make sure you have res_timing_dahdi enabled -- I'd just disable the other 2 |
| 14:03.29 | putnopvut | I think all three timing modules were loaded. Asterisk has logic inside to choose which to actually use. |
| 14:03.38 | leifmadsen | jsmith: not a card, just dahdi itself |
| 14:03.41 | jsmith | OK |
| 14:03.52 | jsmith | will try to remember to install DAHDI on his builders |
| 14:03.55 | leifmadsen | dahdi_dummy essentially (which is built in, not a separate module now) |
| 14:03.59 | jsmith | (if not already) |
| 14:04.10 | leifmadsen | but ya, putnopvut is right about the logic to pick order of load |
| 14:04.20 | leifmadsen | res_timing_dahdi should be first, then timerfd, then pthread |
| 14:04.31 | leifmadsen | timerfd requires kernels > .... 2.6.14 or something? |
| 14:04.54 | leifmadsen | all the timing modules are listed in the Resources menu though |
| 14:05.06 | leifmadsen | so you can double check that, and check which modules were compiled first time around |
| 14:05.25 | putnopvut | leifmadsen: I think it's newer than that. 2.6.30 or something. |
| 14:05.28 | jsmith | Is there something from the CLI I can run to check which timing module is being used? |
| 14:05.29 | leifmadsen | jsmith: but ya, I ran into those types of issues, and most of them a resolved after I install dahdi and res_timing_dahdi on the customers machine |
| 14:05.41 | putnopvut | jsmith: yeah there's a timing test you can run. I think it mentions which module is used. |
| 14:05.41 | leifmadsen | putnopvut: ah ya, I can never remember |
| 14:05.48 | leifmadsen | aye |
| 14:05.50 | putnopvut | let me double-check |
| 14:05.52 | leifmadsen | timing test or osmething |
| 14:05.53 | jsmith | Oooh, a test sounds nice :-) |
| 14:05.57 | leifmadsen | tim<tab> :) |
| 14:06.06 | jsmith | wishes he had direct access to this box |
| 14:06.08 | leifmadsen | omg, now I want a tim-tam |
| 14:06.15 | leifmadsen | eff |
| 14:06.17 | putnopvut | Yeah, it's "timing test" |
| 14:06.24 | jsmith | shakes his fist at sys-admins that can't get the VPN working properly |
| 14:06.27 | leifmadsen | http://en.wikipedia.org/wiki/Tim_Tam |
| 14:06.30 | putnopvut | And it mentions which timer is used. |
| 14:06.39 | leifmadsen | jsmith: I hate sysadmins that require a VPN in order to SSH |
| 14:06.50 | jsmith | leifmadsen: Yummy... |
| 14:06.52 | leifmadsen | jsmith: aye |
| 14:07.03 | jsmith | leifmadsen: Long story, but in this case, it really is necessary |
| 14:07.19 | leifmadsen | gotcha |
| 14:19.25 | putnopvut | So in case I'm not around when you actually do get access to the box. If you open your core file in gdb, "frame 1", "print handle", "print handle->holder", "print handle->holder->iface" |
| 14:19.45 | putnopvut | I figure there's a chance one of those is NULL. |
| 14:23.05 | *** join/#asterisk-bugs Devon_ (~chatzilla@63.214.236.169) |
| 15:05.27 | jsmith | putnopvut: OK, I still have the core file, and I'm running those commands now |
| 15:06.42 | putnopvut | cool cool |
| 15:07.31 | jsmith | http://pastebin.com/MePNUSp9 |
| 15:07.41 | jsmith | Anything else you want me to run? |
| 15:08.12 | putnopvut | jsmith: have you confirmed via the "timing test" CLI command which timing module is being used? |
| 15:08.28 | putnopvut | bah, none of those pointers are NULL. |
| 15:08.29 | jsmith | No -- is that safe to run on a production system with approximately 90 concurrent channels? |
| 15:09.00 | putnopvut | jsmith: yeah, it should be perfectly fine. It does, I think 50 ms of timing. |
| 15:09.36 | putnopvut | oh 1 second of timing in 50 ms intervals. |
| 15:09.51 | putnopvut | Still shouldn't disrupt calls. |
| 15:10.35 | jsmith | Using the 'DAHDI' timing module for this test. |
| 15:10.36 | jsmith | It has been 1015 milliseconds, and we got 51 timer ticks |
| 15:10.54 | putnopvut | Okay, all's well. SO I guess res_timing_pthread was loaded but not actually being used. |
| 15:11.04 | jsmith | Appears that way |
| 15:11.31 | putnopvut | jsmith: out of curiosity, could you print handle->fd from the backtrace? |
| 15:11.46 | jsmith | (gdb) print handle->fd |
| 15:11.46 | jsmith | $3 = -603788128 |
| 15:11.50 | putnopvut | whoa |
| 15:12.02 | jsmith | Yeah... |
| 15:12.02 | putnopvut | that's...weird |
| 15:12.24 | jsmith | To say the least |
| 15:12.27 | putnopvut | How about print *handle |
| 15:12.45 | jsmith | $4 = {fd = -603788128, holder = 0x2aaadc06efb0} |
| 15:13.19 | putnopvut | okay, yeah that fd is messed up. Let's see where the timer is in a channel structure... |
| 15:15.52 | putnopvut | Yeah, not sure what might be doing that. |
| 15:40.33 | *** join/#asterisk-bugs tzafrir_laptop (~tzafrir@local.xorcom.com) |
| 15:54.04 | jsmith | putnopvut: Anything else you want me to try? |
| 15:54.53 | putnopvut | jsmith: well the best thing would be valgrind, but that's not a possibility when you have that many concurrent users. |
| 15:55.01 | jsmith | putnopvut: Correct |
| 15:55.33 | putnopvut | It's interesting that the crash always occurs during a SIP transfer when queuing a connected line update, though. |
| 15:55.46 | jsmith | Well, that's the most common case |
| 15:55.53 | jsmith | We are seeing other crashes in other places as well |
| 15:55.57 | putnopvut | For a memory corruption-type issue, it's happening in a consistent place. I wish my earlier valgrind runs had uncovered something... |
| 15:56.19 | jsmith | Would you like me to past the latest backtraces from the other crashes as well? |
| 15:56.39 | jsmith | (for example, we had three crashes yesterday, but none of them look like connected line updates at first glance) |
| 15:56.59 | putnopvut | jsmith: sure, that might be helpful. |
| 16:00.07 | jsmith | putnopvut: Pastebin? New Bug? Attach them to 20027? |
| 16:00.31 | putnopvut | jsmith: start with a pastebin. |
| 16:01.21 | jsmith | putnopvut: Coming right up |
| 19:32.30 | *** join/#asterisk-bugs tzafrir_laptop (~tzafrir@212.179.75.202) |
| 22:00.07 | *** part/#asterisk-bugs mjordan (~mjordan@nat/digium/x-xixjghemosqmskjb) |