| 03:27.09 | *** join/#asterisk-bugs jmls (n=jmls@81.138.42.77) |
| 05:50.53 | *** join/#asterisk-bugs adker (n=chatzill@74-33-202-237.br1.glv.ny.frontiernet.net) |
| 06:06.30 | *** join/#asterisk-bugs mvanbaak_ (n=michiel@vanbaak.xs4all.nl) |
| 07:59.43 | *** join/#asterisk-bugs oej (n=olle@218.Red-80-34-119.staticIP.rima-tde.net) |
| 09:38.14 | *** join/#asterisk-bugs jmls (n=jmls@81.138.42.77) |
| 09:43.59 | *** join/#asterisk-bugs oej (n=olle@218.Red-80-34-119.staticIP.rima-tde.net) |
| 10:06.41 | *** join/#asterisk-bugs jmls (n=jmls@81.138.42.77) |
| 13:13.32 | *** join/#asterisk-bugs flujan (n=flujan@200-160-115-020.static.spo.ctbc.com.br) |
| 14:38.08 | flujan | M11862 |
| 14:38.09 | MuffinMan | [new] [Asterisk] Applications/app_queue 0011862: Possible deadlock on realtime queues. reported by flujan http://bugs.digium.com/view.php?id=11862 |
| 14:39.08 | flujan | Hi all, I checked this locks I am having with a realtime queue. I looked at the source code of the app queue and I am wondering if this function http://pastie.caboo.se/144816 is the problem. |
| 14:39.46 | flujan | IMHO, the process is entering one of this whiles and then stopping then don't releasing the queue. |
| 14:40.07 | flujan | question is, how can I debug it? |
| 14:40.29 | flujan | I have some C skills and I really want to now better what is happening here. :D |
| 14:48.17 | *** join/#asterisk-bugs lmadsen (n=Leif@asterisk/documenteur-extraordinaire/blitzrage) |
| 14:48.17 | *** mode/#asterisk-bugs [+o lmadsen] by ChanServ |
| 14:49.41 | *** join/#asterisk-bugs jcmoore (n=jcmoore@unaffiliated/tgrman) |
| 15:38.11 | *** join/#asterisk-bugs jmls_work (n=jmls@81.138.42.77) |
| 15:45.07 | *** join/#asterisk-bugs russellb (n=russell@asterisk/developer-and-stable-maintainer/drumkilla) |
| 15:45.07 | *** mode/#asterisk-bugs [+o russellb] by ChanServ |
| 16:05.40 | *** join/#asterisk-bugs putnopvut (n=putnopvu@216.207.245.1) |
| 16:05.40 | *** mode/#asterisk-bugs [+o putnopvut] by ChanServ |
| 16:17.30 | flujan | hi putnopvut |
| 16:17.51 | flujan | I reported: |
| 16:17.59 | flujan | M11862 |
| 16:18.00 | MuffinMan | [assigned] [Asterisk] Applications/app_queue 0011862: Possible deadlock on realtime queues. reported by flujan http://bugs.digium.com/view.php?id=11862 |
| 16:18.35 | flujan | I removed the realtime queue and put it on queues.conf files. About five minutes ago asterisk crashed with a segmentation fault... |
| 16:19.17 | flujan | I have bt output and will run it on valgrind if it stops again ( a production server ). Do you want to take a look on the bt output? |
| 16:21.40 | putnopvut | Yes, the bt output would be useful. |
| 16:21.56 | putnopvut | Sorry for the delayed response, I was buying a coffee. |
| 16:22.15 | Qwell | oh, speaking of which |
| 16:22.18 | Qwell | putnopvut: did you try the bagels? |
| 16:22.25 | russellb | o.O |
| 16:22.28 | russellb | what bagels? |
| 16:22.32 | Qwell | from Aroma's |
| 16:22.32 | putnopvut | Qwell: not yet. |
| 16:22.43 | russellb | and does anyone else find Alice's pronunciation of bagel odd? |
| 16:22.47 | Qwell | he was suggesting that he was considering possibly buying one at some point |
| 16:22.56 | Qwell | russellb: haven't heard her pronounce it.. |
| 16:23.06 | russellb | ah. |
| 16:23.08 | file | how does she say it? |
| 16:23.10 | Qwell | russellb: have YOU had an Aroma's bagel? |
| 16:23.25 | russellb | short A instead of long A |
| 16:23.30 | Qwell | baaaagel? |
| 16:23.38 | russellb | yes. |
| 16:23.41 | Qwell | nice |
| 16:24.41 | Qwell | aren't you Canadian? |
| 16:24.49 | file | maybe, maybe not |
| 16:24.58 | russellb | bay-gul ? or bah - gul ? |
| 16:25.04 | file | bay-gul |
| 16:25.13 | russellb | you win |
| 16:25.23 | file | what do I win? |
| 16:25.34 | Qwell | a bah-gul |
| 16:25.39 | russellb | heh |
| 16:25.48 | file | I don't like those, do you have a bay-gul instead? |
| 16:26.43 | flujan | putnopvut: no problem... here it goes http://pastie.caboo.se/144887 |
| 16:27.02 | flujan | if you need it on the bug report, please let me know. :) |
| 16:27.58 | flujan | I saw the app_queue.c on the bt, but I dunno if this issue is related to the bug I reported. |
| 16:30.43 | putnopvut | flujan: that bt appears to have happened during a Queue call, but it appears that the cause is either in the SIP channel driver or in something having to do with cdrs. |
| 16:31.38 | putnopvut | That deadlock you found is...interesting. It's a three-way deadlock. |
| 16:48.47 | flujan | putnopvut: wow |
| 16:49.08 | Qwell | 3-way? |
| 16:49.14 | flujan | do you need some additional information? I will keep looking the server if it crashs again... |
| 16:49.21 | Qwell | like 3 locks in 3 threads? |
| 16:49.31 | putnopvut | flujan: no, I don't think more info is needed. I think I know how to fix. |
| 16:49.37 | putnopvut | Qwell: yeah, exactly. |
| 16:49.41 | Qwell | wow |
| 16:49.55 | putnopvut | Qwell: well, sort of. Each thread has two of the three locks involved in the deadlock |
| 16:50.05 | Qwell | fun |
| 16:50.28 | putnopvut | I agree. I may know how to work around it though. |
| 16:54.13 | flujan | putnopvut: I have little experience with the C programming. I read the code http://pastie.caboo.se/144816 here and I have some questions. |
| 16:55.20 | flujan | putnopvut: why do you mark the members dead and later treat the deleted ones... Wouldn't be more simple to just fetch the member from the database and replace the ones allocated in the memory? |
| 16:56.24 | flujan | I just asking because I really want to get more involved in the asterisk code... It is pleasing me. I learned a lot just debbuging it... and I really like to learn stuff... |
| 16:58.22 | putnopvut | flujan: you bring up a good point. When I wrote that function, I pretty much copied what was done in the rest of the file. I believe it was done this way because it actually will run quicker this way. |
| 16:59.10 | putnopvut | This way, you just read everything from the database and then later delete everything that doesn't belong instead of reading from the database, searching through the list of members for the appropriate one to replace, and then updating the information. |
| 16:59.58 | flujan | putnopvut: Yeap, my point is just to fetch and REPLACE... do not performing a search on the list members... |
| 17:00.58 | flujan | I believe that in some point there is a linked list of something similiar associated with the queue. Just remove this pointer and point to another new linked list of members fetched from the database. |
| 17:01.19 | flujan | no whiles in the code, that is my point... |
| 17:01.45 | flujan | the whiles will tane 0(n) to process all members and just the aproach of fetch replace will take 0(1). |
| 17:01.47 | flujan | IMHO. |
| 17:02.37 | flujan | please, if I say some bullshit correct me... Sometimes I just travel a lot. :P |
| 17:04.53 | flujan | and I belive the deadlock is happening because some thread just enters a while and never exits so it will not run the ast_mutex_unlock(&q->lock); |
| 17:42.17 | putnopvut | flujan: I see what you're saying about the O(1) replacement. The way members are used now, that would be possible (before finding and replacing would have required traversing a linked list, but now they can be found in a hash table with an O(1) lookup). The deadlock is happening due to bad locking order. |
| 17:42.43 | putnopvut | I'm in the process of adding a note to issue 11862 detailing what's going wrong. |
| 17:42.59 | flujan | putnopvut: ok thanks for the explanations. :) |
| 17:44.26 | flujan | hash table just rocks :D |
| 17:47.07 | putnopvut | I agree. |
| 17:48.25 | flujan | putnopvut: where is the best place to learn and starting coding something to asterisk? I like to help, but I haven't the necessary knowledge of the system to do that. :( |
| 17:48.45 | Qwell | flujan: app modules are probably the easiest place to start |
| 17:49.49 | flujan | Qwell: thanks for the tip. :) And about books and stuff? I already search but found no good books about Asterisk programming of stuff... |
| 17:50.05 | Qwell | yeah, no books to speak of |
| 17:53.43 | flujan | Qwell: that's is sad... |
| 17:54.07 | flujan | Qwell: with a book more and more developers would join the asterisk world... :D |
| 17:55.55 | jsmith | flujan: You have no idea how hard it would be to write a book like that |
| 17:56.07 | jsmith | flujan: Not only that, but by the time it was published, it would be out of date |
| 17:56.33 | flujan | jsmith: for sure, asterisk is changing a lot... I checked the code of 1.6 beta... man... how it changes... |
| 17:56.39 | jsmith | flujan: I would *love* to see something like that happen, but it's too much work for someone to do on their own without major funding |
| 17:57.25 | flujan | jsmith: maybe digium would have interest in doing that... dunno. |
| 17:58.21 | flujan | for instance, I am a third party developer. I developed a solution to work with asterisk from the web... But I would like to join and try to help asterisk. Just haven't knowledge enough to do that. |
| 17:58.55 | Qwell | M11863 |
| 17:58.56 | MuffinMan | No information available. |
| 17:58.59 | jsmith | At some point in the future, Digium may offer some developer classes, but I don't see any written developer documentation coming out anytime soon |
| 17:59.03 | Qwell | Anybody have any idea why that's private? |
| 17:59.13 | file | Qwell: because the submitter put it as private? |
| 17:59.27 | Qwell | jsmith: Digium (Tilghman) is actually currently doing dev classes...internally, anyways |
| 17:59.38 | Qwell | well, mostly internally |
| 18:00.28 | jsmith | Qwell: I thought that was just C classes, and not necessarily Asterisk development classes |
| 18:00.37 | Qwell | asterisk-related C classes |
| 18:00.39 | Corydon76-dig | jsmith: it's both |
| 18:00.44 | jsmith | (no pun intended with the word "class") |
| 18:00.58 | Qwell | C has no classes ;) |
| 18:01.03 | jsmith | Corydon76-dig: Oh? I had no idea... I should join the class (virtually) |
| 18:01.14 | jsmith | Qwell: I know... What's the struct of the class? |
| 18:01.16 | Qwell | jsmith: I keep telling him he needs to do it online :) |
| 18:01.18 | Corydon76-dig | jsmith: I'm an awful teacher, anyway |
| 18:01.32 | jsmith | Corydon76-dig: But you know the material. I"d be happy to present it if I knew it better. |
| 18:01.44 | Corydon76-dig | and what's worse is they're recording it |
| 18:02.26 | Corydon76-dig | jsmith: Part of the good of the class is that I've told the students to interrupt me if something doesn't make sense |
| 18:02.49 | jsmith | Corydon76-dig: Ooooh! I need a copy of the recording |
| 18:02.54 | Qwell | Corydon76-dig: "wait, wait, go back. What's a pointer again?" |
| 18:03.07 | jsmith | Corydon76-dig: (and the notes, and the assignments, and the lecture notes) |
| 18:03.08 | Corydon76-dig | When I teach, I have a history of having a tenth of the students left at the last class |
| 18:03.15 | Qwell | thankfully, the people that are in the class are capable of learning, heh |
| 18:04.52 | Corydon76-dig | Not intentionally, you understand. It just sort of works out that way |
| 18:05.37 | flujan | I really have a lot of interest on these classes... Specifically the C-asterisk ones. I know C. Do not use on my day job... But I love C. |
| 18:06.53 | Corydon76-dig | jsmith: I teach like I program. I'm winging it. |
| 18:07.33 | jsmith | Corydon76-dig: My programming skills are getting rusty since all I do any more is teach, respond to email, and write documentation |
| 18:08.11 | jsmith | Corydon76-dig: I'm not sure I could program myself out of a wet paper bag anymore... I understand the concepts, I just haven't written anything in C in so long... |
| 18:08.22 | Corydon76-dig | Ah, yeah... |
| 18:10.15 | flujan | Corydon76-dig: jsmith maybe the solution is a two part course. |
| 18:10.27 | flujan | for instance, jsmith teach the concepts... |
| 18:10.45 | flujan | so Corydon76-dig will have better prepared students... |
| 18:11.04 | flujan | Or maybe more levels... I say it will be cool to learn for people like you... |
| 18:13.05 | flujan | the point is that there will be a lot of people interested in learning to help asterisk... |
| 18:13.56 | flujan | it really demotivate when do start do learn something and have none to ask question... or you don't know where to start. |
| 18:14.28 | flujan | for instance, when I reported bugs I have never used gdb, valgrind or debug threads before... |
| 18:14.32 | jsmith | There's always the -dev mailing list (for asking questions) |
| 18:20.01 | flujan | jsmith: but I, for example, will not feel comfortable to ask dumb questions there or here. I just like to learn and examine a problem before asking about it... |
| 18:20.42 | flujan | and Corydon76-dig don't matter if you have just having a tenth of the students left at the last class |
| 18:21.09 | flujan | what matters if what happened to this students and if they learned from you what they would like to learn. |
| 18:22.12 | Corydon76-dig | Yeah, that's what I tell myself |
| 18:22.48 | flujan | Corydon76-dig: ;) |
| 18:29.53 | flujan | Corydon76-dig: stop telling and believed it... So you will be able to give some asterisk classes... eheheh. You guys have a lot of knowledge let us mortals learn a bit a evolve too. :P |
| 18:34.16 | Corydon76-dig | Eh, I'm actually better on IRC than I am in person |
| 18:35.52 | flujan | Corydon76-dig: I don't belive in two people... eheheh you are just the same... Just changes the name. From Corydon76-* to Corydon. :D |
| 18:37.24 | flujan | Corydon76-dig: so think well about it. Seriously... ehehe asterisk dev course will rules... |
| 19:54.12 | jmls | M11845 |
| 19:54.13 | MuffinMan | [assigned] [Asterisk] Core-General 0011845: Small memory leak in cdr.c reported by jmls http://bugs.digium.com/view.php?id=11845 |
| 19:54.26 | jmls | is there any chance of this getting fixed before the .18 release ? |
| 19:55.20 | jmls | dammit. |
| 20:00.53 | codefreeze | jmls: we'll see. I'll try to devote a few days/week to cdrs again soon... |
| 20:01.16 | jmls | thanks codefreeze ... |
| 20:01.35 | jmls | it's the leak that concerns me. |
| 20:01.42 | jmls | Who need stinking cdrs ... |
| 20:02.00 | jmls | all your cdrs are belong to us |
| 20:06.44 | russellb | jmls: if you can come up with a simple configuration that makes it easy for me, or someone, to reproduce it, that would rock :) |
| 20:07.36 | putnopvut | russellb: he actually gave a really simple scenario in the bug report. |
| 20:07.42 | russellb | oh. |
| 20:07.45 | jmls | heh |
| 20:10.29 | russellb | gah, that means i have to write something to do an action: redirect :-p |
| 20:10.59 | jmls | real men do it on the command line |
| 20:17.33 | russellb | heh, telnet ftw? |
| 20:18.34 | jmls | that's how I proved it ;) |
| 20:19.14 | Qwell | netcat > telnet |
| 20:19.25 | Qwell | netcat+GAPING_SECURITY_HOLE > telnet, rather |
| 20:19.48 | Qwell | that's like my favorite define out of any application... |
| 20:22.21 | jsmith | Qwell: Where's that one? |
| 20:22.21 | Qwell | netcat |
| 20:22.21 | jsmith | Ah... |
| 20:22.24 | Qwell | it enables the -c option I think |
| 20:22.50 | jsmith | Gotcha |
| 20:22.56 | Qwell | it's horribly insecure |
| 20:30.56 | flujan | putnopvut: do you have a second ? |
| 20:32.28 | putnopvut | flujan: sure, what's up? |
| 20:33.00 | flujan | putnopvut: I also notice another think that concerns with the bug I reported... |
| 20:33.12 | putnopvut | What's that? |
| 20:33.26 | flujan | here asterisk have a memory leak. I need to restart the process every night |
| 20:33.41 | flujan | now I notice that the memory consumption stops at 110 MB. |
| 20:33.47 | flujan | it is running up for 4 hours... |
| 20:33.51 | putnopvut | Really? Do you know where the leak is? |
| 20:34.16 | flujan | putnopvut: no but I think it could be on the realtime engine on in the queue. |
| 20:34.30 | flujan | since it is the only thinks that changed... |
| 20:34.39 | flujan | how can I invetigate it? |
| 20:35.08 | putnopvut | If you compile with the MALLOC_DEBUG flag set in menuselect, there are some CLI commands you can use to see memory allocations. |
| 20:35.28 | putnopvut | I think the commands are "core show memory" and "core show allocations" |
| 20:35.44 | flujan | well, I don't compile with this option... Will recompile it later in the night. |
| 20:35.46 | putnopvut | Also valgrind can be used to help with memory leak detection, but I haven't used it for that purpose. |
| 20:36.23 | flujan | putnopvut: ok. I have valgrind installed... When the patch comes and I return the the realtime queue, I will for sure try to detect it. :D |
| 20:36.28 | flujan | now everything is fine. |
| 20:36.37 | putnopvut | Okay, cool :) |
| 20:36.46 | flujan | dunno if it helps but the memory is consumed in 4 and 4 kb... |
| 20:37.22 | putnopvut | I can look through the realtime section of app_queue to see if I notice any potential memory leaks. |
| 20:38.54 | flujan | ok, thanks again for the help. :D |
| 20:39.59 | putnopvut | Always a pleasure. |
| 21:16.11 | *** part/#asterisk-bugs flujan (n=flujan@200-160-115-020.static.spo.ctbc.com.br) |
| 21:39.51 | *** join/#asterisk-bugs fakhir (n=fakhir@unaffiliated/fakhir) |
| 22:38.20 | *** join/#asterisk-bugs kpfleming (n=kpflemin@216.207.245.1) |
| 22:38.20 | *** mode/#asterisk-bugs [+o kpfleming] by ChanServ |
| 23:03.59 | putnopvut | Aw, flujan left. He was right about a memory leak in the realtime code. |