| 03:13.46 | *** join/#asterisk-dev mjordan (~mjordan@75.76.55.191) |
| 03:13.46 | *** mode/#asterisk-dev [+o mjordan] by ChanServ |
| 06:26.48 | *** join/#asterisk-dev leedm777 (~leedm777@50-81-28-152.client.mchsi.com) |
| 06:58.33 | *** join/#asterisk-dev oej (~oej@h87-96-134-129.dynamic.se.alltele.net) |
| 06:58.33 | *** mode/#asterisk-dev [+o oej] by ChanServ |
| 08:32.01 | *** join/#asterisk-dev tzafrir (~tzafrir@local.xorcom.com) |
| 08:32.01 | *** mode/#asterisk-dev [+o tzafrir] by ChanServ |
| 11:00.58 | *** join/#asterisk-dev oej (~oej@h87-96-134-129.dynamic.se.alltele.net) |
| 11:00.58 | *** mode/#asterisk-dev [+o oej] by ChanServ |
| 11:16.35 | *** join/#asterisk-dev oej (~oej@h87-96-134-129.dynamic.se.alltele.net) |
| 11:16.36 | *** mode/#asterisk-dev [+o oej] by ChanServ |
| 12:21.18 | *** join/#asterisk-dev Demon_VoIP (~demon@ip253.net222.n37.ru) |
| 12:42.12 | *** join/#asterisk-dev oej (~oej@h87-96-134-129.dynamic.se.alltele.net) |
| 12:42.12 | *** mode/#asterisk-dev [+o oej] by ChanServ |
| 13:08.05 | *** join/#asterisk-dev oej (~oej@h87-96-134-129.dynamic.se.alltele.net) |
| 13:08.05 | *** mode/#asterisk-dev [+o oej] by ChanServ |
| 14:16.44 | *** join/#asterisk-dev tzafrir (~tzafrir@local.xorcom.com) |
| 14:16.44 | *** mode/#asterisk-dev [+o tzafrir] by ChanServ |
| 15:44.33 | *** join/#asterisk-dev [TK]D-Fender (~joe@64.235.216.2) |
| 16:08.28 | *** join/#asterisk-dev jozza (~chatzilla@BSN-77-186-224.static.siol.net) |
| 16:10.40 | jozza | hey guys, i want to finally submit a patch for 1.8. Which version should make a diff against? I made one against a download from regular dl site. Is that ok? |
| 16:13.08 | *** join/#asterisk-dev mjordan (~mjordan@75.76.55.191) |
| 16:13.08 | *** mode/#asterisk-dev [+o mjordan] by ChanServ |
| 16:14.44 | file | jozza, Asterisk 1.8 is in security fix only |
| 16:15.05 | file | nods to mjordan |
| 16:15.15 | jozza | file: i see, so no changes anymore |
| 16:15.57 | file | Asterisk 11 and 13 are the current branches |
| 16:16.12 | jozza | file: so i should make a patch agains 11 and 13? |
| 16:16.24 | file | what is the patch for? |
| 16:16.48 | jozza | it is a sip patch that allows setting externaddr per peer |
| 16:16.57 | jozza | i have been using it for a lonf time |
| 16:18.21 | file | that would be a new feature, so trunk but you could propose it for a release branch (provided tests are available) |
| 16:18.57 | jozza | one of my providers is very strict and i have multiple exit points from asterisk to registrants, so this patch makes each trunk present itself as its own extrern ip |
| 16:19.54 | jozza | i dont know what tests means, but this patch is 100% working |
| 16:20.00 | jozza | tested for years |
| 16:21.43 | jozza | i lose interest quickly when i have to do things that i have no time for |
| 16:23.08 | file | you can submit it for trunk then, but it won't go into an earlier branch |
| 16:23.14 | jozza | getting into track with all asterisk demands to get a patch built in, would be that thing |
| 16:23.59 | file | the process is documented at https://wiki.asterisk.org/wiki/display/AST/Patch+Contribution+Process |
| 16:24.26 | jozza | doesnt matter when, i'm on 1.8 for now, but i would like to be able to upgrade sometime and have this feature available |
| 16:24.57 | file | just be aware that as of Asterisk 13 chan_sip is a community supported module |
| 16:25.32 | jozza | what does that mean? and how is it different from 11? |
| 16:27.04 | file | http://lists.digium.com/pipermail/asterisk-dev/2014-August/069709.html |
| 16:27.20 | file | what that means for chan_sip is detailed there |
| 16:31.41 | jozza | a guick glimps looks like that chan_pjsip is replacing chan_sip as official sip stack |
| 16:33.06 | file | yes |
| 16:35.44 | jozza | ok so, chan_pjsip will become only a glue for external pjsip library? |
| 16:36.01 | file | it uses the PJSIP library underneath |
| 16:36.18 | jozza | it's got to be a small file then |
| 16:36.44 | file | it's made of many different files and modules which implement different functionality and features |
| 16:39.17 | jozza | if i install ast 13, is pjsip a default sip stack? what .conf file does it use? Is chan_sip still there, soes it get compiled automatically or does it have to be enabed manually? |
| 16:39.25 | file | they are completely different |
| 16:39.40 | file | https://wiki.asterisk.org/wiki/display/AST/Configuring+res_pjsip |
| 16:39.52 | mjordan | jozza: "getting into track with all asterisk demands to get a patch built in, would be that thing" ==> requiring Asterisk Developerps to do the dirty work usually makes us lose interest in the patch as well |
| 16:40.09 | mjordan | if it's important to you, it's important to make sure that others don't come along and break it. |
| 16:40.19 | WIMPy | The usual dilemma. |
| 16:40.32 | mjordan | if it's not worth your time to make sure that it's in the project correctly and safely, then it's not worth anyone else's time either. |
| 16:45.14 | jozza | i'm just saying that procedures required by you, take up time that i dont have. If i just make a patch for me, takes me minutes. Otherwise i need to strictly follow automatic systems i don't like. I did try to submit a patch before, but space here, space there, new line here and there, i gve up |
| 16:47.08 | mjordan | jozza: if you aren't willing to make sure the patch is maintainable for the future, then your only option is to wait around until someone else is willing to do the work for you. I commiserate on not having enough free time; alas, that's a universal statement, and unfortunately, my time is spent on the patches and issues that affect the most people or get the best bang for the buck. If that includes your patch, great. If not, then it's on you. |
| 16:47.29 | mjordan | and we aren't going to sacrifice quality in the mad dash for throwing more things into Asterisk. Been there, got the T-Shirt, lived through the nightmare. |
| 16:48.12 | WIMPy | Hmmm |
| 16:48.14 | mjordan | if you are willing to invest the time and energy, then there are lots of people willing to collaborate and help. |
| 16:49.32 | mjordan | file: oh hey, crazy - https://issues.asterisk.org/jira/browse/ASTERISK-24852 |
| 16:50.07 | file | mjordan, that individual was in here... Friday, yes, Friday |
| 16:50.15 | mjordan | alas, I was on a plane. |
| 16:58.31 | jozza | thx for info guys |
| 17:09.14 | seanbright | kind of a dick |
| 17:13.27 | mjordan | everyone thinks their time is more valuable. |
| 17:13.45 | mjordan | Alas for him, I feel the same way, particularly when you don't want to put any effort into it. |
| 18:26.07 | *** join/#asterisk-dev oej (~oej@h87-96-134-129.dynamic.se.alltele.net) |
| 18:26.07 | *** mode/#asterisk-dev [+o oej] by ChanServ |
| 18:37.33 | WIMPy | Can someone take a look at the locks on ASTERISK-24850? Do I get it right that the issue is that zombified IAX task holds a lock that prevents dialplan lookups? |
| 19:17.31 | *** join/#asterisk-dev leedm777 (~leedm777@50-81-28-152.client.mchsi.com) |
| 19:19.07 | *** join/#asterisk-dev leedm777 (~leedm777@50-81-28-152.client.mchsi.com) |
| 19:21.46 | *** join/#asterisk-dev leedm777 (~leedm777@50-81-28-152.client.mchsi.com) |
| 19:23.02 | *** join/#asterisk-dev leedm777 (~leedm777@50-81-28-152.client.mchsi.com) |
| 19:25.07 | *** join/#asterisk-dev leedm777 (~leedm777@50-81-28-152.client.mchsi.com) |
| 19:30.00 | *** join/#asterisk-dev leedm777 (~leedm777@50-81-28-152.client.mchsi.com) |
| 19:37.27 | *** join/#asterisk-dev sruffell (sruffell@asterisk/the-kernel-guy/sruffell) |
| 19:37.27 | *** mode/#asterisk-dev [+o sruffell] by ChanServ |
| 20:42.45 | *** join/#asterisk-dev superscrat (~asanders@173-17-133-2.client.mchsi.com) |
| 20:52.36 | *** join/#asterisk-dev oej (~oej@h87-96-134-129.dynamic.se.alltele.net) |
| 20:52.36 | *** mode/#asterisk-dev [+o oej] by ChanServ |
| 21:17.37 | *** join/#asterisk-dev mjordan (~mjordan@75.76.55.191) |
| 21:17.37 | *** mode/#asterisk-dev [+o mjordan] by ChanServ |
| 21:22.10 | mjordan | WIMPy: nope. |
| 21:22.30 | mjordan | WIMPy: chan_lcr is doing something wrong by causing a locking inversion |
| 21:23.19 | WIMPy | Hmm. And that causes IAX to go to zombie state? |
| 21:23.34 | mjordan | lcr_request is holding a channel lock while attempting to get the channels container lock. |
| 21:24.05 | mjordan | It would, because lcr_request is holding a channel lock that causes everyone else to pile up |
| 21:24.21 | WIMPy | Ok. |
| 21:24.42 | mjordan | at least that's what I can tell from the 'core show locks' output. |
| 21:24.55 | WIMPy | Now the interesting thing is where to look for that lock. |
| 21:25.25 | mjordan | some line number in lcr_request. |
| 21:25.36 | mjordan | I would suspect it is attempting to link it into the channels container. |
| 21:25.43 | WIMPy | Or why it only seem to happen if IAX is involved. |
| 21:26.19 | mjordan | because chan_iax2 has to grab the channels container? |
| 21:26.30 | mjordan | Honestly, if you ran a 'core show channels' it would probably block that as well. |
| 21:27.03 | mjordan | it also looks like you've got IAX channels in a bridge with the LCR channels |
| 21:27.25 | mjordan | which would cause those IAX channels to block while their thread attempts to write to the LCR channel (which is locked by chan_lcr) |
| 21:27.31 | WIMPy | The result looks extremely similar to theissue I ran in to on the upgrade from 11.7.0 to 11.7.1, but that did not involve lcr, only iax and sip. |
| 21:27.47 | mjordan | that core show locks doesn't show anything that I would blame on chan_iax. |
| 21:28.10 | WIMPy | But it seems to have trouble writing to IAX not from. |
| 21:28.46 | mjordan | WIMPy: " * \note Absolutely _NO_ channel locks should be held before calling this function." |
| 21:28.57 | mjordan | that's ast_channel_alloc, which chan_lcr is calling into, with a channel lock held |
| 21:29.00 | WIMPy | All these locking things don't really mean much to me :-( |
| 21:29.02 | mjordan | that will absolutely cause a deadlock. |
| 21:29.15 | mjordan | Once a channel deadlock happens, everyone gets screwed |
| 21:29.30 | mjordan | the channels container is permanently locked, and everyone grinds to a halt |
| 21:29.48 | WIMPy | alloc? So that would be when a new channel is set up? |
| 21:30.26 | mjordan | WIMPy: that is when you create a new channel, and I can look at chan_lcr and see them doing it, which I should *not* be doing since that is code that isn't licensed back to the project. |
| 21:30.47 | mjordan | line 2103, followed by line 2125. https://github.com/osmobuntu/lcr/blob/master/chan_lcr.c |
| 21:30.59 | mjordan | It violates the contract of the API, and will absolutely cause deadlocks willy nilly. |
| 21:31.41 | WIMPy | Ok. Thanks for pointing me in the right direction. |
| 21:33.02 | WIMPy | just wonders what version that is yu found, BTW |
| 21:33.10 | WIMPy | you |
| 21:33.42 | mjordan | er |
| 21:34.00 | mjordan | WIMPy: that lock isn't a channel lock. They globally lock everytime they make a channel. Weird. |
| 21:34.03 | mjordan | Hm. |
| 21:34.32 | mjordan | they are still deadlocking it. |
| 21:34.37 | mjordan | but it's a weird one. |
| 21:34.44 | mjordan | mostly because of their stupid global lock. |
| 21:34.57 | mjordan | they grab their global lock (chan_lock), then attempt to grab the channel container lock. |
| 21:35.05 | WIMPy | They way you describe it makes it interesting that it works at all. |
| 21:35.21 | mjordan | that causes a locking inversion when the core - which holds a channel lock - calls ast_write on them |
| 21:35.38 | mjordan | that channel lock blocks another thread which has grabbed the global channels lock and is attempting to grab that channel lock |
| 21:35.47 | mjordan | that global lock is still the problem. |
| 21:35.56 | mjordan | but it is a bit more complex than I thought. |
| 21:36.37 | mjordan | WIMPy: generally, you shouldn't hold locks all over the place, and the locking order is supposed to be "channel container, then channel" |
| 21:37.07 | mjordan | what they've done is thrown a third lock into the mix (generally a bad idea), then complicate that by having it grabbed when a channel lock may be held. |
| 21:37.29 | mjordan | They create the inversion when they hold that global lock and grab the channel container lock by creating a new channel (when the new channel is done being created, it has to go into the container). |
| 21:38.05 | mjordan | Why they have a global lock and are locking it left & right, I don't know. That seems like a terrible idea for performance. They probably thought they were solving some concurrency problem, and stepped right into creating worse ones. |
| 21:38.12 | WIMPy | That again sounds like it should happen regularly. |
| 21:38.25 | mjordan | well, kind of. It involves at least three threads. |
| 21:38.43 | WIMPy | The shit is horribly old :-( |
| 21:50.41 | WIMPy | Do you think there needs to be a bridged lcr channel when a new lcr channel is created for the issue to happen? |
| 21:56.24 | WIMPy | Could it be that the whole lock isn't neccessay at all? |
| 22:35.07 | WIMPy | is confused. |
| 22:35.51 | WIMPy | While chan_misdn (which obviousely has been the template for chan_cr at some point) doesn't seem to do locking, while chan_dahdi looks like chan_lcr looks like now. |
| 22:38.07 | WIMPy | Also alsa does it an console doesn't. |
| 23:05.17 | mjordan | chan_dahdi doesn't have a global lock. |
| 23:06.13 | mjordan | it has an interface lock for its private structures, but it doesn't use it in the same way. |
| 23:06.30 | mjordan | chan_misdn is garbage, so I can't comment on that. |
| 23:07.12 | mjordan | granted, everything has issues, but chan_misdn does some horrible stuff with holding onto memory in unsafe ways |
| 23:07.24 | WIMPy | I have to admit that I'm too clueless about that locking stuff to see the difference :-( |
| 23:07.30 | mjordan | and generally can't be compiled any more, thanks to libmisdn's reliance on an old version of the kernel. |
| 23:08.03 | mjordan | WIMPy: lock inversions are hard. In this case, I'm positive it is chan_lcr. They are holding onto their global lock irrespective of the locking contract that a channel driver must adhere to. |
| 23:08.05 | WIMPy | Well, it has been "old and deprecated" for many years. |
| 23:08.43 | mjordan | to be frank, while deadlocks do still happen in Asterisk from time to time, in the core channel drivers, we have been aggressive about killing them. |
| 23:09.03 | WIMPy | Do I get it right that this lock would prevent other threads from accessing an unfinished structure? |
| 23:09.13 | mjordan | Richard's latest fix for one in chan_sip is one of the few I've seen crop up in the last year. |
| 23:09.58 | WIMPy | Maybe that was the one I got with SIP transfers? |
| 23:10.37 | mjordan | WIMPy: maybe. It occurs rarely when you poke a peer while inside of certain messages. He just committed the fix, I think (I was out most last week) |
| 23:11.14 | mjordan | WIMPy: I'm not sure what they are doing with that global lock. You don't need to protect something that is allocated on the stack, so I'm not sure why they are aggressively holding it. |
| 23:11.41 | WIMPy | Actually that one was a lot more interesting on the also involved IAX side. |
| 23:12.14 | mjordan | keep in mind that once something deadlocks the channels container, all channel drivers are screwed |
| 23:12.25 | mjordan | what is on the other side of a bridge usually doesn't matter. |
| 23:12.29 | WIMPy | I have tried to just remove the locking which does work. But I guess that doesn't really mean much. |
| 23:12.43 | mjordan | heh, yeah, now all you know is that you won't deadlock. May blow up instead. |
| 23:12.58 | WIMPy | For the sip transfer issue it did. |
| 23:12.58 | mjordan | honestly, we get very few deadlocks on the issue tracker when using just the core channel drivers. |
| 23:13.08 | mjordan | shrugs |
| 23:13.17 | mjordan | I can't recall what that issue was, so I'm not sure |
| 23:14.29 | WIMPy | That was the very strange thing that my box would lock up in the same way as on this issue when I tied to transfer a call that come in via IAX on the sip phone. |
| 23:15.19 | WIMPy | The thing that made it interesting was that it would happen again on the next try until I restarted the other Asterisk box where the call came from. |
| 23:15.55 | WIMPy | That's why I asked that time if there is anything in IAX in any way related to realtime. |
| 23:17.22 | *** join/#asterisk-dev superscrat (~asanders@173-17-133-2.client.mchsi.com) |
| 23:20.17 | WIMPy | But back to this one: Can you think of some action that might have an issue with the lock removed? |
| 23:20.45 | WIMPy | Something like 'core show channels' displaying incomplete information would be obvious, but harmless. |
| 23:21.05 | mjordan | which lock? |
| 23:21.08 | mjordan | the one in chan_lcr? |
| 23:21.17 | WIMPy | yes |
| 23:23.24 | mjordan | I'd say go for it. Particularly the one in the request callback. |
| 23:23.40 | mjordan | the other option would be to unlock around the ast_channel_alloc call and then re-lock. |
| 23:23.48 | WIMPy | Yes, that's the one I removed. |
| 23:23.58 | mjordan | not knowing all of their code, I'm really not sure, but that would prevent the locking inversion. |
| 23:26.43 | WIMPy | alloc_call is the only thig happening before. Doesn't sound like something that needs any locks. Maybe I just move the lock to after ast_channel_alloc? That would seem to be the safe bet. |
| 23:27.00 | WIMPy | That was actually my first idea until I looked in to the other channels. |
| 23:40.14 | WIMPy | OTOH, if other channels work without loking anything, there can't be a need to do it, can there? |
| 23:40.49 | file | that's an implementation detail of the channel driver itself |
| 23:41.07 | WIMPy | I will try to find out if Jolly remembers why it was there. |
| 23:41.33 | WIMPy | But can it be called again for the channel to be created before the function returns? |
| 23:42.08 | *** join/#asterisk-dev sgriepentrog (sgriepentr@nat/digium/x-qvecpniglwsxdign) |
| 23:50.07 | file | I don't understand the question |
| 23:52.04 | WIMPy | Is there any chance the channel will will be called for the channel that might be incompletely created, befor that very function completes? |
| 23:53.11 | file | it is possible for something to get the channel after it is allocated. |
| 23:54.38 | WIMPy | Yes, but is there anyhting that could have an issue with the incomplete channel? |
| 23:55.03 | file | no? |
| 23:55.35 | WIMPy | That's what I hoped. |
| 23:56.16 | WIMPy | And in the end there would always remain the possibility to hit the very moment beween ast_channel_alloc and any locking. |
| 23:56.52 | WIMPy | I think I'm positive that the lock can just go. |
| 23:59.19 | WIMPy | I'll keep it running without here and see what Jolly can tell about it. |