| 00:10.04 | *** join/#asterisk-dev tzafrir (~tzafrir@212.179.75.202) |
| 00:10.04 | *** mode/#asterisk-dev [+o tzafrir] by ChanServ |
| 01:47.06 | *** join/#asterisk-dev leifmadsen (~Leif@asterisk/documenteur-extraordinaire/blitzrage) |
| 01:47.07 | *** mode/#asterisk-dev [+o leifmadsen] by ChanServ |
| 02:26.49 | *** join/#asterisk-dev seanbright (~sean@asterisk/contributor-and-bug-marshal/seanbright) |
| 02:26.49 | *** mode/#asterisk-dev [+o seanbright] by ChanServ |
| 03:11.09 | *** join/#asterisk-dev rgagnon (~rgagnon@rrcs-71-42-183-54.sw.biz.rr.com) |
| 04:04.27 | *** join/#asterisk-dev otherwiseguy (~otherwise@cpe-69-76-221-79.kc.res.rr.com) |
| 04:21.19 | *** join/#asterisk-dev joshaidan (~brianj@S0106001c1023e838.tb.shawcable.net) |
| 04:28.26 | *** join/#asterisk-dev irroot (~gregory@197.172.239.36) |
| 05:38.22 | *** join/#asterisk-dev sruffell (~Adium@asterisk/the-kernel-guy/sruffell) |
| 05:38.22 | *** mode/#asterisk-dev [+o sruffell] by ChanServ |
| 06:49.16 | *** join/#asterisk-dev irroot (~gregory@41.53.55.133) |
| 08:14.56 | *** join/#asterisk-dev Tim_Toady (~fuzzy@193.92.238.205.dsl.dyn.forthnet.gr) |
| 08:35.18 | *** join/#asterisk-dev tzafrir (~tzafrir@local.xorcom.com) |
| 08:35.18 | *** mode/#asterisk-dev [+o tzafrir] by ChanServ |
| 09:15.01 | *** join/#asterisk-dev oej (~olle@fw1.gml.g.icnet.infracom.se) |
| 09:15.01 | *** mode/#asterisk-dev [+o oej] by ChanServ |
| 10:28.09 | *** join/#asterisk-dev otherwiseguy (~otherwise@69.76.221.79) |
| 10:46.03 | *** join/#asterisk-dev otherwiseguy (~otherwise@cpe-69-76-221-79.kc.res.rr.com) |
| 11:30.34 | *** join/#asterisk-dev micols (~ident@rlogin.dk) |
| 12:02.11 | *** join/#asterisk-dev irroot (~gregory@196-210-140-50.dynamic.isadsl.co.za) |
| 12:10.45 | *** join/#asterisk-dev kpfleming (~kpfleming@asterisk/digium-director-of-software-tech/kpfleming) |
| 12:10.45 | *** mode/#asterisk-dev [+o kpfleming] by ChanServ |
| 12:30.57 | oej | afternoon |
| 12:48.04 | *** join/#asterisk-dev leifmadsen (~Leif@asterisk/documenteur-extraordinaire/blitzrage) |
| 12:48.04 | *** mode/#asterisk-dev [+o leifmadsen] by ChanServ |
| 13:33.46 | *** join/#asterisk-dev acidfoo (~nib@modemcable094.94-70-69.static.videotron.ca) |
| 13:41.08 | *** join/#asterisk-dev WIMPy (~wimpy@e183095026.adsl.alicedsl.de) |
| 13:44.38 | *** join/#asterisk-dev irroot (~gregory@196-210-140-50.dynamic.isadsl.co.za) |
| 14:04.25 | *** join/#asterisk-dev mjordan (~mjordan@nat/digium/x-mlhscgzwiugeeljy) |
| 14:04.25 | *** mode/#asterisk-dev [+o mjordan] by ChanServ |
| 14:05.14 | *** join/#asterisk-dev serafie (~erin@nat/digium/x-xqprwmuvnbcethue) |
| 14:08.29 | *** join/#asterisk-dev elred_ (~elred_@unaffiliated/elred-/x-5010831) |
| 14:20.45 | *** join/#asterisk-dev irroot (~gregory@196-210-140-50.dynamic.isadsl.co.za) |
| 14:31.38 | *** join/#asterisk-dev Tim_Toady (~fuzzy@193.92.238.205.dsl.dyn.forthnet.gr) |
| 14:42.27 | *** join/#asterisk-dev joshaidan (~brianj@S0106001c1023e838.tb.shawcable.net) |
| 14:53.53 | *** join/#asterisk-dev eliel_ (~quassel@200.32.121.66) |
| 15:10.23 | *** join/#asterisk-dev The_Boy_Wonder (~vossel@asterisk/batman-developer/dvossel) |
| 15:10.23 | *** mode/#asterisk-dev [+o The_Boy_Wonder] by ChanServ |
| 15:27.24 | *** join/#asterisk-dev cemeh4 (~cemeh4@193.19.228.130) |
| 15:29.29 | *** join/#asterisk-dev kpfleming (~kpfleming@asterisk/digium-director-of-software-tech/kpfleming) |
| 15:29.29 | *** mode/#asterisk-dev [+o kpfleming] by ChanServ |
| 15:30.39 | *** join/#asterisk-dev anonymouz666 (~anonymouz@187-28-37-118.poolip.RJO.embratel.net.br) |
| 15:52.43 | *** join/#asterisk-dev anthm (~anthm@freeswitch/developer/anthm) |
| 16:24.29 | *** join/#asterisk-dev JunK-Y (~junky@pdpc/supporter/active/junk-y) |
| 16:32.55 | *** join/#asterisk-dev sruffell (~Adium@asterisk/the-kernel-guy/sruffell) |
| 16:32.55 | *** mode/#asterisk-dev [+o sruffell] by ChanServ |
| 16:50.53 | *** join/#asterisk-dev putnopvut (~putnopvut@asterisk/master-of-queues/mmichelson) |
| 16:50.53 | *** mode/#asterisk-dev [+o putnopvut] by ChanServ |
| 17:00.25 | *** join/#asterisk-dev bklang (~bklang@2001:470:8:5de:225:ff:fe49:f8cd) |
| 17:01.41 | bklang | Hey all. I've got what smells like a race condition, I'm hoping someone can sanity check my assumptions. This is Asterisk 1.8-svn from this morning. The issue is an AMI Redirect that intermittently fails. |
| 17:01.45 | bklang | When it works, I get this message: DEBUG[28915] channel.c: Planning to masquerade channel SIP/grant-0000000c into the structure of AsyncGoto/SIP/grant-0000000c |
| 17:01.52 | bklang | And when it doesn't work, that message does not appear |
| 17:02.12 | kpfleming | yes, that sounds like a race or deadlock |
| 17:02.26 | bklang | deadlocks would be noisy right? |
| 17:03.07 | Kobaz | not necessarily |
| 17:03.08 | Kobaz | core show locks |
| 17:03.19 | Kobaz | speaking of deadlocks |
| 17:03.28 | bklang | well what happens is that the Redirected channel simply hangs up immediately |
| 17:03.53 | Kobaz | kpfleming: what's your thoughts on replacing channel variables with lockless data structures, and convering control frames to be in lockless queues |
| 17:04.33 | bklang | I don't have a "core show locks" command. Do I need to recompile with a debug option? |
| 17:04.39 | kpfleming | bklang: yes, DEBUG_THREADS |
| 17:04.40 | Kobaz | been talking to a concurrency guy, there's methods to use trees and copy on write for lockless structures with high concurrency |
| 17:04.44 | bklang | thanks |
| 17:04.57 | kpfleming | Kobaz: yes, some of those methods exist, but many of them are also covered by patents |
| 17:05.08 | Kobaz | mmm, patents |
| 17:05.14 | kpfleming | i don't really thing that channel variables qualify as 'high concurrency' |
| 17:05.22 | kpfleming | control frames either, for that matter |
| 17:05.32 | Kobaz | no, not necessarily, but it's one less thing to lock |
| 17:05.53 | Kobaz | i like lockless queues |
| 17:06.04 | kpfleming | there are still atomic operations required, and lockless containers use significantly more memory, so there is a tradeoff |
| 17:06.22 | Kobaz | yeah, the spots that need atomic ops go way down |
| 17:06.30 | Kobaz | and they tend to get pushed to one spot in code |
| 17:06.38 | Kobaz | versus locking/unlocking all over the place |
| 17:06.39 | kpfleming | if someone wanted to experiment with it and devise metrics to show how common workloads would be improved, that would be interesting |
| 17:07.12 | Kobaz | yeah.. i was talking with qwell about this... my long term side project will be changing the threading to make deadlocks much less likely |
| 17:07.35 | Kobaz | i care less about workload improvement and more about avoiding deadlocks alltogether |
| 17:07.53 | Kobaz | efficiency can always be worked on after fixing the base issues |
| 17:07.59 | kpfleming | unless you switch to a pure message-passing architecture, that's not practical (and maybe impossible) |
| 17:08.07 | Kobaz | yeah that's my plan |
| 17:08.08 | kpfleming | and Asterisk SCF is already a message passing architecture |
| 17:08.42 | Kobaz | but since scf isn't asterisk, and according to the devs.. never will be, asterisk could use the boost |
| 17:09.18 | kpfleming | what you would create wouldn't be asterisk eitehr |
| 17:09.21 | Kobaz | the plan is to convert things little by little to be message passing |
| 17:09.42 | kpfleming | if 90% of the code requires extensive modification, the result is no longer asterisk |
| 17:10.13 | Kobaz | it would be iterative to avoid the 90% change problem |
| 17:10.28 | Kobaz | i see certain things right off the bat that could be converted, like channel variables |
| 17:10.35 | Kobaz | hangup handling, masquerades |
| 17:10.48 | Kobaz | without affecting a huge part of the other systems |
| 17:10.51 | kpfleming | no, you are missing the point. if the end result is that nearly every module has very different code than what it started with, the result is no longer asterisk. it's like asterisk, but very different. |
| 17:11.18 | kpfleming | that's not possible. every module that interacts with channels is affected by hangups, masquerades, and all that other stuff. |
| 17:11.33 | kpfleming | i'm not trying to be a wet blanket, but we've been down this road already. |
| 17:11.37 | Kobaz | mm |
| 17:12.03 | kpfleming | in the end, if you really want to gain the most benefit from the sorts of changes you want to make, you'll end up with something that only looks like asterisk (at best), but isn't. |
| 17:12.26 | Kobaz | based on the code i've waded through, i think it would be possible to do a module at a time approach with certain things |
| 17:12.47 | Kobaz | module in the sense of core components (not the loadable modules) |
| 17:12.51 | kpfleming | good luck then. it's possible that the 12+ people who looked at this before were wrong. |
| 17:12.57 | Kobaz | heh, hmm |
| 17:13.34 | Kobaz | i'm not doubting the previous efforts... my point was i think it can be done piecewise perhaps they were looking at a full on conversion |
| 17:14.00 | kpfleming | you are completely ignoring my point. there is no value in making just a few small changes; if you are heading down that road, your goal is to make all the changes. |
| 17:14.10 | Kobaz | yeah eventually |
| 17:14.16 | kpfleming | whether you do them all at once or one at a time does not matter, unless you plan to take 10 years to do it. |
| 17:14.38 | Kobaz | but my idea was convert the smallest pieces that have the most probable amount of problems |
| 17:14.50 | *** join/#asterisk-dev otherwiseguy (~otherwise@69.76.221.79) |
| 17:14.51 | kpfleming | and in the meantime, every time an architectural change is 'in the works' its very disruptive to all other asterisk development. just looked at how things were affected by russell's conversion of ast_channel to astobj2. |
| 17:15.03 | Kobaz | yeah |
| 17:15.28 | kpfleming | and it takes months and months to find all the corner cases that were missed, many of them can't be found until a real release is out the door and users have put it into production. |
| 17:15.37 | Kobaz | yeah that's the biggest issue |
| 17:15.40 | kpfleming | there just isn't a 'small, slow' way to approach something like this. |
| 17:17.47 | Kobaz | it just had me thinking, watching the mailing list and seeing deadlock fixes flying around monthly |
| 17:18.08 | Kobaz | there's gotta be some core changeset that could help avoid deadlocks in the first place |
| 17:18.38 | Kobaz | that preferably wouldn't be a huge disruption |
| 17:25.59 | kpfleming | like i said... you are not the first person to have these thoughts. so far, nobody has been able to come up with anything that would help in any material way, without a significant architectural change. |
| 17:27.15 | *** part/#asterisk-dev rgagnon (~rgagnon@rrcs-71-42-183-54.sw.biz.rr.com) |
| 17:29.15 | Kobaz | k, good to know |
| 17:36.06 | Corydon76-work | Kobaz: if you want a fairly big project, there's a fundamental problem with refcounting and the scheduler. If the scheduler's items did not disappear when the scheduled item runs, we could eliminate the very messy workarounds. |
| 17:37.29 | kpfleming | indeed |
| 17:37.37 | Corydon76-work | Because of that disappearing and reappearing act, there's lots of retry loops when cancelling scheduler jobs |
| 17:40.38 | Kobaz | ah, interesting |
| 17:41.09 | Kobaz | the items disappear because they are no longer reffed? |
| 17:41.15 | Kobaz | or there's some other problem? |
| 17:43.08 | *** join/#asterisk-dev irroot (~gregory@41.49.16.67) |
| 17:44.03 | kpfleming | scheduler items don't actually have refcounts yet, iirc |
| 17:45.08 | Kobaz | is there a jira for it? |
| 17:47.40 | Corydon76-work | Items in the scheduler disappear from the list of scheduled items when they're running... and then return to the list when they return a non-zero result |
| 17:48.26 | Kobaz | And I suppose it's not supposed to work like that? should they stay in the list and be marked 'processing' |
| 17:48.28 | Corydon76-work | kpfleming: No, but scheduled items can hold a reference to an object, which makes cancelling them particularly tricky when you are competing with the scheduler |
| 17:48.56 | Corydon76-work | Kobaz: it's supposed to work exactly like that... but it is a fundamental design flaw |
| 17:49.02 | Kobaz | ah |
| 17:49.33 | Corydon76-work | The scheduler, I believe, was originally written for a single-threaded application. |
| 17:52.12 | kpfleming | for a single CPU deployment model, is probably more accurate |
| 17:52.40 | Corydon76-work | I was thinking it was recycled from Cheops or another one of Mark's applications |
| 17:54.00 | Kobaz | so the way it should work would be refcounted and not disappearing during running so you can cancel stuff |
| 17:55.02 | *** join/#asterisk-dev irroot (~gregory@41.52.251.77) |
| 17:55.05 | *** join/#asterisk-dev otherwiseguy_ (~otherwise@cpe-69-76-221-79.kc.res.rr.com) |
| 17:55.35 | Corydon76-work | Kobaz: not refcounted, simply a design change... |
| 17:56.12 | Corydon76-work | 1) Items should not disappear when they are running, so they can be flagged to be cancelled when the scheduled job ends. |
| 17:56.15 | Kobaz | trying to get a handle on the issue, I've never played with the scheduler |
| 17:56.15 | kpfleming | i think they'd have to be refcounted to make it practical |
| 17:56.39 | Kobaz | anything with multiple readers or multiple writers should be ref counted probably |
| 17:56.48 | kpfleming | right |
| 17:57.06 | Corydon76-work | 2) Scheduled items should always return a code to reschedule themselves instead of explicitly calling the scheduler to run another instance of themselves, and then returning 0. |
| 17:57.08 | kpfleming | and then they can be marked 'dead' if they have run and been cancelled |
| 17:57.17 | Kobaz | what would clean them up? |
| 17:57.48 | Corydon76-work | Kobaz: probably would need a destructor passed to the scheduler |
| 17:57.59 | Kobaz | ah that makes sense |
| 17:58.27 | irroot | been captured by the insane greets folks |
| 17:59.31 | Corydon76-work | kpfleming: I think refcounting is overkill, but I also think that the astobj2 code could be reused to avoid code duplication |
| 17:59.45 | kpfleming | right on both counts |
| 17:59.56 | Corydon76-work | I don't think that the scheduler ever needs a refcount greater than 1. |
| 18:00.17 | kpfleming | it does when the code that wants to cancel an item needs a reference to it to be able to modify the item |
| 18:00.45 | kpfleming | unless the schedule is never going to destroy entries, only mark them 'dead' |
| 18:01.16 | Kobaz | wouldn't that be a leak |
| 18:01.20 | Corydon76-work | kpfleming: well, but we cancel it based upon an integer id, and the scheduler run thread becomes responsible for the actual destruction. We only really need a lock on the scheduler item (which I believe we already have) |
| 18:01.28 | Kobaz | or would the destructor have to destroy it's own scheduler item |
| 18:01.37 | kpfleming | ahh, yeah |
| 18:01.49 | kpfleming | the id is a weak reference to the scheduler item |
| 18:02.47 | Corydon76-work | Right |
| 18:06.15 | *** join/#asterisk-dev moy (~moy@173.239.155.74) |
| 18:43.17 | bklang | kpfleming: so that Redirect problem appears to be because the channel is somehow getting marked as a Zombie before the masq happens. Pardon my ignorance, but why might an "up" channel get marked zombie like that? |
| 18:50.47 | kpfleming | channels usually (only?) become zombies as a result of being masqueraded, so that's pretty bizarre |
| 18:51.39 | bklang | I may be reading the logs wrong, or it may be a side-effect of the race. When the Redirect fails, I get "Can't setup masquerade. One or both channels is dead. (%s <-- %s)" (which is inside a test for AST_FLAG_ZOMBIE). |
| 18:52.55 | bklang | I think I'm going to add some logging to see when the ZOMBIE flag is getting set. That's all I can think of at the moment |
| 18:53.12 | kpfleming | that's probably the right thing to do |
| 19:00.44 | *** join/#asterisk-dev snuff-home (~snuffy@203-219-29-182.tpgi.com.au) |
| 19:00.44 | *** mode/#asterisk-dev [+o snuff-home] by ChanServ |
| 19:07.30 | *** join/#asterisk-dev philipp64|laptop (~chatzilla@64.244.80.202) |
| 19:50.23 | *** join/#asterisk-dev joshaidan (~brianj@S010600095bfe15df.tb.shawcable.net) |
| 19:50.41 | *** part/#asterisk-dev serafie (~erin@nat/digium/x-xqprwmuvnbcethue) |
| 19:51.30 | *** join/#asterisk-dev serafie (~erin@nat/digium/x-xqprwmuvnbcethue) |
| 20:07.47 | bklang | naturally, adding logging makes the problem disappear...Sign. |
| 20:07.49 | bklang | Sigh. |
| 20:08.00 | kpfleming | or it's intermittent in the first place |
| 20:08.03 | kpfleming | but yeah... sigh. |
| 20:08.21 | bklang | it's was definitely intermittent before, but I could trigger it reliably about 50% of the time |
| 20:08.33 | bklang | now I've done 15 calls in a row without hitting it |
| 20:26.26 | bklang | OK, I think I've got it. We are using a single Redirect action to split two bridged channels to the dialplan. The ZOMBIE flag comes normally as part of the Redirect function. The race appears to be if one of the channels (seems to consistently be the second leg is impacted) gets out of the bridge (and ast_read() is called?) before it gets masqueraded then it gets hung up because it is marked ZOMBIE. |
| 20:26.49 | kpfleming | hmm |
| 20:27.09 | bklang | I'm not entirely clear why Redirect calls ast_soft_hangup, but I think that's my ignorance of channel handling internally |
| 20:27.22 | kpfleming | the 'need masquerade' indication is just a flag on the channel, which is then processed by ast_read() being called on that channel |
| 20:28.01 | bklang | I think it's chan_sip.c that sees the ZOMBIE flag and hangs it up before the masquerade completes |
| 20:28.04 | kpfleming | masquerades are asynchronous action |
| 20:28.16 | bklang | I'm guessing it has to do with that giant lock grab that the masq attempts repeatedly |
| 20:28.29 | kpfleming | right... a channel driver's read() callback should never be called on a ZOMBIE channel |
| 20:28.54 | kpfleming | so somehow the masquerade is not getting triggered like it should be |
| 20:28.58 | bklang | so I guess I need to chase chan_sip.c up the stack from the hangup and find out why it's trying to read that channel before the masq completes? |
| 20:29.19 | kpfleming | well, not quite, but close |
| 20:29.42 | kpfleming | there is a complication here that one of the two original channels has a thread associated with it (the one running the bridge loop) and the other does not |
| 20:30.05 | kpfleming | i'm not quite sure how Redirect sets things up so that the second channel will have a PBX thread after the masquerade |
| 20:30.31 | bklang | is there an example of "the right way" that I can compare? |
| 20:31.15 | kpfleming | but the soft hangup is done to break any native/direct bridging loop that may be in place, and this should result in the normal bridging loop calling ast_read() on each of the two channels when they are ready... while ast_read() is handling a channel, the need for a masquerade on that channel will be noticed and applied |
| 20:31.23 | kpfleming | no, there is no 'right' way |
| 20:31.49 | kpfleming | i should say, there is no right nor wrong way... there is only 'the way that works' :-) |
| 20:31.58 | bklang | heh |
| 20:33.21 | kpfleming | so... theorizing here: if the Redirec action is creating a new PBX thread for the second channel to use after it has been pulled out of the bridge, and somehow that thread ends up on the second channel before the original thread has caused the masquerade to occur, things will get very messy |
| 20:33.28 | kpfleming | but tracking that down is not going to be easy |
| 20:33.46 | *** join/#asterisk-dev cresl1n (~Adium@asterisk/libpri-and-libss7-expert/Cresl1n) |
| 20:33.46 | *** mode/#asterisk-dev [+o cresl1n] by ChanServ |
| 20:33.52 | kpfleming | cresl1n: !!! |
| 20:33.59 | cresl1n | Hey all!!!!! |
| 20:34.00 | cresl1n | :-) |
| 20:34.14 | file | woah |
| 20:34.20 | kpfleming | i know |
| 20:34.24 | kpfleming | it's like 2009 in here |
| 20:34.53 | cresl1n | checks his ntp setings |
| 20:36.04 | bklang | kpfleming: so you're thinking I need to set MASQUERADE on the second channel before the first thread gets back to it? |
| 20:36.19 | kpfleming | i'm not really sure |
| 20:36.30 | kpfleming | i'd have to investigate what the Redirect action is really doing |
| 20:36.44 | kpfleming | or... someone with the time available would have to do it, i guess i should say :-) |
| 20:36.53 | bklang | ok. I guess I'll go ahead and open the issue in JIRA and keep chasing it |
| 20:38.34 | file | cresl1n, I'm in ur box messin' with ur clock! |
| 20:38.56 | cresl1n | jumps |
| 20:38.58 | cresl1n | ah! |
| 20:40.12 | bklang | Clock skew detected! |
| 21:02.09 | *** join/#asterisk-dev luckman212 (~irc@pool-108-41-8-176.nycmny.fios.verizon.net) |
| 21:08.19 | bklang | kpfleming: is there an obvious place you can think of where a new thread would be create to handle the splitting channel? I must be missing something in the code |
| 21:09.35 | kpfleming | well, it's probably calling ast_async_goto() or something similar to that, which would handle the mechanics of threads |
| 21:10.39 | bklang | ah, yes it is |
| 21:11.15 | bklang | so if (chan->pbx) indicates that the channel already has a thread servicing it? |
| 21:12.14 | kpfleming | yep |
| 21:21.13 | bklang | Ah, I think I have it! |
| 21:21.30 | bklang | The first channel's structure is updated by ast_softhangup_nolock before the second thread copies the structure |
| 21:21.36 | bklang | and so it inherits the ZOMBIE flag |
| 21:21.51 | bklang | but it shouldn't get that flag |
| 21:22.18 | mvanbaak | ZOMBIE |
| 21:22.21 | mvanbaak | <--- ZOMBIE |
| 21:22.39 | kpfleming | that sounds promising, yes |
| 21:22.59 | mvanbaak | maybe I should not have got in the plane with only clothes for 36C and fly to holland |
| 21:23.12 | mvanbaak | I just returned from Brasil, where it was 36C |
| 21:23.23 | mvanbaak | and arrived in Holland where it was 4C |
| 21:23.37 | mvanbaak | with only my shorts, flipflops and threadless shirt |
| 21:23.46 | mvanbaak | this was sunday |
| 21:23.52 | mvanbaak | and today I feel sick :( |
| 21:23.53 | kpfleming | not good |
| 21:52.52 | *** join/#asterisk-dev tzafrir (~tzafrir@bzq-218-155-147.cablep.bezeqint.net) |
| 21:52.53 | *** mode/#asterisk-dev [+o tzafrir] by ChanServ |
| 22:40.43 | *** part/#asterisk-dev mjordan (~mjordan@nat/digium/x-mlhscgzwiugeeljy) |
| 22:56.57 | *** join/#asterisk-dev angler (~angler@pdpc/sponsor/digium/angler) |
| 23:07.52 | *** part/#asterisk-dev cresl1n (~Adium@asterisk/libpri-and-libss7-expert/Cresl1n) |