| 01:57.00 | *** join/#asterisk-dev pchero (~pchero@211.178.226.108) |
| 01:57.01 | *** mode/#asterisk-dev [+o pchero] by ChanServ |
| 03:52.07 | *** join/#asterisk-dev Gugge_ (gugge@guggemand.dk) |
| 03:57.03 | *** join/#asterisk-dev electronic_eel_ (~quassel@electroniceel.org) |
| 04:01.00 | *** join/#asterisk-dev leifmadsen (~leifmadse@asterisk/documenteur-extraordinaire/blitzrage) |
| 04:01.01 | *** mode/#asterisk-dev [+o leifmadsen] by ChanServ |
| 05:37.50 | *** join/#asterisk-dev pchero (~pchero@211.178.226.108) |
| 05:37.50 | *** mode/#asterisk-dev [+o pchero] by ChanServ |
| 05:56.21 | jkroon | seanbright, so what's the correct semantics for ao2_replace? |
| 06:09.00 | jkroon | i think it's fine, effectively you're doing dst = src; ... so references to original dst should decrease, and src should increase, so makes sense to me at least. |
| 06:14.31 | jkroon | # /var/lib/asterisk/scripts/refcounter.py -f /var/log/asterisk/refs -n > /tmp/refs.txt |
| 06:14.31 | jkroon | Killed |
| 06:15.39 | jkroon | is there some "quick fix" or am I down to splitting the log file into smaller chunks based on the actual pointers and then sub-processing? |
| 13:12.12 | *** join/#asterisk-dev sduthil (~sduthil@198-48-220-170.cpe.pppoe.ca) |
| 14:38.40 | seanbright | jkroon: well if you think it's fine then it is |
| 14:39.15 | jkroon | seanbright, would like to understand if I'm missing something. |
| 14:39.36 | seanbright | i already shared my thoughts |
| 14:39.45 | seanbright | you can read them up there ^^ |
| 14:39.55 | seanbright | you just don't agree with them, and that is fine |
| 14:57.26 | file | it took me this morning and asking gtjoseph for help, but I seem to have MODULEINFO from files going into the core XML doc |
| 14:58.24 | gtjoseph | so _that's_ what you were doing |
| 14:58.34 | file | yes |
| 15:06.04 | *** join/#asterisk-dev kharwell (uid358942@gateway/web/irccloud.com/x-htoohyvnogqcnvwv) |
| 15:06.04 | *** mode/#asterisk-dev [+o kharwell] by ChanServ |
| 15:06.59 | seanbright | jkroon: i apologize. woke up on the wrong side of the bed this morning. |
| 15:07.59 | jkroon | seanbright, no worries, i was honestly just trying to better understand why it's confusing / what's confusing, and I have more than my share of those days as well. hope the day gets better. |
| 15:10.12 | seanbright | in my mind i am thinking if it like "take ownership of this reference, releasing the old" |
| 15:10.46 | seanbright | i should say that is how i thought about it. now seeing how it actually works i understand what it is doing, but it doesn't do what i want it to do. |
| 15:11.26 | jkroon | that's fair |
| 15:22.03 | file | now that I'm not talking to cresl1n... |
| 15:22.17 | file | gtjoseph: yeah I've been poking at the module deprecation stuff here and there, and wanted to get moduleinfo into core XML so it can be used internally for things |
| 15:22.42 | gtjoseph | +1 |
| 15:22.52 | file | so we can add a "deprecated_in" or something to the MODULEINFO, then use it in menuselect and the module loader |
| 15:23.13 | file | also means we could use MODULEINFO as the definitive support level source for a module |
| 15:23.43 | gtjoseph | YAY! |
| 15:24.18 | file | and the module loader could spit out a "You have loaded 'blah', this module is deprecated and has been replaced by 'yay'" |
| 15:24.51 | file | lots'o'cool stuff |
| 15:25.08 | gtjoseph | very nice! |
| 15:29.10 | *** join/#asterisk-dev bford (uid283514@gateway/web/irccloud.com/x-ymwmfpcfsnmzeima) |
| 15:29.10 | *** mode/#asterisk-dev [+o bford] by ChanServ |
| 16:46.46 | file | ah Linkedin, a source of entertainment for myself with various people trying to sell me stuff or show me whitepapers to then try to sell me stuff |
| 17:06.16 | jkroon | that is well phrased. |
| 17:07.17 | jkroon | so, https://wiki.asterisk.org/wiki/display/AST/Reference+Count+Debugging - under normal conditions I'm betting the output should be somewhat less than a 130MB output file? |
| 17:07.30 | jkroon | (not the raw log, the generated refs.txt) |
| 17:11.35 | jkroon | grep constructor refs.txt | sort | uniq -c | sort -g |
| 17:11.41 | jkroon | <PROTECTED> |
| 17:16.29 | jkroon | right, so is there someone that would care to sanity check me but I believe __ast_format_cap_append will always leak a ref here ... |
| 17:18.48 | jkroon | it creates a framed ... (ref=1), passes that to format_cap_framed_init which will oa2_ref(-1) it on filure paths, but +1 it on success paths ... resulting in ref count=2, and then when unlinked from the list will drop to =1 and end up leaking. |
| 17:26.49 | *** join/#asterisk-dev pchero (~pchero@211.178.226.108) |
| 17:26.49 | *** mode/#asterisk-dev [+o pchero] by ChanServ |
| 17:30.19 | jkroon | is it possible to gate a review through the test suite whilst it's still a wip? |
| 17:40.48 | jkroon | https://community.asterisk.org/t/memory-leak-consumes-16gb-in-less-then-24-hours/71916/18 ... so I'm not the only one. it just seems we're hitting it hard and fast. |
| 17:43.34 | jkroon | https://jenkins2.asterisk.org/job/Asterisk%20Checks/job/master/170/console: ERROR: No test report files were found. Configuration error? Finished: FAILURE - something I broke? |
| 18:10.27 | seanbright | jkroon: line 163: /* This takes the allocation reference */ |
| 18:10.57 | jkroon | bad comment ... suggestions? |
| 18:11.22 | seanbright | bad comment? that is in the existing code |
| 18:11.44 | seanbright | i assume it means the vector is taking ownership of the framed ref? |
| 18:11.51 | jkroon | sorry, i've not slept enough in the last 48h, let me look at that for a moment. |
| 18:12.05 | kharwell | that's what it appears to be doing. When it's removed from the vector it looks like it removes the added reference |
| 18:12.56 | jkroon | hmm, if that's true my patch is badly wrong. |
| 18:13.30 | kharwell | it's removed "ast_format_cap_remove" |
| 18:14.11 | jkroon | [32758] format_cap.c:205 __ast_format_cap_append: +1 - [**constructor**] |
| 18:14.11 | jkroon | [32758] format_cap.c:170 format_cap_framed_init: +1 - [1] |
| 18:14.11 | jkroon | [32758] format_cap.c:80 format_cap_destroy: -1 - [2] |
| 18:14.18 | jkroon | well, that's the leak. |
| 18:14.30 | jkroon | getting about 1k of those per minute on the last run. |
| 18:18.51 | seanbright | ok, so preference_order should own 1 ref and format_cap_framed_list should own 1 |
| 18:19.02 | kharwell | appears there are two lists (cap->preference_order and cap->formats) that each hold a reference. Both appear to remove the reference in format_cap_destroy |
| 18:19.11 | kharwell | seanbright: yeah that's what I'm seeing too. |
| 18:19.21 | jkroon | seanbright, is on it, AST_VECTOR_FREE is called, but those references are never released. |
| 18:19.30 | kharwell | looks like maybe the cap->preference_order is not get removed? |
| 18:19.57 | jkroon | no wait, thatlooks like it should clear both ... |
| 18:21.01 | seanbright | i would add some logging in the line 85 for loop |
| 18:21.16 | seanbright | although it is pretty straight forward... |
| 18:21.45 | seanbright | actually i wonder if that could be replaced with AST_VECTOR_CLEANUP (or whatever it is called) |
| 18:21.51 | seanbright | but that is neither here nor there |
| 18:22.00 | file | then where is it? |
| 18:23.06 | seanbright | points |
| 18:23.32 | file | oh, makes sense |
| 18:24.03 | kharwell | jkroon: I'd also check the result of INSERT_HEAD (~ln1 171) in format_cap_framed_init right after the ref gets bumped. Maybe it is failing to insert it for some reason |
| 18:24.22 | seanbright | good idea |
| 18:24.23 | kharwell | nvm appears there is no result for that macro |
| 18:24.27 | jkroon | kharwell, that makes sense. |
| 18:24.28 | seanbright | or not |
| 18:24.44 | seanbright | i would print the state of the world in format_cap_destroy |
| 18:24.50 | seanbright | for debugging purposes |
| 18:25.02 | seanbright | and then i would find and fix the leak |
| 18:25.14 | seanbright | then i would return to troy victorious |
| 18:25.39 | jkroon | is AST_LIST_ and AST_VECTOR_ interchangeable? |
| 18:26.17 | jkroon | sorry, these are new constructs for me :$ |
| 18:26.45 | seanbright | they are not |
| 18:26.50 | seanbright | AST_LIST is a linked list (bad) |
| 18:27.01 | seanbright | AST_VECTOR is a resizable array (less bad) |
| 18:27.07 | file | bad is a stretch, it can be fine |
| 18:27.32 | jkroon | I'm seeing references to both AST_LIST_ as well as AST_VECTOR_ in both construction and the dtor functions. |
| 18:27.45 | seanbright | yes. there is a linked list and a vector in use |
| 18:28.27 | seanbright | bad for cache coherence. is that better? |
| 18:29.42 | seanbright | jkroon: what version of asterisk is this? no modifications or patches or anything? |
| 18:30.45 | jkroon | 16 |
| 18:31.06 | seanbright | latest from git? no patches? |
| 18:32.12 | seanbright | what do i want to turn on? REF_DEBUG or AO2_DEBUG? |
| 18:32.15 | jkroon | 16.15.2 just confirming. there is a few patches, specifically for the build system to not intsall /var/run/asterisk, the lua patches that's now in tree, and the func_lock fixes that I submitted a while bck. |
| 18:33.11 | jkroon | REF_DEBUG only affects the default setting ref asterisk.conf [options] refdebug - so just set that = yes. |
| 18:34.21 | jkroon | ok, i've had like 4 hours of sleep in the last 38 hours ... so I'm happy to continue fighting with this in the AM, but right now I unfortunately do need to get some sleep. |
| 18:34.44 | seanbright | god speed |
| 18:36.27 | jkroon | thanks, if you do manage to pick something up per chance ... just leave a message here i'll pick up on it in the morning. |
| 18:37.01 | seanbright | k |
| 18:38.09 | seanbright | my refs output looks different than yours |
| 18:39.01 | jkroon | you using channel origination? |
| 18:39.17 | jkroon | via AMI ... |
| 18:39.22 | seanbright | ah, you're using refcounter.py |
| 18:39.26 | jkroon | yes |
| 18:39.31 | seanbright | would be hilarious if there was a bug in that... |
| 18:39.43 | file | you aren't using MuteAudio, right? |
| 18:39.49 | jkroon | I'm not no. |
| 18:39.57 | file | carries on |
| 18:40.30 | seanbright | ok, so originate via AMI |
| 18:40.32 | seanbright | got it |
| 18:41.01 | jkroon | cheers, and thank you for looking at this as well. |
| 18:41.05 | seanbright | sure |
| 19:03.42 | seanbright | not able to reproduce |
| 19:18.44 | seanbright | jkroon: could not reproduce. would need to know specifics of your test setup. if you don't get it figured out tomorrow morning i can take another look |
| 21:17.23 | seanbright | my builds are not the only thing about me that are unstable |
| 21:56.41 | *** join/#asterisk-dev electronic_eel (~quassel@electroniceel.org) |
| 23:44.32 | *** join/#asterisk-dev Cyrillax (~Cyrillax@unaffiliated/cyrillax) |