| 00:18.59 | *** join/#asterisk-dev Synthase_ (uid63346@gateway/web/irccloud.com/x-pnvzbhuvovjbgtjk) |
| 00:38.54 | file | aw, no Corey |
| 00:40.52 | gtjoseph | i was looking for him as well :) |
| 00:53.08 | file | I'm working on Jenkins |
| 00:53.19 | file | theoretically as soon as I hit enter and something runs he'll have his periodic ref debug job |
| 00:53.34 | gtjoseph | cool |
| 00:53.53 | file | or not |
| 00:53.55 | file | huh |
| 00:54.45 | file | oh |
| 00:56.23 | file | there they are |
| 00:58.57 | file | that gets them there, but they aren't scheduled |
| 01:03.41 | file | I changed the server that the Jenkins script use for git away from gerrit, I'm going to let it run in that config over the weekend and see how it works |
| 01:04.53 | file | is apparently the new mjordan |
| 01:15.02 | *** join/#asterisk-dev craigify (~craigify@ip184-186-18-123.no.no.cox.net) |
| 01:27.35 | file | and now published drafts will trigger the check pipeline |
| 01:27.52 | gtjoseph | really? why drafts? |
| 01:28.59 | file | oh, you probably misunderstand |
| 01:29.09 | gtjoseph | wouldn't be the first time |
| 01:29.17 | file | not the act of putting a review up as a draft |
| 01:29.23 | file | but hitting the publish button to turn it into a real review |
| 01:29.32 | gtjoseph | ah, gotcha |
| 01:29.40 | file | previously that didn't trigger the check pipeline, so I'd have to go in and do recheck |
| 01:29.58 | gtjoseph | that makes sense |
| 01:30.45 | file | now it should |
| 01:31.06 | file | and if my server change pans out, no more corrupted git clones and bad failed jobs |
| 01:31.27 | gtjoseph | wouldn't that be nice. :) |
| 01:32.04 | file | says the one who doesn't have to SSH to the Jenkins nodes and manually clean out workspaces |
| 01:32.25 | gtjoseph | :) |
| 01:33.02 | file | tomorrow I shall look at adding a periodic for the ref debug and cleaning out the corrupted workspaces I can find |
| 01:33.32 | file | but for right this moment I'm going to trigger the master ref debug job and see what happens |
| 01:33.35 | file | and brush my teeth. |
| 01:33.55 | gtjoseph | you could probably brush them twice while that job runs |
| 01:34.17 | file | it has to do a fresh git clone too |
| 01:35.50 | file | aha |
| 01:38.19 | file | ponders |
| 01:42.01 | file | and off it goes |
| 01:44.15 | file | doesn't look like menuselect got told to enable REF_DEBUG, so there's something to fix |
| 01:44.31 | gtjoseph | yeah, like all of menuselect |
| 01:44.50 | file | it's probably because I had to mess with the yaml |
| 01:45.01 | file | it failed validation, and I wanted to get my stuff going |
| 02:23.48 | file | enough mucking about with it tonight |
| 03:12.04 | *** join/#asterisk-dev coreyfarrell (~coreyfarr@wsip-98-179-138-226.ri.ri.cox.net) |
| 03:43.18 | *** join/#asterisk-dev Juggie (~Juggie@unaffiliated/juggie) |
| 13:28.22 | *** join/#asterisk-dev slavon (5bf65cda@gateway/web/freenode/ip.91.246.92.218) |
| 13:29.43 | *** join/#asterisk-dev coreyfarrell (~coreyfarr@wsip-98-179-138-226.ri.ri.cox.net) |
| 13:40.56 | slavon | Hello @dev. Have many leak logs. Intasting? Or i must stop spam jira?)) |
| 13:45.43 | slavon | Did you run tests? Simple run testsute + asan found 10+ bugs and many memory leaks places ) |
| 13:48.09 | *** join/#asterisk-dev file (~file@asterisk/developer-and-muffin-lover/file) |
| 13:48.09 | *** mode/#asterisk-dev [+o file] by ChanServ |
| 13:50.48 | *** join/#asterisk-dev dar123 (~bad53@107-203-254-31.lightspeed.sntcca.sbcglobal.net) |
| 15:53.36 | *** join/#asterisk-dev [TK]D-Fender (~joe@64.235.216.2) |
| 16:24.33 | *** join/#asterisk-dev file (~file@asterisk/developer-and-muffin-lover/file) |
| 16:24.34 | *** mode/#asterisk-dev [+o file] by ChanServ |
| 17:00.21 | file | coreyfarrell, I've got the REF_DEBUG jobs working |
| 17:02.44 | coreyfarrell | file: it says that run #2 passed all tests, I'm guessing you changed something for run #3? |
| 17:02.50 | file | yes |
| 17:03.34 | file | I don't have it scheduled yet, just got the jobs working |
| 17:03.46 | coreyfarrell | from the console output: asterisk:243 errReceived: Asterisk 127.0.0.1 received error: FD 32766 exceeds the maximum size of ast_fdset! |
| 17:04.17 | file | that's happening for the other jobs too |
| 17:06.38 | coreyfarrell | well I'm about to bring some friends to the airport, when I get back I'll look at the logs. looks like every test has REF_DEBUG leaks so I'll have to get that fixed before we bother scheduling. |
| 17:07.41 | coreyfarrell | btw thanks for helping with this |
| 17:07.53 | file | yw! |
| 17:08.05 | file | hopefully by the end of the weekend the infrastructure stuff should be... good |
| 17:08.14 | file | and it'll just be tests really |
| 17:11.27 | coreyfarrell | oh does this new infrastructure have the ability to always keep at least 1 copy of the artifacts from ref_debug tests? |
| 17:12.22 | file | I don't know if we're getting rid of artifacts at all currently |
| 17:12.41 | file | LET'S SEE |
| 17:12.56 | coreyfarrell | in bamboo the ref-debug ran weekly the results were only kept for like 3 days.. so more than half the week no artifacts were available |
| 17:14.23 | file | yeah we're not purging currently |
| 17:15.17 | file | but when we get to that point I'll make sure it's kept for awhile |
| 17:18.36 | coreyfarrell | cool thanks. have to go, thanks again |
| 17:18.43 | file | cheers |
| 18:27.39 | gtjoseph | coreyfarrell: figured out the loader issue (but not a fix) |
| 18:28.01 | gtjoseph | the entire module LOAD_PRIORITY thins is a misnomer |
| 18:28.27 | gtjoseph | priority only controls the order that 'load_module' is called, not the actual dlopen |
| 19:20.08 | coreyfarrell | gtjoseph: ok so if it exports globals it's first, but no other ordering after that. for modules that do not export globals the dlopen order does not matter.. |
| 19:21.09 | gtjoseph | correct but... |
| 19:21.36 | gtjoseph | to find load order, all the module shave to be dlopened once |
| 19:21.57 | gtjoseph | so they're done in simple directly order |
| 19:22.19 | gtjoseph | i.e. opendir. |
| 19:22.51 | gtjoseph | so res_pjsip_session happens to be opened before res_pjproject and poof. |
| 19:23.52 | coreyfarrell | does res_pjsip_session not use symbols from res_pjproject? |
| 19:23.57 | gtjoseph | even though LAZY is defined, the issue is with things like (&pjsip_resister_method) |
| 19:24.05 | gtjoseph | yes it does |
| 19:24.15 | gtjoseph | that's the point |
| 19:25.20 | gtjoseph | res_pjsip_session isn't linked against res_pjproject though so things like addresses of functions are undefined |
| 19:25.53 | coreyfarrell | gtjoseph: so in load_dynamic_module: if (!(lib = dlopen(fn, wants_global ? RTLD_LAZY | RTLD_GLOBAL : RTLD_NOW | RTLD_LOCAL))) { |
| 19:25.53 | gtjoseph | so even though function CALLS are deferred, references to function addresses aren't |
| 19:26.05 | coreyfarrell | what happens if you replace RTLD_LAZY with RTLD_NOW? |
| 19:26.37 | gtjoseph | can't work because res_pjproject isn't there yet. it will fail even more spectacularly |
| 19:27.12 | coreyfarrell | would probably need to remove the LOG_ERROR on the next line, but the caller of load_dynamic_module already ignores errors during global module loads. |
| 19:27.40 | gtjoseph | yes it does, but it also doesn't retry |
| 19:28.35 | coreyfarrell | weird I thought there was some kind of retry when that happened |
| 19:29.23 | gtjoseph | not for global modules and res_pjsip_session exports symbols of its own |
| 19:30.09 | gtjoseph | at least i understand the mechanism now |
| 19:30.20 | gtjoseph | and there are several options |
| 19:30.48 | gtjoseph | i could... allow global modules to retry... |
| 19:31.25 | gtjoseph | create a new flag AST_MODFLAG_PRELOAD which does what preload in modules.conf des |
| 19:32.08 | gtjoseph | find a way to link the pjsip modules to res_pjsip modules that doesn't cause a loop or require an rpath |
| 19:32.27 | gtjoseph | s/ res_pjsip modules / res_pjproject / |
| 19:32.59 | gtjoseph | i'm thinking option 1 |
| 19:33.21 | gtjoseph | just because you export global symbold doesn't mean you shouldn't retry |
| 19:34.01 | coreyfarrell | yeah I don't like option 2 or 3, option 1 is the most generic fix so it might deal with other similar issues in the future |
| 19:34.10 | gtjoseph | yep |
| 19:34.40 | gtjoseph | i may change the logging so it only logs on the last retry |
| 19:35.11 | coreyfarrell | what do you mean? |
| 19:35.49 | gtjoseph | so they don't spit out errors when they're going to be retried anyway |
| 19:36.52 | coreyfarrell | oh you mean the warning after the final dlopen of load_dynamic_module? I would do: if (!global_symbols_only) { ast_log.. } |
| 19:37.32 | gtjoseph | yep |
| 19:38.39 | coreyfarrell | I think this does have a side-effect though. |
| 19:38.54 | coreyfarrell | i'm not sure it'll be ok for 13 |
| 19:39.39 | gtjoseph | the retry or the logging? |
| 19:40.14 | coreyfarrell | if a module currently exports symbols it can have references to symbols in another module. if those symbols are only used if enabled in config then it's like an optional dependency. |
| 19:40.58 | gtjoseph | not sure i follow |
| 19:41.14 | coreyfarrell | so you have module_1 and module_2, both export globals. module_2 has code: if (config_option_enabled) { run_something_from_module_1(); } |
| 19:41.41 | gtjoseph | yeah |
| 19:41.50 | coreyfarrell | if !config_option_enabled doesn't RTLD_LAZY effectively make module_1 not required for module_2 to load and run? |
| 19:42.28 | gtjoseph | kinda |
| 19:43.35 | gtjoseph | LAZY only gets triggered on calls but it doesn't get called on references so if module 2 references &module_1_function, |
| 19:43.48 | gtjoseph | then the dlopen is going to fail |
| 19:44.07 | gtjoseph | so just because 2 didn't call module1_function, it still needs it |
| 19:44.46 | gtjoseph | those are rare cases of course |
| 19:45.04 | gtjoseph | and i'm not sure how a retry will alter the behavior negatively |
| 19:50.14 | coreyfarrell | hmm so the modules are dlopen'ed in random order, then the global exporters are loaded in order of priority? |
| 19:51.23 | gtjoseph | so the sequence is... |
| 19:51.46 | gtjoseph | dlopen all global modules in random order to find their order... |
| 19:51.58 | coreyfarrell | what if we re-dlopen the module when it's time to load_module? |
| 19:52.39 | gtjoseph | well, if it fails the first dlopen, it's removed from the list |
| 19:52.40 | coreyfarrell | ie when it hits the order in priority, run a dlopen then dlclose to balance the module references. |
| 19:52.52 | gtjoseph | and we can't get it's requested order without dlopeningf it |
| 19:53.20 | gtjoseph | we can't even get it's priority without being able to dlopen it |
| 19:53.29 | coreyfarrell | oh and you can't dlopen even LAZY in certain cases? |
| 19:53.58 | gtjoseph | correct because LAZY doesn't work on addresses of things |
| 19:54.01 | gtjoseph | only calls |
| 19:54.30 | coreyfarrell | ok so from module_2: do_something(&module_1_callback) - this does not work.. |
| 19:54.53 | gtjoseph | correct. that MUST be resolced with dlopen |
| 19:54.56 | gtjoseph | resolved |
| 19:55.10 | gtjoseph | if module 2 is linked with module 1 it works |
| 19:55.16 | coreyfarrell | so my best idea is a little ugly and is a hack: do_something(module_1_get_callback()); |
| 19:55.24 | gtjoseph | because that address is resolved at link time |
| 19:56.05 | gtjoseph | actually, i'm thinking of just retraversing the list |
| 19:56.44 | gtjoseph | maybe try twice then remove the module if it doesn't load |
| 19:57.42 | coreyfarrell | I feel like trying to fix this in the module loader is difficult for 13 |
| 19:58.03 | gtjoseph | so in load_resource_list, if global_symbols==1, re-traverse twice |
| 19:58.05 | gtjoseph | why? |
| 19:58.22 | gtjoseph | it can onlt improve the load process unless you have a specific case in mind |
| 19:58.26 | gtjoseph | only |
| 19:59.43 | gtjoseph | i'm off to lunch. i'll check in with you later |
| 19:59.52 | coreyfarrell | not specifically I'm just concerned that it changes the behavior of something that has a pretty big effect |
| 20:01.50 | coreyfarrell | something that can have a big effect I meant. working around it from pjsip modules is a self contained change vs nothing in the module loader is self contained |
| 20:27.56 | coreyfarrell | file: looks like every test in the REF_DEBUG job is failing. I'm guessing the build agent has some module that i'm missing. if we stop it now will it allow us to look at the results? |
| 21:41.31 | *** join/#asterisk-dev SirLouen (~pickupdog@81.61.195.121.dyn.user.ono.com) |