| 00:29.36 | *** join/#edev jayabharath (~jayabhara@nat/ti/x-ozuuhunkjuemkazw) |
| 01:01.03 | *** join/#edev n17ikh (~n17ikh@c-68-59-25-51.hsd1.sc.comcast.net) |
| 01:03.07 | *** join/#edev n17ikh (~n17ikh@c-68-59-25-51.hsd1.sc.comcast.net) |
| 01:06.35 | *** join/#edev n17ikh (~n17ikh@c-68-59-25-51.hsd1.sc.comcast.net) |
| 01:51.04 | *** join/#edev jayabharath (~jayabhara@nat/ti/x-ahltscctpbwjmice) |
| 02:06.17 | *** join/#edev Crofton (~balister@pool-96-240-180-126.ronkva.east.verizon.net) |
| 02:30.46 | *** join/#edev kergoth_ (~kergoth@ip24-251-170-95.ph.ph.cox.net) |
| 04:13.55 | *** join/#edev mbuf (~user@118.102.130.34) |
| 04:48.14 | *** join/#edev JoeLlama (~snork@unaffiliated/joellama) |
| 05:13.14 | *** join/#edev nullpuppy (~dustin@freematrix/staff/nullpuppy) |
| 05:52.16 | *** join/#edev ja2 (~marcus@81-233-153-250-no83.tbcn.telia.com) |
| 06:44.48 | *** join/#edev harpal (~harpal@121.246.75.165) |
| 08:15.06 | *** join/#edev NullMoogleCable (~fake@cpe-69-205-123-203.rochester.res.rr.com) |
| 08:33.11 | *** join/#edev NullMoogleCable (~fake@cpe-69-205-123-203.rochester.res.rr.com) |
| 09:16.30 | *** join/#edev florian_kc (~fuchs@Maemo/community/contributor/florian) |
| 09:24.18 | *** join/#edev sassan (~vadmin@cvs.mpc-ogw.co.uk) |
| 09:39.12 | *** join/#edev sassan (~vadmin@cvs.mpc-ogw.co.uk) |
| 09:53.37 | *** join/#edev sassan (~vadmin@cvs.mpc-ogw.co.uk) |
| 10:11.26 | *** join/#edev pirho (debian-tor@gateway/tor-sasl/pirho) |
| 10:49.02 | *** join/#edev madsy_ (~madsy@ti0207a340-0132.bb.online.no) |
| 10:59.52 | *** join/#edev Crofton (~balister@pool-96-240-180-126.ronkva.east.verizon.net) |
| 11:10.12 | *** join/#edev prpplague (~danders@65.48.240.51) |
| 11:10.21 | *** mode/#edev [+o prpplague] by ChanServ |
| 11:57.28 | *** join/#edev sassan (~vadmin@cvs.mpc-ogw.co.uk) |
| 12:03.57 | *** join/#edev gustavoz (~gustavoz@host57.190-30-11.telecom.net.ar) |
| 12:27.18 | *** join/#edev calculus (~calculus@gentoo/user/calculus) |
| 12:37.24 | *** join/#edev sassan (~vadmin@cvs.mpc-ogw.co.uk) |
| 12:55.27 | *** join/#edev sassan (~vadmin@cvs.mpc-ogw.co.uk) |
| 13:04.18 | *** join/#edev sassan (~vadmin@cvs.mpc-ogw.co.uk) |
| 13:16.47 | *** join/#edev sassan (~vadmin@cvs.mpc-ogw.co.uk) |
| 13:28.11 | *** join/#edev sassan (~vadmin@81.2.99.171) |
| 13:46.02 | *** join/#edev sassan (~vadmin@cvs.mpc-ogw.co.uk) |
| 13:56.46 | *** join/#edev mbuf (~mbuf@124.123.187.59) |
| 14:08.42 | *** join/#edev jayabharath (~jayabhara@nat/ti/x-jnzxzqtsfymlscsg) |
| 14:10.57 | *** join/#edev avinashhm (~avinash-h@192.163.20.231) |
| 14:16.57 | *** join/#edev sassan (~vadmin@cvs.mpc-ogw.co.uk) |
| 14:39.20 | *** join/#edev madsy (~madsy@fu/coder/madsy) |
| 14:43.03 | *** join/#edev sassan (~vadmin@cvs.mpc-ogw.co.uk) |
| 14:52.15 | *** join/#edev sassan (~vadmin@cvs.mpc-ogw.co.uk) |
| 15:08.48 | *** join/#edev sassan (~vadmin@cvs.mpc-ogw.co.uk) |
| 15:22.09 | *** join/#edev sndcrb (~sndcrb@88.194.0.144) |
| 15:48.50 | *** join/#edev sassan (~vadmin@81.2.99.171) |
| 16:11.18 | *** join/#edev jayabharath (~jayabhara@nat/ti/x-hzvefzpajwvcsqzj) |
| 16:54.47 | *** join/#edev miltonm (~miltonm@32.97.110.53) |
| 16:54.54 | *** join/#edev n900evil (~Speedevil@tor/regular/SpeedEvil) |
| 17:26.38 | *** join/#edev JoeLlama (~snork@unaffiliated/joellama) |
| 17:33.06 | *** join/#edev avinashhm (~avinash-h@192.163.20.231) |
| 18:23.01 | *** join/#edev wmat (~btraynor@gromit.mixdown.ca) |
| 18:24.19 | *** join/#edev NightMonkey (~NightMonk@pdpc/supporter/professional/nightmonkey) |
| 18:34.04 | *** join/#edev avinashhm (~avinash-h@192.163.20.231) |
| 18:38.34 | *** join/#edev aisa (~aisa@c-68-35-167-179.hsd1.nm.comcast.net) |
| 18:39.08 | aisa | hello everyone. |
| 18:39.20 | aisa | I've recently ported a python program to the NanoNote, an embedded device. |
| 18:39.32 | aisa | The program is running slowly, too slowly to use. |
| 18:39.47 | aisa | Because it is a python program, I was able to rapidly generate profiling information and see which areas of the code I can improve. |
| 18:39.48 | SpeedEvil | So profile it. |
| 18:39.51 | SpeedEvil | ah |
| 18:39.57 | aisa | I'm worried after profiling it, that it may still not be fast enough. |
| 18:39.58 | aisa | so, |
| 18:40.09 | aisa | I really like that I could quickly debug this problem, |
| 18:40.17 | aisa | I'm really concerned that it won't be fast enough. |
| 18:40.36 | aisa | I'd like some perspective on this, what other people's experience is on the trade-off using interpreted languages. |
| 18:40.39 | SpeedEvil | Well - youcould always recode it in something faster. |
| 18:40.43 | SpeedEvil | It depends. |
| 18:40.46 | aisa | I could, yes. |
| 18:40.52 | aisa | What does it depend on? |
| 18:40.55 | SpeedEvil | If you're doing a simple GUI thing, then it doesn't matter. |
| 18:41.08 | SpeedEvil | If you're popping up simple dialog boxes, speed is not an issue. |
| 18:41.11 | aisa | this is a data-heavy application. |
| 18:41.22 | aisa | so it is a search application across a reasonably large dataset. |
| 18:41.28 | SpeedEvil | If you're doing FFTs in python - you probably really need to reconsider that. |
| 18:41.35 | SpeedEvil | Howembedded. |
| 18:41.46 | aisa | reasonably large in this case is a 1.5mb python pickle, |
| 18:41.55 | aisa | which expands to several times that size in memory. |
| 18:42.01 | aisa | ~50% of the available memory on the machine. |
| 18:42.05 | SpeedEvil | you're not swapping? |
| 18:42.18 | aisa | I don't have swap on the device yet. |
| 18:42.26 | aisa | I am able to create a swapfile, of course. |
| 18:42.44 | SpeedEvil | Is the program under memory pressure? |
| 18:42.52 | aisa | 32mb DRAM 336 MHz XBurst processor. |
| 18:43.05 | aisa | What do you mean? |
| 18:43.05 | SpeedEvil | I'm not sure if python will slow down in that case. |
| 18:43.14 | aisa | There is not OOMkiller after it, if that is what you are asking. |
| 18:43.23 | aisa | it does fit in memory. |
| 18:43.55 | SpeedEvil | I mean - sometimes if a system runs very low in RAM, it though it does not OOMkill, it drops stuff from RAM that it can page back in from executables. |
| 18:44.00 | SpeedEvil | This makes performance tank. |
| 18:44.08 | aisa | ah, how do I measure this? |
| 18:44.24 | SpeedEvil | Do you have free RAM, does the performance change if you add swap |
| 18:44.45 | SpeedEvil | Is free ram lower than... |
| 18:45.39 | SpeedEvil | <PROTECTED> |
| 18:46.02 | SpeedEvil | This indicates that the kernel is doing aggressive things to try to optimise. |
| 18:46.11 | SpeedEvil | Also - strace -c may help. |
| 18:46.21 | SpeedEvil | To try to free RAM. |
| 18:46.24 | aisa | what calls am I looking for in strace? |
| 18:46.33 | aisa | memory unmaps? |
| 18:46.42 | SpeedEvil | Also - what is the actual bit that's doing lots of work? |
| 18:46.59 | aisa | the most expensive part is reading the pickle file off disk. |
| 18:47.01 | aisa | I can fix that. |
| 18:47.07 | aisa | then I have a bunch of very simple regexes, |
| 18:47.11 | aisa | that are called a whole bunch of times. |
| 18:47.12 | SpeedEvil | Might you be able to offload it to mysql or something |
| 18:47.20 | aisa | I'm guessing I'll have to, yes. |
| 18:47.29 | aisa | I believe that the application essentially reads the whole table anyway, |
| 18:47.50 | aisa | but it is possible that a) sql will be faster at that and b) I could avoid doing it. |
| 18:48.10 | aisa | b) would limit functionality, but it is what I'll do if I can't get performance to be acceptable any other way. |
| 18:48.12 | SpeedEvil | what's the app doing - if youcan reveal that. |
| 18:48.19 | aisa | oh, totally: |
| 18:48.27 | aisa | http://lists.en.qi-hardware.com/pipermail/discussion/2010-September/005469.html |
| 18:48.34 | aisa | it is a dictionary program for the language Lojban. |
| 18:49.15 | aisa | it is used to look up words either by English or Lojban, and the English=>Lojban translation tries an exhaustive search. |
| 18:50.57 | aisa | This application performs fine on a desktop, I doubt it has ever been run on hardward this constrained. |
| 18:51.07 | aisa | s/hardward/hardware/ |
| 18:51.08 | SpeedEvil | ah |
| 18:51.22 | aisa | s/purl/gurl/ |
| 18:51.34 | aisa | sorry, is purl a bot or helpful? |
| 18:51.44 | Mirell | Purl is a bot. |
| 18:52.33 | SpeedEvil | Fundamentally - you need to look at your code. |
| 18:52.49 | SpeedEvil | Work out if you can reorder the task to be more efficient. |
| 18:52.58 | aisa | indeed, my original question was this: I'll look at this code, I'll optimize this code, but I've never tried doing anything like this with Python. |
| 18:52.59 | SpeedEvil | Or make o(n) algorithms o(n-1) |
| 18:53.06 | aisa | Do I have any chance at all of making it perform well? |
| 18:53.17 | SpeedEvil | Well - that depends. |
| 18:53.18 | aisa | How goes python on embedded machines, for anyone who has done it. |
| 18:53.30 | aisa | What does it depend on? |
| 18:53.36 | Mirell | I've done Ruby on embedded platforms. |
| 18:53.40 | Mirell | And it's horrible. |
| 18:53.51 | SpeedEvil | If the issue is the algorithm isn't properly optimised - then you might easily get a factor of a hundred speedup. |
| 18:53.53 | aisa | Can you articulate that? What didn't work very well? |
| 18:54.15 | SpeedEvil | Simply if nobodies looked at it from a performance aspect. Even without going to other languages than python. |
| 18:54.30 | Mirell | The garbage collector in Ruby is not particular optimized. |
| 18:54.36 | SpeedEvil | Often perople write code to be logical and amintainable. |
| 18:54.40 | Mirell | And a lot of libraries are poorly written. |
| 18:54.41 | SpeedEvil | maintainable |
| 18:54.55 | SpeedEvil | And if the performance is adequate, they don't optimise fro CPU. |
| 18:55.28 | SpeedEvil | And yes - languages that don't require the programmer to explictly consider malloc() and free() can be hideously expensive. |
| 18:55.36 | Mirell | Yah. |
| 18:55.39 | SpeedEvil | Garbage collection can be a huge cost. |
| 18:55.52 | Mirell | For a few specialized things, I ended up writing small C extensions. |
| 18:56.01 | Mirell | That were 100x faster than doing it in Ruby. |
| 18:56.26 | SpeedEvil | Is your application for example using a temporary variable and freeing it and allocating it a hundred thousand times. |
| 18:56.50 | Mirell | Yah, keeping track of the heap is a lot of overhead when you're doing embedded. |
| 18:56.53 | aisa | Right, I'm thinking in this application I need to mark the big data structure as permanent, and then have a separate area for temporary objects. |
| 18:57.01 | aisa | not that I think python can do that... |
| 18:57.10 | aisa | but this is giving me a better understanding of what I'm dealing with. |
| 18:57.34 | SpeedEvil | A lot of the things you need to do when optimising are 'wrong' or 'unbeautiful' |
| 18:57.48 | Mirell | Breaking the OO model! |
| 18:57.52 | SpeedEvil | From the perspective of a nice clean high-level language implementation that looks pretty. |
| 18:58.21 | aisa | I'm afraid that the changes I'll be making won't be accepted upstream for that reason. |
| 18:58.22 | Mirell | I just wish Lua was retarded. |
| 18:58.24 | Mirell | Er, wasn't. |
| 18:59.37 | SpeedEvil | Thoughthis is not always the case. |
| 18:59.49 | SpeedEvil | Sometimes fairly minor refactoring can boost performance. |
| 19:00.07 | SpeedEvil | Stuff like changing searching from depth-first to breadth-first or stuff like that. |
| 19:00.31 | Mirell | I remember seeing someone write GiantArray.sortby{rand(GiantArray.size}[0] |
| 19:00.37 | SpeedEvil | Design choices that are easy to alter, and have no impact on the readability of the program but orders of magnitude in the execution speed. |
| 19:02.17 | aisa | This has been a very helpful conversation. Thank you both Mirell and SpeedEvil. |
| 19:02.24 | SpeedEvil | Good luck. |
| 19:02.41 | aisa | I'm going to first try to make the python faster, but give myself a time limit for effort. |
| 19:02.53 | aisa | if I find a magic bullet or three, I'll have saved a bunch of time. |
| 19:03.10 | aisa | and if not, then I'll redo it in C. |
| 19:03.12 | SpeedEvil | http://shed-skin.blogspot.com/ |
| 19:03.18 | SpeedEvil | and stuff like that |
| 19:03.20 | Mirell | A lot of the nice high-level languages don't like to deal with embedded targets. |
| 19:03.27 | SpeedEvil | I have just released Shed Skin 0.5, an optimizing (restricted) Python-to-C++ compiler. L |
| 19:03.30 | Mirell | "Oh, this is for that embedded shit" |
| 19:03.34 | Mirell | </Ulrich Drepper> |
| 19:03.49 | aisa | heh. |
| 19:06.43 | *** join/#edev jayabharath (~jayabhara@nat/ti/x-zqyrcwvqoasclzoo) |
| 19:06.53 | *** part/#edev aisa (~aisa@c-68-35-167-179.hsd1.nm.comcast.net) |
| 19:08.27 | mnemoc | wonders if there is someone more hated that ulrich drepper in the free software world |
| 19:10.52 | *** join/#edev NullMoogleCable (~fake@cpe-69-205-123-203.rochester.res.rr.com) |
| 19:11.42 | Mirell | The only thing really complex that glibc has had to deal with is re-entrant functions. |
| 19:11.48 | Mirell | Other than that...it's a C library. Woo. |
| 19:12.33 | mnemoc | :) |
| 19:13.27 | Mirell | Hurry up UPS, I want to give my laptop a hot shower. |
| 19:13.45 | Mirell | Going to do a full reinstall and triple-boot with a new 1TB 2.5" drive for my laptop. |
| 19:23.49 | *** join/#edev florian (~fuchs@Maemo/community/contributor/florian) |
| 19:41.18 | *** join/#edev CosmicPenguin (~nobody@rrcs-67-52-130-30.west.biz.rr.com) |
| 19:41.18 | *** mode/#edev [+o CosmicPenguin] by ChanServ |
| 20:02.59 | *** join/#edev gratz|home (~gratz@cpc3-brig15-2-0-cust237.3-3.cable.virginmedia.com) |
| 20:12.33 | *** join/#edev miknix (~miknix@gentoo/developer/miknix) |
| 20:56.25 | *** join/#edev calculus (~calculus@gentoo/user/calculus) |
| 21:11.01 | *** join/#edev calculus (~calculus@gentoo/user/calculus) |
| 21:19.16 | *** join/#edev joshc_ (~joshc@cpe-24-28-84-71.austin.res.rr.com) |
| 21:20.22 | joshc_ | hey guys, got a question for you. I've got an ARM kernel booting to busybox's getty, but when I try to log in, it kicks me back to getty |
| 21:20.46 | joshc_ | so, for some reason it can't spawn a shell? |
| 21:21.23 | joshc_ | (I'm booting directly into an initrd) |
| 21:22.24 | mnemoc | try straight /bin/sh instead of getty |
| 21:22.58 | joshc_ | mnemoc: woops, I forgot to mention I tried that, and it panics saying that it tried to kill init (so I'm assuming there is something wrong spawning the shell) |
| 21:23.34 | joshc_ | in the buildroot image, /etc/passwd has root's shell as being /bin/sh, which is a link to busybox |
| 21:24.40 | joshc_ | i'm going to give a full rebuild a try |
| 21:38.34 | *** join/#edev calculus (~calculus@gentoo/user/calculus) |
| 21:40.16 | mnemoc | joshc_: dev/ properly populated? devtmpfs? |
| 21:40.48 | mnemoc | it shouldn't affect anyway :< |
| 21:41.49 | mnemoc | if /bin/sh crashes don't expect anything useful from getty, try another bb shell |
| 21:43.37 | joshc_ | mnemoc: i'll give another shell a try if this rebuild fails |
| 21:44.14 | joshc_ | mnemoc: I'm assuming that /dev is probably populated, buildroot seems to have chosen a default list of nodes to create |
| 21:47.17 | mnemoc | you could test with qemu user mode, or been even uglier and run /bin/sh inside strace or gdb... |
| 21:48.04 | mnemoc | uhm, nope, init= needs a single arg and an script wont work if bin/sh doesn't |
| 21:48.09 | mnemoc | ignore the suggestion |
| 22:03.14 | *** join/#edev PierreLuc (~PierreLuc@ip-66-254-44-78.mqdsl.megaquebec.net) |
| 22:17.23 | *** join/#edev SpeedEvil (~user@tor/regular/SpeedEvil) |
| 22:35.14 | *** join/#edev SpeedEvil (1000@tor/regular/SpeedEvil) |
| 22:51.33 | joshc_ | hmm, is buildroot supposed to create the busybox symlinks in the build image? |
| 23:08.30 | *** join/#edev JoeLlama (~snork@unaffiliated/joellama) |