| 00:08.03 | *** join/#storm artista_frustrad (~artista_f@201-25-167-171.ctame704.dsl.brasiltelecom.net.br) |
| 00:17.36 | *** join/#storm danilos (~danilo@canonical/launchpad/danilos) |
| 00:34.16 | *** join/#storm fcorrea (~fcorrea@201-75-152-196-nd.cpe.vivax.com.br) |
| 01:08.42 | *** join/#storm shaunm (~shaunm@c-98-212-133-244.hsd1.il.comcast.net) |
| 02:43.03 | *** join/#storm stub (~stub@ppp-58-8-14-142.revip2.asianet.co.th) |
| 02:43.03 | *** join/#storm stub (~stub@canonical/launchpad/stub) |
| 03:32.30 | *** join/#storm stub (~stub@ppp-58-8-14-142.revip2.asianet.co.th) |
| 03:32.30 | *** join/#storm stub (~stub@canonical/launchpad/stub) |
| 04:58.50 | *** join/#storm jukart (~jukart@d86-32-163-62.cust.tele2.at) |
| 05:47.02 | *** join/#storm jukart (lovely@81.189.156.94) |
| 07:13.26 | *** join/#storm elmom_ (~elmom@vallila-gw.hupnet.helsinki.fi) |
| 08:21.04 | *** join/#storm zmijunkie (~chatzilla@i59F5ED4D.versanet.de) |
| 08:44.12 | *** join/#storm jkakar (~jkakar@83.34.90.69) |
| 09:07.43 | *** join/#storm keppla (~keppla@i577B1598.versanet.de) |
| 10:50.30 | *** join/#storm stub (~stub@canonical/launchpad/stub) |
| 11:01.21 | *** join/#storm stub1 (~stub@ppp-58-8-14-126.revip2.asianet.co.th) |
| 11:01.24 | *** join/#storm stub (~stub@canonical/launchpad/stub) |
| 12:21.16 | *** join/#storm niemeyer_ (~niemeyer@189.74.206.84) |
| 13:27.08 | *** join/#storm fcorrea (~fcorrea@187.3.15.40) |
| 13:42.49 | *** join/#storm stub (~stub@canonical/launchpad/stub) |
| 16:18.54 | midgetspy | In my attempt to make storm/sqlite work in a multi-threaded environment I've built a single thread that contains the only storm.locals.Store() object in my app, and have built a queue system where I pass functions in to that thread and it gives me results. So instead of store.find(..) in a different thread I do my.safe_store("find", ...). |
| 16:19.19 | jkakar | midgetspy: Sounds nice! |
| 16:19.23 | midgetspy | What I've encountered with this approach is the constant attempt by Storm to create store objects all over the place and use them implicitly |
| 16:20.10 | jkakar | midgetspy: Yeah, you don't want to use the objects outside the thread you got them from. Which I guess is a big problem for the approach you describe. |
| 16:20.11 | midgetspy | err, not create them, just pass them and use them |
| 16:21.07 | midgetspy | I specifically worked around ResultSet.one(), and now I've discovered that [x for x in myResultSet] also attempts to use a store. So I'm wondering, is it worth trying to work around these problems or is this approach flawed to begin with? And if so, how should storm/sqlite multithreading be done properly (or is it even possible)? |
| 16:22.18 | midgetspy | jkakar: so it is literally not possible to use storm/sqlite in a multi-threaded app unless all storm objects are kept in a single thread and only the data is passed out/in? |
| 16:24.37 | midgetspy | bob2 mentioned zstorm, but from what I can tell it is only going to manage store objects in a manner similar to how I'm already trying to do it so I'm not sure if that's really the solution either? |
| 16:24.39 | *** join/#storm keppla1 (~keppla@i577B05E7.versanet.de) |
| 16:24.49 | jkakar | midgetspy: Most likely yes... this is because of SQLite, it's single-connection, right? |
| 16:28.14 | midgetspy | yes but that's not really the problem |
| 16:28.21 | *** join/#storm jukart (~jukart@d86-32-163-62.cust.tele2.at) |
| 16:28.47 | midgetspy | the problem is that you can't use an sqlite object in a thread other than the one it was created in |
| 16:28.50 | jkakar | midgetspy: Well, sure, it's part of the problem. |
| 16:29.19 | jkakar | midgetspy: But yeah, I see your point. |
| 16:29.32 | midgetspy | well the single access just means that the DB may be locked at any time so I may need to wait to get access. waiting isn't a big deal though |
| 16:30.53 | midgetspy | but a Store keeps an sqlite object inside it it seems which means it can't be used in any thread other than the one it was created in (which means my storm objects can't be, either) |
| 16:31.32 | jkakar | midgetspy: Yeah, true. |
| 16:32.06 | jkakar | midgetspy: I haven't used Storm to actually write an application with SQLite (the applications I work on are all PostgreSQL-based) so sorry for the fuzzy answers. :/ |
| 16:32.22 | midgetspy | np |
| 16:32.53 | midgetspy | I think ideally it should create a new sqlite object for every new transaction which would then only limit any single transaction to a single thread... any idea how hard that'd be to implement? I imagine it's all generic at that point, eh... |
| 16:36.02 | jkakar | midgetspy: That wouldn't be too much different than creating a store and throwing it away when you're done. |
| 16:36.15 | jkakar | midgetspy: It would involve connection overhead for each transaction though. |
| 16:37.29 | jkakar | midgetspy: You could probably do some refactoring in Storm itself to avoid that overhead, using a pool of connections, one per thread. I think it'd mostly be tricky because you'd need to maintain the existing behaviour. |
| 16:38.43 | midgetspy | right, I can live with some overhead for now (at least when the other option is starting over from scratch without storm, hehe) |
| 16:38.48 | jkakar | Hehe |
| 16:39.01 | midgetspy | I always worry about performance last anyway |
| 16:41.11 | jkakar | Make it right, then make it fast. :) |
| 16:41.30 | jkakar | Sometimes that doesn't work, but much of the time it does. |
| 16:45.06 | midgetspy | so having never looked at the storm code, how difficult is it going to be to make storm throw away store objects after every transaction? is it something that's feasible for me to implement myself or would it be a major job? |
| 16:47.14 | jkakar | midgetspy: Instead of changing Storm, what about just creating a new store for each transaction and then calling store.close on it when you're done...? |
| 16:52.27 | midgetspy | hm that might work, I will have to try that. If I do a myResultSet = store.find() then store.close() in one thread then pass myResultSet to another thread and do myResultSet.one() on it, will it just make a new store? or will that store still be alive? (or should I just try it and figure out for myself? ;-P) |
| 16:52.54 | jkakar | Things will blow up. You need to make the store and use the objects and then close the store all in the same thread. |
| 16:56.13 | midgetspy | hm |
| 17:09.24 | midgetspy | yeah so I still have the same problem, closing the store can't solve it. For example, I have Foo with bar=Reference(_bar_id, Bar.id). If I get an instance of Foo from my single-thread store, there's no way I can use Foo.bar in another thread because it requires the store instance. If I close it it blows up, if I leave it alone it dies because I'm using a sqlite object in a different thread than it was created in. |
| 17:17.34 | jkakar | midgetspy: Right, so in a given thread you need to: (1) get a store, (2) use it and do whatever computation you want to, (3) call store.close. |
| 17:17.42 | jkakar | midgetspy: IOW, you can't share objects between threads. |
| 17:22.17 | midgetspy | jkakar: well computations in one thread would be one thing, but since accessing referenced data (Foo.bar.attr in the above example) on my storm-backed objects requires a store, I can't even access my data in more than one thread I guess |
| 17:22.46 | midgetspy | perhaps I could make a special accessor on the objects that passes itself to the store thread, does the access there, and returns the value |
| 17:23.28 | midgetspy | After working on this right before bed I dreamt about this problem last night... tonight it will be a nightmare ;-P |
| 17:25.31 | jkakar | Heh |
| 17:25.47 | jkakar | midgetspy: Yes, you can't access data in more than one thread. |
| 17:25.47 | jkakar | I'm stepping away, good luck! |
| 17:25.54 | midgetspy | ok, thanks |
| 20:40.54 | *** part/#storm jkakar (~jkakar@83.34.90.69) |
| 23:25.38 | *** join/#storm shaunm (~shaunm@c-98-212-133-244.hsd1.il.comcast.net) |