| 01:01.40 | *** join/#storm gord (n=gord@5ac32860.bb.sky.com) |
| 02:39.26 | *** join/#storm sidnei (n=sidnei@plone/dreamcatcher) |
| 03:37.48 | *** join/#storm shaunm (n=shaunm@c-98-212-133-244.hsd1.il.comcast.net) |
| 03:50.49 | *** join/#storm infobot (i=ibot@rikers.org) |
| 03:50.49 | *** topic/#storm is The Storm Python ORM - http://storm.canonical.com/ - 0.14 released! || Review branches: https://code.launchpad.net/storm/+activereviews || IRC logs: http://ibot.rikers.org/#storm/ |
| 05:34.51 | *** join/#storm sidnei_ (n=sidnei@201-35-176-220.cslce701.dsl.brasiltelecom.net.br) |
| 06:01.32 | *** join/#storm jkakar (n=jkakar@S0106001ee57a879c.vc.shawcable.net) |
| 06:08.48 | *** join/#storm jukart (i=lovely@81.189.156.94) |
| 12:11.58 | *** join/#storm thumper (n=quassel@canonical/launchpad/thumper) [NETSPLIT VICTIM] |
| 12:40.37 | *** join/#storm gord (n=gord@5ac32860.bb.sky.com) |
| 13:53.50 | *** join/#storm sidnei__ (n=sidnei@201-35-176-220.cslce701.dsl.brasiltelecom.net.br) |
| 14:03.39 | *** join/#storm oubiwann (n=oubiwann@97-119-11-160.omah.qwest.net) |
| 14:34.14 | *** join/#storm andrea-bs (n=andrea-b@ubuntu/member/beeseek.developer.andrea-bs) |
| 15:07.20 | *** join/#storm vvinet (n=vince@132.210.76.201) |
| 15:18.00 | *** part/#storm sidnei (n=sidnei@201-35-176-220.cslce701.dsl.brasiltelecom.net.br) |
| 16:10.13 | *** join/#storm shaunm (n=shaunm@proxyserver.wolfram.com) |
| 16:28.12 | *** join/#storm jukart (n=jukart@d91-128-122-97.cust.tele2.at) |
| 16:44.37 | *** join/#storm bigdog (n=scmikes@72-197-8-8-arpa.cust.cinci.current.net) |
| 18:09.03 | *** join/#storm gord (n=gord@5ac32860.bb.sky.com) |
| 18:43.39 | *** join/#storm bigdog (n=scmikes@72-197-8-8-arpa.cust.cinci.current.net) |
| 19:12.45 | *** join/#storm sidnei (n=sidnei@plone/dreamcatcher) |
| 19:14.13 | *** part/#storm sidnei (n=sidnei@plone/dreamcatcher) |
| 19:37.39 | *** join/#storm intellectronica (n=tom@intellectronica.net) |
| 19:37.59 | intellectronica | hi |
| 19:38.24 | intellectronica | is there a way to do a multiple row insert with a storm expression? |
| 20:52.20 | jkakar | intellectronica: Nope. |
| 20:53.04 | intellectronica | jkakar: thanks for confirming that :) |
| 20:53.10 | jkakar | intellectronica: np. :) |
| 20:53.33 | intellectronica | now let's go and concatenate some SQL commands |
| 20:58.00 | *** part/#storm intellectronica (n=tom@intellectronica.net) |
| 20:59.04 | *** join/#storm vvinet (n=vince@ip200-215.natdynamique.USherbrooke.ca) |
| 21:00.53 | *** join/#storm bigdog (n=scmikes@72-197-8-8-arpa.cust.cinci.current.net) |
| 21:17.53 | *** join/#storm jkakar (n=jkakar@S0106001ee57a879c.vc.shawcable.net) |
| 21:56.49 | *** join/#storm jkakar (n=jkakar@S0106001ee57a879c.vc.shawcable.net) |
| 22:24.31 | *** join/#storm MFen (n=cdodt@demo.goonmill.org) |
| 22:24.47 | MFen | is there a way to tell storm to separate transactions? |
| 22:25.01 | jkakar | MFen: Call store.commit()...? |
| 22:25.09 | MFen | we're having major problems with autosequenced primary keys stepping on each other |
| 22:25.29 | jkakar | Are you using MySQL with a non-MyISAM table backend? |
| 22:25.34 | MFen | postgres |
| 22:25.39 | jkakar | Oh, okay, good. |
| 22:25.52 | jkakar | Are you running very long-lived transactions? |
| 22:26.24 | MFen | no |
| 22:26.37 | MFen | asynchronous web application |
| 22:26.43 | MFen | very short transactions |
| 22:27.17 | *** join/#storm boneskull (n=chiller@c-98-246-131-168.hsd1.or.comcast.net) |
| 22:28.38 | radix | MFen: hm |
| 22:28.42 | MFen | boneskull can tel you more :) |
| 22:28.51 | radix | MFen: see what happens if you put a rollback() at the *beginning* of request processing? |
| 22:30.17 | radix | (which is a good idea anyway, because that's where storm does reconnection if the connection was lost) |
| 22:30.17 | MFen | how many transactions do you get in a single store? is it at all tied to function scope? is everything that happens through storm in the same transaction (until you commit()) ? |
| 22:30.27 | radix | MFen: no, one store can only have one transaction at a time |
| 22:30.37 | MFen | ok |
| 22:30.44 | radix | you should have as many stores as you have concurrent transactions |
| 22:30.50 | MFen | ahh. |
| 22:30.55 | MFen | so one per request, basically |
| 22:31.20 | radix | well, that would be semnaatically correct |
| 22:31.24 | radix | semantically |
| 22:31.49 | radix | but often you'll want to cache stores between requests so you don't have to reconnect to the database, for example |
| 22:32.11 | MFen | well, what about the database object that you initialize the store with. can we keep that around? |
| 22:32.32 | jkakar | Yes, and it can be shared with stores in different threads. |
| 22:32.45 | MFen | ok, so that saves us connection costs right? |
| 22:32.57 | jkakar | The Store's themselves aren't thread-safe, so only use a Store in one thread. |
| 22:33.06 | MFen | no threads, yay |
| 22:33.15 | jkakar | Not really, reusing Store's saves you reconnection cost. |
| 22:33.20 | MFen | hmm |
| 22:33.23 | MFen | Storepool? :-) |
| 22:33.37 | jkakar | That's basically what IZStorm and the Django integration do, yeah. |
| 22:33.55 | MFen | boneskull: what does db_upsert do about transactions? |
| 22:34.03 | boneskull | let me see |
| 22:34.42 | jkakar | Are you using Twisted for your asynchronous application (out of curiousity)? |
| 22:35.07 | MFen | no. homegrown socket server here |
| 22:35.08 | radix | oh, are you using the twisted-storm integration? |
| 22:35.10 | jkakar | I guess you're blocking in the main loop to talk to the database (since you say 'no threads, yay')? |
| 22:35.17 | jkakar | MFen: Ah, okay. |
| 22:35.17 | radix | oh |
| 22:35.19 | MFen | in our defense, it predates really viable twisted |
| 22:35.54 | MFen | yes, blocking in the main thread |
| 22:35.59 | boneskull | our db_* code does not use transactions; it autocommits |
| 22:36.16 | boneskull | can we just make storm autocommit? |
| 22:36.18 | MFen | i really don't see how what we're seeing can be happening if we're really committing everywhere we should be |
| 22:36.19 | jkakar | Ah, I see. |
| 22:37.19 | jkakar | MFen: Right, since you only have one connection to the database presumably and you're serializing all access through that one connection, right? |
| 22:37.21 | boneskull | I'm commit()-ting all over the place |
| 22:38.12 | MFen | commit() more! |
| 22:38.15 | MFen | :P |
| 22:38.30 | MFen | are you sure there's no yield somewhere there shouldn't be perhaps |
| 22:38.40 | boneskull | there is a yield |
| 22:38.42 | MFen | or a try without a finally |
| 22:38.45 | jkakar | Well, isn't "commit" basically a no-op when your isolation level is autocommit? |
| 22:39.23 | jkakar | You might want to try calling Store.flush() at the end of your logical transaction, to make sure Storm's caching isn't delaying database writes. |
| 22:39.46 | MFen | yeah. i was thinking about flush() yesterday boneskull |
| 22:39.47 | boneskull | storm is not autocommitting... |
| 22:39.54 | radix | indeed, storm doesn't autocommit |
| 22:40.04 | boneskull | so store.commit() then store.flush() ? |
| 22:40.12 | radix | no, there's no reason to call flush() next to a commit() call |
| 22:40.21 | MFen | flush() makes the sql calls |
| 22:40.22 | boneskull | MFen, is the scope of the store OK? I don't understand that |
| 22:40.34 | MFen | commit makes the sql calls and closes the transaction |
| 22:40.56 | MFen | (right radix?) |
| 22:41.29 | radix | yes |
| 22:41.49 | radix | also, many things invoke flush(): pretty much every time storm does an implicit read operation on the database, it first invokes flush |
| 22:42.02 | mzz | its implementation is not as scary as you might think: it's just self.flush(); self.invalidate(); self._connection.commit() (which does an actual sql commit) |
| 22:44.51 | MFen | hmm |
| 22:44.55 | MFen | we're using multiple processes |
| 22:46.00 | MFen | if i create an object with an autoseq pk |
| 22:46.06 | MFen | and then another process does that at the same time |
| 22:46.18 | MFen | the database doesn't enforce pk integrity until one of them calls flush |
| 22:46.23 | MFen | the second one then loses |
| 22:46.35 | MFen | i think that's what we're doing wrong.. need to flush immediately when we create one |
| 22:46.37 | boneskull | that's an autoseq pk? |
| 22:46.49 | MFen | autoincrementing primary key |
| 22:46.53 | boneskull | in what table? |
| 22:47.25 | MFen | don't all of them have a sequence? |
| 22:47.53 | boneskull | we are explicitly inserting 'project' into action_stack |
| 22:48.41 | MFen | what's the constraint that's being violated again? |
| 22:48.59 | MFen | i guess only action has a serial id |
| 22:49.11 | boneskull | duplicate 'project' |
| 22:49.12 | MFen | but the project numbers are also dynamically assigned, but from application code |
| 22:49.16 | boneskull | yes |
| 22:49.18 | MFen | which is probably worse |
| 22:49.24 | boneskull | yes that is wack |
| 22:49.31 | jkakar | (Maybe this is not a helpful question, but, have you thought about using transactions to avoid all this kind of nastiness?) |
| 22:49.34 | boneskull | ...but it worked before |
| 22:49.34 | radix | you're talking about SERIAL, right? |
| 22:49.45 | MFen | jkakar: aren't we *always* using transactions? |
| 22:50.03 | MFen | radix: actually i thought we were, but in fact we're generating a primary key string in application code |
| 22:50.08 | radix | oh, dude |
| 22:50.10 | jkakar | MFen: Not if you've set your isolation mode to autocommit, no. |
| 22:50.16 | radix | ah, yes |
| 22:50.22 | MFen | jkakar: we haven't done anything like that |
| 22:50.25 | MFen | that i know of |
| 22:50.32 | boneskull | only our homebrew code autocommits |
| 22:50.35 | radix | I find boneskull's explanation of that situation apropos ;-) |
| 22:50.38 | radix | "wack" indeed ;-) |
| 22:51.02 | boneskull | this is embarrassing |
| 22:51.13 | MFen | learning experience! |
| 22:51.19 | radix | hehe, I know what it's like to work with nasty legacy code :) |
| 22:51.19 | boneskull | we are actually looking at a directory structure to determine the key |
| 22:51.21 | jkakar | Okay, I guess I misunderstood this then: <boneskull> our db_* code does not use transactions; it autocommits |
| 22:51.49 | jkakar | Ah, I see. Yeah, legacy code can be hard. :) |
| 22:51.56 | MFen | jkakar: db_foo is our non-Storm db layer |
| 22:52.02 | MFen | which we're working on replacing |
| 22:52.07 | jkakar | MFen: Right, catching onto that now. |
| 22:52.12 | boneskull | if directory foo3 does not exist, and doesn't exist in the db, use foo3 |
| 22:52.45 | MFen | yeah, extract bonus wackness there |
| 22:52.52 | jkakar | Heh |
| 22:53.04 | MFen | in order for this to work, we have to flush *and* create that directory before the other process makes its move |
| 22:53.23 | MFen | the directory creating thing is one thing we should burn. |
| 22:53.45 | boneskull | we check for the path; if it does not exist we store.flush(), then we store.find() |
| 22:54.01 | MFen | is there any way to create a lock through storm? just a teeny one? |
| 22:54.06 | boneskull | heh |
| 22:56.09 | boneskull | we could touch a lockfile or something :P |
| 22:56.45 | MFen | gah. |
| 22:57.35 | jkakar | MFen, boneskull: Maybe's PostgreSQL's table locking can help you: http://www.postgresql.org/docs/8.3/static/sql-lock.html |
| 23:00.08 | MFen | it would if we weren't creating that directory first |
| 23:00.37 | MFen | we have to check for a free directory, then announce our intention to use it to the database |
| 23:01.32 | MFen | which is just friggin stupid. chris, we should be using a pseudorandom directory instead. |
| 23:02.02 | boneskull | heh |
| 23:02.16 | boneskull | I don't see why not |
| 23:02.22 | boneskull | these directory names don't really matter, do they |
| 23:02.48 | boneskull | still the fact remains, when autocommit was on we didn't have this problem... |
| 23:03.01 | MFen | not much. it would be nice if it still had the year in there, but just YYMM+4 alphanumerics should be plenty of random |
| 23:03.29 | MFen | yeah, but that's not how storm works |
| 23:03.53 | MFen | in theory we should be reaping performance benefits from this design |
| 23:04.31 | boneskull | if we include transactional logic, we need error handling to support it |
| 23:05.03 | radix | well, transactions *are* slower than just writing random corrupted data to disk ;-) |
| 23:05.26 | MFen | we just need to be more precise about when we use locks |
| 23:05.33 | MFen | this is one of those places |
| 23:05.46 | MFen | pseudorandom directory plus table lock equals success |
| 23:58.01 | *** join/#storm jkakar (n=jkakar@S0106001ee57a879c.vc.shawcable.net) |