| 00:00.19 | *** join/#storm _mup_ (~mup@li37-90.members.linode.com) |
| 04:02.49 | *** join/#storm stub (~stub@canonical/launchpad/stub) |
| 07:57.34 | *** join/#storm jkakar (~jkakar@S0106602ad06e3f4b.vc.shawcable.net) |
| 08:01.38 | *** join/#storm esteve (~quassel@cli-5b7eed72.bcn.adamo.es) |
| 08:22.48 | _mup_ | storm/postgres-repeatable-read r443 committed by thomas@canonical.com |
| 08:22.48 | _mup_ | Remove the dynamic setting, and use REPEATABLE READ everywhere |
| 08:28.54 | _mup_ | storm/postgres-repeatable-read r444 committed by thomas@canonical.com |
| 08:28.54 | _mup_ | Add NEWS statement |
| 09:09.03 | ant384 | Hello. I've used CherryPy a bit, in a pretty standard setup. I'd like now to explore xmlrpc, and I'm trying to find some documentation that takes me from zero to usable xmlrpc code, prefferably with examples. Any tips ? |
| 09:11.52 | ant384 | Wrong window, sorry. |
| 14:13.55 | _mup_ | storm/trunk r442 committed by thomas@canonical.com |
| 14:13.55 | _mup_ | Merge postgres-repeatable-read [r=jkakar,stub] [f=968298] |
| 14:13.55 | _mup_ | Switch to REPEATABLE READ as default isolation level for Postgres, as it's the |
| 14:13.55 | _mup_ | same for Postgres < 9 as SERIALIZABLE, and provides something saner on Postgres |
| 14:13.56 | _mup_ | 9. psycopg2 >= 2.4 needs to be used to see the change. |
| 15:35.43 | webm0nk3y | hello storm hackers, I am getting "out of memory for query result" when doing something like [r for r in store.find(...).order_by(...)] ... AIUI it shold be opening a cursor in PG and reading from it. so why would i get that error? |
| 15:48.35 | jkakar | webm0nk3y: I noticed a couple of days ago that Storm (or psycopg2, not sure which one) is loaded large result sets into memory. |
| 15:48.53 | jkakar | webm0nk3y: I haven't had time to debug it fully, to really understand where the issue is, so no bug has been filed. |
| 15:49.13 | jkakar | webm0nk3y: It's definitely a bug somewhere. Storm *should* be using fetchmany to pull batches of rows as you iterate like that. |
| 15:51.35 | webm0nk3y | jkakar: right |
| 15:51.52 | webm0nk3y | jkakar: or at least a cursor doing one row at a time |
| 15:51.54 | jkakar | webm0nk3y: Are you on Precise? |
| 15:52.10 | webm0nk3y | jkakar: the code in question is lucid |
| 15:52.19 | jkakar | webm0nk3y: Ah, interesting. |
| 15:52.44 | webm0nk3y | jkakar: should be easy to track down though |
| 15:53.24 | jkakar | webm0nk3y: A good place to start (with pdb or something) is in storm.database:Result.__iter__. |
| 15:53.36 | webm0nk3y | jkakar: thanks |
| 15:54.15 | jkakar | np |
| 15:54.56 | webm0nk3y | jkakar: do you think it's relevant which version (precise or lucid) we're running is? |
| 15:55.05 | webm0nk3y | jkakar: maybe the psycopg version? |
| 15:59.08 | jkakar | webm0nk3y: I was wondering if it was a Precise-only thing, but since you're seeing the same behaviour on Lucid I guess it isn't. |
| 15:59.27 | jkakar | webm0nk3y: I have a feeling it might be related to psycopg2. We're just calling psycopg2.fetchmany without arguments, which should (I think), return data in batches... |
| 15:59.35 | jkakar | webm0nk3y: But perhaps we need to be passing a count value? |
| 15:59.42 | webm0nk3y | k |
| 16:02.08 | webm0nk3y | jkakar: according to the documentation: If it is not given, the cursor’s arraysize determines the number of rows fetched |
| 16:02.38 | jkakar | webm0nk3y: Yeah, so that why I suspect there may be a bug in psycopg2...? |
| 16:05.42 | webm0nk3y | <PROTECTED> |
| 16:05.42 | webm0nk3y | <PROTECTED> |
| 16:06.02 | jkakar | webm0nk3y: Yeah, and then _check_disconnect just calls fetchmany without arguments. |
| 16:06.17 | jkakar | webm0nk3y: So I'd expect psycopg2 to do the right thing wrt to deciding how many rows to fetch. |
| 16:06.28 | magmatt | webm0nk3y: you could print out the query storm is running (by turning on the tracer), then execute the same query directly with psycopg2 |
| 16:06.40 | magmatt | that would rule out or indicate psycopg2 as the problem, no? |
| 16:07.02 | webm0nk3y | i don't think seeing the query will help |
| 16:07.15 | webm0nk3y | it's how the underlying code is fetching the results |
| 16:07.16 | magmatt | I mean running the query |
| 16:07.21 | magmatt | oh, okay |
| 16:10.13 | webm0nk3y | jkakar: in storm.database, why is that fetchmany not a function call? |
| 16:10.38 | jkakar | webm0nk3y: It get's passed to a _check_disconnect function which handles disconnection logic... it's called there. |
| 16:10.47 | webm0nk3y | ahh ok |
| 16:11.00 | jkakar | webm0nk3y: See the same file, line 364 or so. |
| 16:11.04 | webm0nk3y | k |
| 16:12.54 | webm0nk3y | jkakar: ahh and we set it to 10 |
| 16:13.00 | webm0nk3y | arraysize i mean |
| 16:14.28 | jkakar | webm0nk3y: Ah, I'd not noticed that... so it still points to psycopg2 ignoring that value, I think. |
| 16:15.19 | webm0nk3y | jkakar: that error is from psycopg2 right, i mean it's not a postgres error is it? I wondered if having the sort may have caused it |
| 16:15.39 | jkakar | webm0nk3y: I guess you can tell from the traceback where it came from...? |
| 16:16.05 | magmatt | webm0nk3y: so... wouldn't executing the same query without storm as a layer tell you whether psycopg2 is the problem or not? I think it will |
| 16:16.27 | jkakar | magmatt: I think it would too. |
| 16:16.35 | webm0nk3y | https://gist.github.com/2252562 |
| 16:17.01 | webm0nk3y | magmatt: if i could run that against production perhaps, but i cant |
| 16:18.01 | webm0nk3y | hmmm...get_select |
| 16:18.02 | jkakar | webm0nk3y: Hmm. That looks like a psycopg2 exception that we're doing magic to wrap in a storm.exceptions.ProgrammingError. |
| 16:18.41 | magmatt | oh, dang... let me see if I can duplicate it. So it's suspected to happen by calling fetchmany on a large result set? |
| 16:19.51 | webm0nk3y | jkakar: but that's not erroring at fetchmany... it's at get_select(..) isn't it? |
| 16:21.04 | webm0nk3y | oh wait im just reading the traceback wrong |
| 16:22.18 | jkakar | webm0nk3y: It looks like that traceback is truncated somehow. |
| 16:22.30 | jkakar | webm0nk3y: I mean, I don't get how that error could come out of that final line. |
| 16:23.41 | webm0nk3y | i'll dig further when i get home...working from a coffeeshop this morning |
| 16:33.36 | jkakar | webm0nk3y: Cool, thanks. |
| 17:55.31 | *** join/#storm _mup_ (~mup@li37-90.members.linode.com) |
| 18:12.40 | *** join/#storm zztw (~travis@router300.sf.archive.org) |
| 18:16.05 | *** part/#storm zztw (~travis@router300.sf.archive.org) |
| 18:37.14 | *** join/#storm Peaceful (~Peaceful@63-235-131-194.dia.static.qwest.net) |
| 18:37.46 | Peaceful | Is there a way to introspect an Enum property to see what the valid values are? |
| 18:38.34 | Peaceful | For example, lets say class Pet has a property: kind = Enum(map={u'dog':u'dog', u'cat':u'cat'}) |
| 18:38.42 | Peaceful | I want to get the list 'dog', 'cat' |
| 18:38.58 | Peaceful | well, ['dog', 'cat'] |
| 18:51.15 | *** join/#storm niemeyer (~niemeyer@177.18.138.61) |
| 18:51.15 | *** mode/#storm [+o niemeyer] by ChanServ |
| 21:24.22 | *** join/#storm kr4d10 (~kr4d10@2001:49f0:a01d::203) |
| 21:28.52 | *** join/#storm kr4d10 (~kr4d10@2001:49f0:a01d::203) |
| 21:29.09 | *** join/#storm kr4d10 (~kr4d10@2001:49f0:a01d::203) |
| 21:36.09 | *** join/#storm kr4d10 (~kr4d10@2001:49f0:a01d::203) |
| 21:38.38 | *** join/#storm kr4d10 (~kr4d10@2001:49f0:a01d::203) |
| 21:38.56 | *** join/#storm kr4d10 (~kr4d10@2001:49f0:a01d::203) |
| 22:29.40 | *** join/#storm magmatt (~Matt@63-235-131-194.dia.static.qwest.net) |
| 23:24.51 | *** join/#storm fcorrea (~fcorrea@189.121.216.174) |