| 00:00.19 | *** join/#storm wallflower (n=wallflow@ip-205-246-113-216.pool.grokthis.net) |
| 00:40.01 | *** join/#storm gord (n=gord@5acacaac.bb.sky.com) |
| 02:26.54 | thumper | jkakar: you around? |
| 02:27.06 | jkakar | thumper: Heya. |
| 02:27.09 | thumper | jkakar: hi |
| 02:27.25 | thumper | jkakar: can you remind me about the default ordering that we get? |
| 02:27.29 | thumper | jkakar: and how to not get it? |
| 02:28.12 | jkakar | thumper: You can specify a default order by defining, on a Storm class, __storm_order__ = (tuple of columns). |
| 02:28.13 | thumper | jkakar: I have some _interesting_ queries being generated |
| 02:28.34 | jkakar | thumper: You can call order_by on the result set to set it to something else. |
| 02:28.43 | jkakar | thumper: Yeah, I figured you guys would run into this with your design. |
| 02:28.48 | thumper | jkakar: how do I set it to nothing |
| 02:29.08 | thumper | jkakar: we have this query right from .branches() |
| 02:29.10 | jkakar | thumper: In Landscape we used to have something similar, but ended up going with functions instead, because we could optimize each query individually, which is hard when you have nodes of builders all lined up. |
| 02:29.24 | thumper | hmm... |
| 02:29.30 | thumper | jkakar: got pointers for us? |
| 02:29.38 | *** join/#storm shaunm (n=shaunm@c-98-212-133-244.hsd1.il.comcast.net) |
| 02:30.02 | jkakar | thumper: Calling result_set.order_by() looks like it'll work. |
| 02:30.37 | thumper | jkakar: I was thinking of pointers re: our design |
| 02:30.48 | thumper | jkakar: if you guys have been down this road before, and discarded it |
| 02:30.52 | thumper | jkakar: I'd really like to know |
| 02:31.41 | jkakar | thumper: In Landscape we had functions like get_computer(tags=(), distribution=None, ...). Each one would dynamically build up a where clause, just like your classes do, though not with the cool chaining you have. |
| 02:32.27 | jkakar | thumper: Anyway, we found that in many cases we had to break them out into individual classes like get_computers_with_tags, get_computers_with_distribution, etc. so that we could create the exact query that would perform as best as possible. |
| 02:32.38 | thumper | jkakar: ok |
| 02:32.47 | thumper | jkakar: I think I can bend this to my will |
| 02:32.51 | thumper | jkakar: for our design anyway |
| 02:32.53 | jkakar | thumper: I have a feeling you'll eventually have queries being produced by your collection of where clause builders that will cause you grief. |
| 02:33.11 | jkakar | thumper: Maybe, in those case, you'll be able to do some special casing, or maybe those cases will be rare enough that you can just provide a function. |
| 02:33.13 | thumper | jkakar: possibly, but not just yet |
| 02:33.49 | jkakar | thumper: In any case, I like the elegance of your system. I'm curious to see how it works in practice. |
| 02:34.06 | thumper | jkakar: it works very well for what it was originally used for |
| 02:34.13 | thumper | jkakar: which was to reduce the set of branches |
| 02:34.22 | thumper | jkakar: however we're now using it to get merge proposals as well |
| 02:34.31 | thumper | jkakar: which have a relationship with branches |
| 02:34.53 | thumper | jkakar: we are using .branches().result_set._select() |
| 02:34.57 | jkakar | thumper: Ah, right. Yeah, it was join logic (ie: join or do a subselect) that caused us lots of performance grief. |
| 02:35.01 | thumper | jkakar: (we have decorated result set) |
| 02:35.03 | jkakar | thumper: Right. |
| 02:35.15 | thumper | jkakar: but the select has extra tables and ordering I need to kill |
| 02:35.31 | jkakar | thumper: I see. |
| 02:36.00 | jkakar | thumper: That's an annoying problem. I mean, you can easily set select_expr.order_by to Undef, but it's pretty ugly. |
| 02:36.28 | thumper | jkakar: so, I need to chain another builder without the explicit tables |
| 02:36.50 | thumper | jkakar: then go something like .branches().result_set.order_by()._select() |
| 02:36.56 | jkakar | thumper: Do you need to remove the explicit tables because they cause a performance problem? |
| 02:37.00 | thumper | yes |
| 02:37.04 | jkakar | I see. |
| 02:37.09 | thumper | they are only joined for the order by |
| 02:37.19 | jkakar | I see. |
| 02:37.26 | thumper | so, kill the order by |
| 02:37.30 | thumper | don't need the tables |
| 02:37.43 | thumper | ... |
| 02:37.48 | thumper | however I forsee a problem |
| 02:37.52 | thumper | shudders |
| 02:38.09 | jkakar | Well, I can easily think of ways to mangle the Select expression to do what you want, but I'm trying to think of what the right thing would be. |
| 02:38.13 | jkakar | Problem? |
| 02:41.20 | thumper | one of our tables is a default one |
| 02:41.30 | thumper | and there is the possibility that it is in one of the expressions |
| 02:41.46 | thumper | will storm barf if it isn't in the store.using(...) ? |
| 02:42.11 | thumper | does .using override the table inference engine? |
| 02:45.00 | jamesh | if you use store.using(), that needs to specify all the tables used by the query |
| 02:45.47 | jamesh | the sqlobject compatibility layer doesn't use store.using(), instead using AutoTables() -- something that compiles to a truth value but also introduces a set of tables to the query |
| 02:46.27 | jamesh | it is a bit of a hack, but it seems like a feature direct users of the Storm API want |
| 02:46.30 | thumper | jamesh: we are using pure storm queries for this |
| 02:48.33 | jamesh | thumper: so, store.find(A, AutoTables(B.id == None, LeftJoin(A, B, A.id == B.foo))) would be an example |
| 02:49.33 | thumper | jamesh: interesting |
| 02:49.39 | thumper | AutoTables is in storm? |
| 02:49.49 | jamesh | storm.expr.AutoTables |
| 03:07.20 | thumper | jamesh: I think I'm going to have to come back to autotables |
| 03:07.38 | thumper | jamesh: maybe I'll hit you up for a call some time, and we can go over the branchcollection code |
| 03:07.45 | jamesh | sure. |
| 03:19.33 | thumper | here goes another question |
| 03:20.08 | thumper | if I have something like rs = store.find(Branch, some_expressions) not including Person |
| 03:20.54 | thumper | can we do something like rs.order_by(AutoTables(Person.name, Join(Branch, Person, Branch.ower == Person.id))) ? |
| 03:23.19 | thumper | jamesh: ?? |
| 03:23.51 | jamesh | thumper: possibly. |
| 03:23.57 | thumper | heh |
| 03:24.09 | thumper | so.., not sure then? |
| 03:24.28 | jamesh | it'll probably work, but I don't know whether we'd want to be held to that ... |
| 03:24.43 | thumper | why not? |
| 03:24.46 | thumper | it seems reasonable |
| 03:24.57 | jamesh | AutoTables was added as a way to get prejoins working in the sqlobject compatibility layer |
| 03:25.06 | thumper | ah, |
| 03:25.35 | thumper | can tables be added to a result set after it is created? |
| 03:25.42 | jamesh | I realise it is a useful feature, but I don't know whether it is the interface we want long term |
| 03:26.22 | thumper | to do this type of orderby? |
| 03:26.29 | jamesh | go ahead and use it to add left/right joins in the where clause, but it'd be good to limit it to cases where there aren't good alternatives |
| 03:27.14 | thumper | my actual use case is for the order by clause |
| 03:27.30 | thumper | so if we can add tables after the store.find() to do the order by bits |
| 03:27.34 | thumper | then we have a winner |
| 03:28.26 | jamesh | perhaps mail the list saying what you want to do |
| 03:28.36 | thumper | hmm.. ok |
| 03:29.08 | jamesh | I'd personally be happier with something like this though: result2 = result1.find(Branch.owner == Person.id); result2.order_by(Person.name) |
| 03:29.19 | jamesh | although I haven't implemented ResultSet.find() yet :) |
| 03:31.48 | thumper | :) |
| 03:34.35 | jml | oh hello |
| 04:54.19 | *** join/#storm bigdog (n=scmikes@72-197-8-8-arpa.cust.cinci.current.net) |
| 04:56.37 | jkakar | thumper: I wonder if your branch collection pattern could be augmented with some kind adaptation. So, whenever a query is about to be run, with all the where parameters, you try to adapt all the parameters to an OptimizedBranchSearch, or something. |
| 04:57.21 | jkakar | thumper: IOW, I wonder if the problems with needing to optimize certain queries could be overcome with some kind of adaptation system to make it possible to optionally provide better queries. |
| 04:58.15 | jkakar | I wonder how heinous the adaptation code would get over time. I can imagine it getting hairy. |
| 05:08.27 | jml | jkakar: did I show you the version of the module that has visibleByUser return an object of a different class? |
| 05:08.50 | jml | (it does this to optimize queries) |
| 05:38.58 | *** join/#storm sidnei (n=sidnei@plone/dreamcatcher) |
| 06:12.54 | *** join/#storm jukart (n=jukart@d91-128-122-97.cust.tele2.at) |
| 06:13.15 | jkakar | jml: Ah, yes, I see what you've done here. Nice. |
| 06:43.52 | jml | jkakar: thanks. |
| 06:59.44 | *** join/#storm sidnei (n=sidnei@plone/dreamcatcher) |
| 07:24.46 | *** join/#storm jukart (i=lovely@81.189.156.94) |
| 11:28.10 | *** join/#storm bac` (n=bac@cpe-065-190-186-163.nc.res.rr.com) |
| 12:28.15 | *** join/#storm niemeyer (n=niemeyer@200-138-42-229.ctame705.dsl.brasiltelecom.net.br) |
| 13:33.24 | *** join/#storm bigdog (n=scmikes@72-197-8-8-arpa.cust.cinci.current.net) |
| 14:34.18 | *** join/#storm bac` (n=bac@cpe-065-190-186-163.nc.res.rr.com) |
| 14:42.53 | *** join/#storm deryck (n=deryck@samba/team/deryck) |
| 16:07.17 | *** join/#storm shaunm (n=shaunm@proxyserver.wolfram.com) |
| 17:05.30 | *** join/#storm jukart (n=jukart@d91-128-122-97.cust.tele2.at) |
| 17:16.33 | *** join/#storm jukart_ (n=jukart@d91-128-122-97.cust.tele2.at) |
| 17:54.48 | *** join/#storm jukart (n=jukart@d91-128-122-97.cust.tele2.at) |
| 17:55.36 | *** join/#storm MFen (n=cdodt@demo.goonmill.org) |
| 17:55.52 | MFen | are numeric comparison operators in store.find() queries known to work, for postgres tables with schemas? |
| 17:57.20 | MFen | i.e. class Bar(object): __storm_table__ = 'foo.bar' ; xyz = Int() and store.find(Bar, Bar.xyz > 10) |
| 17:57.29 | MFen | we're getting no results for that, although Bar.xyz == 10 works |
| 17:57.38 | MFen | and it works in a sqlite table i have here |
| 18:00.24 | radix | it should work |
| 18:01.19 | MFen | do the unit tests run on all database backends? |
| 18:01.35 | MFen | (i'm sure you have one for numeric comparisons) |
| 18:04.13 | radix | yes |
| 18:04.15 | radix | looks |
| 18:04.56 | *** join/#storm sidnei_ (n=sidnei@201-66-129-211.cslce701.dsl.brasiltelecom.net.br) |
| 18:05.57 | radix | MFen: are you sure that the field is an int column? |
| 18:06.13 | MFen | yeah, we checked that |
| 18:06.36 | MFen | i'm setting up a minimal test case, i'll report back.. |
| 18:06.40 | radix | okie |
| 18:07.25 | MFen | ok it's working for me but not him. something's wrong in the code. thanks. :) |
| 18:08.08 | radix | success |
| 19:28.17 | *** join/#storm sidnei (n=sidnei@plone/dreamcatcher) |
| 22:19.31 | pyCube | Has anybody done any work on an oracle database backend for storm? |
| 22:19.45 | pyCube | i am fearing we might end up going oracle |
| 22:20.05 | jkakar | pyCube: There's a branch, I think, with work in that direction. I don't know how far along it is. |
| 22:20.19 | pyCube | eesh.. |
| 22:20.35 | pyCube | i hope i can convince peopel that postgresql would be a better way to go |
| 22:21.07 | jkakar | pyCube: https://bugs.edge.launchpad.net/storm/+bug/127176 |
| 22:21.08 | mup | Bug #127176: Storm should support Oracle databases <review> <Storm:New> <https://launchpad.net/bugs/127176> |
| 22:21.20 | jkakar | pyCube: https://bugs.edge.launchpad.net/storm/+bug/309382 |
| 22:21.21 | mup | Bug #309382: add Sequence compilation to oracle-support <Storm:New> <https://launchpad.net/bugs/309382> |
| 22:21.39 | jkakar | pyCube: https://bugs.edge.launchpad.net/storm/+bug/307516 |
| 22:21.40 | mup | Bug #307516: NamerError due to typo <Storm:New for kov-alfaiati> <https://launchpad.net/bugs/307516> |
| 22:22.03 | jkakar | pyCube: Maybe some combination of the branches there will give you a working Storm on Oracle? |
| 22:23.57 | pyCube | but ya know.. people feel all safe when they blow wads of cash on stuff that "fortune 500" companies use |
| 22:24.13 | pyCube | yeah.. i just hope i dont have to worry abuot it... |
| 22:26.26 | jkakar | pyCube: PostgreSQL works pretty well for Canonical. |
| 22:29.07 | pyCube | yeah.. our biggest issue now is that we have mulitple geographically separate locations.. and we'd really like to have multiple master replaicationiness going on |
| 22:32.18 | jkakar | pyCube: I see. |
| 22:32.38 | pyCube | it appears that that is a bitch. no matter the backend |
| 22:34.40 | jkakar | Heh |
| 22:40.54 | pyCube | i think we just need to hire a kickass postgresql dba and actually have an employee to maintain things for us, rather than dump a ton of cash at oracle |
| 22:41.00 | pyCube | it just doesnt 'feel' right |
| 22:41.54 | pyCube | oh well.. heh |
| 23:01.13 | jkakar | Yeah, I would be inclined in that direction. Smart people can be way more valuable that fancy stuff. |
| 23:24.07 | MFen | does store have anything like Axiom's upgraders? |
| 23:24.09 | MFen | storm* |
| 23:24.24 | MFen | i.e. def upgrade1to2(self): ... modify the object to make it a version 2 object ... |
| 23:24.45 | MFen | this seems like an easy thing to implement myself, but if it happens to be built into storm, why should i? |
| 23:33.06 | jkakar | MFen: No, it doesn't. |
| 23:33.36 | jkakar | MFen: We use a patching system in Landscape (and I think some other internal Canonical projects) that there's been some talk of open sourcing and including with Storm, but no action yet. |
| 23:35.52 | MFen | is that what that's called? a patching system? |
| 23:36.09 | MFen | jkakar: or are you talking about something different |
| 23:39.08 | *** join/#storm pyCube (n=jmayfiel@12.87.213.242) |
| 23:54.44 | jkakar | MFen: We have a system where we add Python modules called patch1.py, patch2.py, etc. |
| 23:54.59 | jkakar | MFen: A patch table in the database stores the versions that have been applied. |
| 23:55.27 | jkakar | MFen: Whenever we upgrade a database we get all the patches that haven't been applied, and apply them. |
| 23:56.00 | jkakar | MFen: Each patch has an 'apply' method that takes a Store for the database and uses it to make the needed changes. |
| 23:56.02 | MFen | so you patch the entire database |
| 23:56.07 | jkakar | MFen: Yes. |
| 23:56.12 | MFen | it's not per-object somehow. ok |
| 23:56.31 | jkakar | MFen: You could create a policy where you only manipulate one object per patch, but the system doesn't enforce that. |
| 23:56.51 | MFen | the Axiom system upgrades an object as it loads, when it loads |
| 23:56.53 | jkakar | MFen: In practice, the patches tend to make small changes. |
| 23:57.08 | MFen | so your database can contain mixed versions of objects, but as soon as one finds its way into memory, it is up-to-date |
| 23:57.09 | jkakar | MFen: I see. That's cool. |
| 23:57.17 | jkakar | MFen: Yeah, nice and lazy. :) |
| 23:57.35 | jkakar | Man, jml is right, we should port Storm to Haskell. |
| 23:57.48 | pyCube | heh |
| 23:57.54 | MFen | yeah. it's a cool idea. we have an option to do either one, but our database cannot really tolerate a long "down for upgrades" so i'm leaning to per-object |
| 23:58.07 | jkakar | I see. |
| 23:58.43 | jkakar | It would be cool if storm offered a per-object onload event that you could subscribe updaters to. |
| 23:59.01 | MFen | yes, it would be cool. |
| 23:59.10 | jkakar | MFen: Have you filed a bug about the idea? |
| 23:59.51 | jkakar | MFen: If not, please do. It would be nice to hear what the perfect API looks like for what you want. |
| 23:59.54 | MFen | no, just started talking about it. one possible incompatibility with storm is that storm uses your existing tables, and axiom builds them for you |