Tutorial: Using Storm with Nevow¶
Note: There is work on integrating Storm with Twisted (storm/twisted-integration). Until then, the best way is probably to use deferToThread.
Before a tutorial can be written on how to use storm in Nevow, someone needs to figure out ‘best practices for using Storm in asynchronous codebases.’ From a chat in #storm on freenode (11 July 2007):
[5:13pm] dialtone: storm API is fine, one just needs to see if: a) it's thread safe and b) objects returned can be modified in a different thread [5:13pm] dialtone: and then committed in another one yet [5:13pm] radix: dialtone: It is certainly thread-safe, but you should not use the same Store in multiple threads [5:14pm] dialtone: how do you do with deferToThread then? [5:14pm] radix: dialtone: I don't know. [5:14pm] radix: dialtone: Probably you don't load the objects until you get into the other thread. [5:14pm] dialtone: that's sort of a problem probably, unless there's an alternative solution [5:15pm] radix: dialtone: Nobody has figured out best practices for using Storm in asynchronous codebases yet. [5:15pm] dialtone: ouch :( [5:15pm] radix: dialtone: (That is, assuming your application actually wants to make the database operations asynchronous) [5:16pm] dialtone: I can accept it being synchronous, but run in a different thread using deferToThread [5:16pm] radix: dialtone: for example, it would be very reasonable to use Storm like Axiom: with SQLite, just blocking on database interaction [5:16pm] dialtone: radix: yes well... but that comes with a lot of restrictions in itself [5:17pm] radix: dialtone: I even have a Twisted server which uses Storm to talk to postgres in a totally blocking manner [5:17pm] radix: but it's not a very typical server [5:17pm] dialtone: sqlite doesn't really scale that well with high concurrency and scaling a shared sqlite instance is pretty hard [5:17pm] dotz: dialtone: What's wrong with 'ceate a high level API for data manipulation and retrival, use some rpc mechanism, like pb, and make the hi-level database backend another process'? [5:18pm] dialtone: dotz: that' waaay more effort than what I'm currently doing [5:18pm] radix: dialtone: I'm sure there are some clever things that can be done to use Storm in asynchronous contexts, and I'm fairly interested in learning about them [5:18pm] dotz: dialtone: and what are you currently doing? [5:18pm] dialtone: as I said: I'm using the query builder from sqlalchemy to build and execute queries that return resultsets with a dict-like API [5:19pm] dialtone: I simply write the queries and use the dict-like APi in nevow [5:19pm] dialtone: and I run the functions in a separate thread with deferToThread [5:20pm] dialtone: radix: maybe writing a store class that keeps track of which threads owns which connection and then reuse the connection when needed [5:20pm] dialtone: this way you could share the store object and run queries independently of the context [5:20pm] radix: dialtone: maybe. I'm not sure it'd be that simple.