| 01:23.38 | *** join/#asterisk-dev infobot (ibot@rikers.org) |
| 01:23.38 | *** topic/#asterisk-dev is Asterisk Development Discussion -=- http://www.asterisk.org/developers -=- Tier 2 and 3.14159265 support is in #asterisk -=- Check out our blog! blogs.asterisk.org -=- Follow on Twitter at @AsteriskDev |
| 02:17.03 | *** join/#asterisk-dev seanbright (~sean@asterisk/community-developer/seanbright) |
| 02:17.03 | *** mode/#asterisk-dev [+o seanbright] by ChanServ |
| 08:15.03 | *** join/#asterisk-dev tzafrir (~tzafrir@local.xorcom.com) |
| 08:15.03 | *** mode/#asterisk-dev [+o tzafrir] by ChanServ |
| 08:41.54 | *** join/#asterisk-dev Worldexe (~Worldexe@95-107-33-134.dsl.orel.ru) |
| 10:56.56 | *** join/#asterisk-dev jkroon__ (~jkroon@165.16.204.168) |
| 19:03.16 | seanbright | coreyfarrell: so i hopped over to #sqlite and asked about thread safety, because i was unclear from their documentation |
| 19:03.41 | seanbright | essentially, a 'sqlite3 *' is safe to use from multiple threads |
| 19:04.14 | seanbright | assuming you don't explicitly turn off some of sqlite's internal synchronization mechanisms, which we don't |
| 19:04.45 | gtjoseph | that's why i was considering using it as an in memory db |
| 19:04.54 | seanbright | now, the way that we use prepared statements (sqlite_stmt *) is not thread safe. |
| 19:05.05 | gtjoseph | heh |
| 19:05.14 | seanbright | well, let me rephrase that |
| 19:05.20 | seanbright | we are currently thread safe |
| 19:05.34 | seanbright | we can't just turn off the big-damn-lock and be thread safe, however |
| 19:05.36 | gtjoseph | because of the "big" lock |
| 19:05.44 | seanbright | right |
| 19:05.59 | seanbright | but, we could create prepared statements per thread and we would be fine |
| 19:06.13 | seanbright | so that is the approach i am going to take. TLS storage for statement objects |
| 19:06.35 | seanbright | also going to remove the "db_sync()" nonsense |
| 19:06.39 | gtjoseph | makes sense. what's the overhead of not using prepared statments at all? |
| 19:06.46 | seanbright | there's no such thing in sqlite |
| 19:07.08 | gtjoseph | oh yeah. i forget the api. |
| 19:07.13 | seanbright | there is no ad hoc querying. if you want to query, you call sqlite3_prepare |
| 19:07.16 | seanbright | which is lamesies |
| 19:07.22 | seanbright | but whatever |
| 19:08.15 | seanbright | i think having stuff be in TLS will be a huge win, especially when loading modules |
| 19:08.21 | seanbright | because everything is happening in one thread |
| 19:08.28 | seanbright | but i dunno... we'll see |
| 19:08.48 | gtjoseph | th eother thing you could do is create a "cache" of prepared statments for all threads to access and if there's an instance available, great, if not, create one and store it back to the cache |
| 19:09.33 | gtjoseph | that way if there 100 threads they all don't need every statment in TLS |
| 19:09.33 | seanbright | ehhhh |
| 19:09.38 | coreyfarrell | I think anything will be a win over a global mutex. ^^^^ yeah I was wondering the same about using a cache.. |
| 19:09.54 | seanbright | well, it's not actually stored in TLS right? i mean, a pointer is. |
| 19:10.09 | seanbright | but i see your point |
| 19:10.17 | gtjoseph | each statement has to be allocated somewhere |
| 19:10.27 | seanbright | i wouldn't want the cache to become another choke point |
| 19:10.52 | gtjoseph | nah, even with a lock, it's be super fast. |
| 19:11.10 | coreyfarrell | the cache have an rwlock that would have much shorter time locked compared to sqlite3 write and sync. |
| 19:11.38 | seanbright | has anyone experimented with lock striping? specially with ao2_containers? |
| 19:11.47 | seanbright | specifically* |
| 19:11.49 | coreyfarrell | or sorry the cache would still have a mutex but the time under lock would be much shorter. |
| 19:11.57 | gtjoseph | not me |
| 19:12.31 | coreyfarrell | once I make coffee I'll have to read about lock striping.. be back in 10 |
| 19:12.38 | gtjoseph | heh, you could use a sqlite3 table as the lock table with a long as the data column (pointer to statement) :) |
| 19:12.43 | seanbright | NO |
| 19:12.44 | seanbright | heh |
| 19:13.09 | seanbright | i am going to start simple with a TLS statement factory |
| 19:13.15 | seanbright | we can add a cache in once we've tested |
| 19:13.19 | gtjoseph | makes sense :) |
| 19:15.52 | file | seanbright: the voice of sanity. |
| 19:17.13 | seanbright | heh |
| 19:17.23 | seanbright | i've never been accused of that before |
| 19:18.09 | file | Dear Amazon, I do not want you to gift wrap some AA batteries and shower cleaner. |
| 19:27.52 | seanbright | gtjoseph: i lied, there is a sqlite3_exec() command, but it just wraps sqlite3_prepare and friends |
| 19:28.24 | gtjoseph | well, depending on the cost of the prepare it might be worth letting sqlite3 just do it. |
| 19:28.57 | gtjoseph | sometimes things we think are expensive really aren't |
| 19:29.23 | gtjoseph | especially compared to the thinks we have to do to compensate for "expensive" things |
| 19:29.59 | gtjoseph | not saying you should change your approach though |
| 19:30.26 | coreyfarrell | worth evaluating, TLS isn't free.. |
| 19:31.07 | seanbright | well, my first approach was to open a new sqlite * for each call |
| 19:31.11 | seanbright | which was not a good idea |
| 19:31.26 | seanbright | i don't have a problem testing preparing the statement each go around |
| 19:32.02 | coreyfarrell | wouldn't a new sqlite * per call be the same as using the same DB file from multiple programs (corruption)? |
| 19:32.49 | seanbright | you can use the same DB file from multiple programs |
| 19:32.51 | gtjoseph | i don't remember if sqlite caches prepared statments itself, it may. i know some postgresql drivers do cache prepared statements and reuse them if it finds a cache hit |
| 19:33.28 | seanbright | https://sqlite.org/threadsafe.html |
| 19:33.52 | gtjoseph | i think multiple process access is dependent on filesystem lock granularity, no? |
| 19:34.15 | seanbright | dude i dunno |
| 19:34.18 | seanbright | can i just write this code? |
| 19:34.19 | seanbright | heh |
| 19:34.39 | gtjoseph | sorry, go for it! :) |
| 19:34.56 | file | seanbright: Just Do It(tm) |
| 19:38.45 | *** join/#asterisk-dev tzafrir (~tzafrir@62-90-199-170.barak.net.il) |
| 19:38.45 | *** mode/#asterisk-dev [+o tzafrir] by ChanServ |
| 20:09.33 | seanbright | so prepping the statement each call is slow... |
| 20:09.52 | seanbright | the perftest unit test takes 100 seconds to run |
| 20:09.53 | seanbright | heh |
| 20:10.09 | seanbright | normally takes sub-1 second i believe |
| 20:13.01 | file | ouch |
| 20:13.54 | coreyfarrell | nice. so are you trying TLS or shared statement caches? |
| 20:13.58 | gtjoseph | yeah, that's a result i think. |
| 20:37.42 | seanbright | first test was just to create the prepped statement each call |
| 20:37.49 | seanbright | now testing with TLS and it's still super slow |
| 20:37.52 | seanbright | so something isn't right |
| 20:39.58 | seanbright | is ast_threadstorage_get() slow by its nature? |
| 20:41.30 | file | dunno, it's a thin wrapper around pthread stuff so that could be slow |
| 20:42.51 | seanbright | sub millisecond time. weird. |
| 20:50.43 | seanbright | well, the wind is out of my sails. TLS is almost as slow as prepping the query each time. |
| 20:50.53 | seanbright | maybe some guacamole will help |
| 20:59.10 | coreyfarrell | seanbright: if you want to post what you have I can take a look |
| 21:00.03 | *** join/#asterisk-dev tuxd00d (~tuxd00d@ip72-222-145-83.ph.ph.cox.net) |
| 21:13.27 | seanbright | i think it's fairly straight forward |
| 21:15.55 | seanbright | so it seems like the db_sync() stuff is actually beneficial from a performance standpoint... which is odd |
| 21:26.02 | seanbright | actually, no, that does make sense... because the sync thread is basically controlling when data is committed to the database. because everything is happening within a transaction and the sync thread is only periodically committing |
| 21:26.30 | seanbright | with my approach, autocommit is on, is every write is actually going to disk |
| 21:53.33 | *** join/#asterisk-dev tuxd00d (~tuxd00d@ip72-222-145-83.ph.ph.cox.net) |