IRC log for #gllug on 20130521

03:48.13*** join/#gllug Leeds (~richardc@42-3-157-009.static.netvigator.com)
04:55.28AndyMillarmornings
05:54.11*** join/#gllug Vyizis (~Vyizis@37.77.81.145)
06:00.08*** join/#gllug Vyizis (~Vyizis@37.77.81.145)
06:28.03ChoHagThere are 4 of us in the office of ~ 100.
06:28.06ChoHagLazy bastards.
06:31.13ChoHagOh now some cunt came and turned all the fucking lights on, then went and sat at the far end of the office.
07:06.47AndyMillarhah
07:44.27gregjresults are back
07:44.30gregjAndyMillar: :-)
07:44.36AndyMillargood or bad?
07:44.56gregjyou tell me
07:45.25gregjapparently write back is faster then write through (dah!)
07:45.47AndyMillarno surprises there :)
07:45.52gregjnot sure tho how to change that using cli
07:47.05gregjhowever SSD caching gave about 20% improvement over raid 1+0 without ssd caching
07:47.10gregjso completly worthless
07:47.24AndyMillarMegaCli64 -LDSetProp WB -lall -aall
07:47.46AndyMillargregj: try running the test 3-4 times on the cachecade volume
07:47.52AndyMillaryou might start to see an improvement
07:48.04AndyMillarand then, you'll want to run the test just on the SSDs, incase the SSDs are the bottleneck
07:48.20gregjhttp://pastebin.com/Giz8x1Ew
07:48.21AndyMillarI've got Intel 320s and OCZ Vertex 4s in mine, and they're relatively fast
07:49.14AndyMillarJobs: 3 (f=3): [mmm__] [100.0% done] [7454K/30707K /s] [1820 /7497  iops] [eta 00m:00s]
07:49.26AndyMillarcan you grab that line from the outputs, or is it gone?
07:49.39gregjwhat is the VM4k for
07:51.00gregjon ssd one Jobs: 3 (f=3): [_mmm_] [99.8% done] [932K/4368K /s] [233 /1092  iops] [eta 00m:02s]
07:51.59gregjlastly what is the VM4k group for in the ini file ?
07:52.41gregjthe blocksize. 4k is there, probably because modern drives operate using 4k sectors, right ?
07:53.20gregjI'll change number of jobs to 20, because that's more realistic of what's going to be happening here
08:03.02AndyMillarprobably because we run kvm and that uses 4k blocks - at a guess
08:03.15gregjpostgresql uses 4k blocks afaik too
08:05.27gregjapparently postgresql uses 8k blocks
08:06.50*** join/#gllug cityLights (~nivw@bzq-218-29-26.cablep.bezeqint.net)
08:11.44*** join/#gllug ess_tee_u (~NULL@host81-158-50-178.range81-158.btcentralplus.com)
08:16.15AndyMillargregj: the next thing to try is going to be SSDs only, depending on how much you care
08:17.50gregjor trying 1+0 + ssd caching
08:17.57gregjand instead of sda+sdb, just have sda
08:18.02gregjie, everything on the same drive
08:18.25AndyMillartbh, you might find the SSDs aren't fast enough
08:18.33gregjlol
08:19.11AndyMillarI've had that :)
08:19.45AndyMillarwhat model SSDs a did you have with it?
08:19.57AndyMillar(~1k IOPS is probably 10x what you see on sda)
08:24.38gregj"IBM 256GB SATA 2.5in MLC HS Enterprise Value SSD
08:24.43gregjis what I got on invoice
08:25.39AndyMillarspecced to 50k read, 7.5k write IOPS
08:26.17AndyMillar350mbps read, 140mbps write
08:26.33AndyMillarso sequential IO is unlikely to be much better than rust
08:26.44gregjok
08:26.50gregjtmie to do pgbench then :-)
08:28.55gregjcan you test random io with fio ?
08:29.03gregjaltho in general comparing it is probably not a good idea
08:30.59yaMattmorn
08:35.59AndyMillarthat test is random
08:36.30gregjok
08:37.42*** join/#gllug chrisp (~wibble@host86-157-34-217.range86-157.btcentralplus.com)
08:55.16mindbenderz00dax: yeah, what is he on about? :)
08:56.14z00daxnot sure, he's a good guy otherwise. I blame the brighton air
08:57.02mindbenderz00dax: is it expected to get told off when you help someone about linux @#centos ?
08:58.08z00daxmindbender: the regular mandate is to keep it centos specific, suse specific chatter will get the boot thrown in
08:58.29z00daxplus, you haveto give the guy his due - the problem was mostly the braind deadness of suse
08:59.31z00daxone 'interesting problem' we have is that #centos has become a place to get free consulting, free homework, and free advice on ahdoc contracting thing, whats it called
08:59.32mindbenderz00dax: yeah, I could understand if there was a big chatter going on in the channel, preventing people to read stuff but once there are only leave/join messages. I wonder why that dude was in the centos channel, If I was Trevor, I'd convert him, not tell him off :)
08:59.47z00daxelance ?
08:59.50z00daxyeah
09:00.24mindbenderz00dax: since you're around, got a question :)
09:00.33z00daxthe flip side is that when someone comes in asking basic stuff like that : most of the regular assume he is doing paid for work, and just proxuing the questions from his client to the channel
09:00.47z00daxthis theory has been proven  a couple of times :/
09:01.23z00daxsure
09:01.26mindbenderz00dax: that's strange if he gets paid for not knowing how to setup basic networking but fair enough, besides we didn't know it was suse initially
09:01.53z00daxmindbender: its mostly an attitude problem in #centos.. been trying to work on fixing it
09:01.57mindbenderz00dax: i'm just testing XFS on CentOS6 and It appears to be 6 times slower than the one on CentOS5, have you heard anything like that?
09:02.23z00daxno, that is strange. xfs on centos-6/x86_64 is a much better supported stack than it ever was on 5
09:02.24mindbenderz00dax: from what I've read, XFS on CentOS6 should be a lot faster
09:02.34z00daxsame here
09:03.40mindbenderi suppose i need to catch sandeen
09:04.18z00daxif you can define the setup, a bugs.centos.org entry would be appreciated
09:04.32z00daxthe qa guys have been pretty pro-active in getting upstream traction on stuff
09:04.56z00daxalso, the glusterfs guys seem to be taking up some of the xfs troubleshooting, since they recommend it
09:05.10mindbenderhmm
09:05.16*** join/#gllug cityLights (~nivw@bzq-218-29-26.cablep.bezeqint.net)
09:05.33mindbenderyeah, it's my last week at this work place, if I can finish it before I leave, I'll report it
09:05.54mindbenderwill check with the people in #glusterfs
09:13.01z00daxmindbender: if you want to see a $150/hr contractor get #centos to fix his client machines, now is a great time to drop in
09:15.02mindbenderz00dax: well, that's how all we started learning :p
09:15.30mindbender$150 is cheap anyway
09:15.40z00daxthis guy has been around since 2006(ish?)
09:15.46mindbenderlol
09:15.52mindbenderwell, at least he's still in the business
09:15.57mindbenderwhat's his nick?
09:16.07z00daxKaushal
09:16.49mindbenderheh yeah just seen it on nickserv
09:16.52gregjanyone with experience in configuring bind for round robin ?
09:17.09z00daxgregj: you mean beyond just adding the A's ?
09:17.18gregjlike troubleshooting
09:17.22mindbendergregj: z00dax is right :)
09:17.41mindbendergregj: you need to describe it, it's as simple as adding A records afaik
09:17.41gregjI know it is a matter of adding A's, I'm not that daft
09:17.43gregjthing is tho
09:18.11gregjsome hosts using it use always same IP, others do proper round robin
09:18.15gregjand I don't know why
09:18.22z00daxdns works as a server and a client
09:18.23mindbendergregj: what's the client OS?
09:18.30gregjthe only difference is that the ones that connect to single IP, are hooked to the same switch sa the dns server
09:18.36gregjit's all centos 6 64bit
09:18.43z00daxand may gethostbynames do random crap at the client side, so make sure its the right side you try to troubleshoot
09:18.47gregjclient is pgbench (using libpq)
09:18.53mindbendergregj: the client is centos too?
09:18.56gregjyeah
09:18.59gregjall machines are exactly same
09:19.15z00daxdo you get the right thing happen with 'dig' ?
09:19.18mindbendergregj: make sure you aren't running some sort of dns caching
09:19.32gregjI'm not
09:19.39gregjdig is showing right results (so is host)
09:19.41mindbendergregj: no avahi, nscd ?
09:19.46gregjnope, none of that
09:19.55z00daxso its libpq that is doing something wierd ( like an accidental sort )
09:20.23gregjnope, just gethostbyname()
09:20.29gregjsame version of postgresql everywhere
09:20.33gregjI'm puzzeled
09:20.37mindbendergregj: have you tried stracing it?
09:20.47z00daxyes, because you just said that gethostbyname() was doing the right thing
09:20.48gregjonly variable is that one host is going through a router (on a different subnet), rest are on the same network
09:20.59gregjones going through router work, ones on the same subnet don't
09:21.08mindbendergregj: are you a 100$/h contractor?
09:21.11z00daxplot thickens
09:21.15gregjnope :-)
09:21.16mindbender:D
09:21.24gregjmore like £20ph
09:21.32gregjI wish I had 100$ph
09:21.54gregjI'm a software developer, not an admin - but I dipped my toe in that camp too - which is why I'm doing it too
09:22.03mindbendergregj: try adding it to /etc/hosts to make sure, that's the problem, i'd try that.
09:23.20gregjthis has nothing to do with /etc/hosts
09:23.27gregjI can resolve the host in both cases
09:23.39gregjhowever, the one that is on localsubnet with the server and dns, uses only single IP
09:23.44gregjso it clearly is caching it somehow
09:23.55gregjthe one that goes via router, frmo different subnet - doesn't cache it
09:24.07mindbendergregj: yeah, my point was more like, if you add the record to resolve the one you wanted it to resolve or to the one that doesn't get resolved in /etc/hosts and run your app, may give a clue if the problem is elsewhere
09:24.41mindbendergregj: just use strace to see what it actually does then
09:25.08mindbendergregj: you may as well run wireshark after that to see how it queries
09:25.36gregjyeah
09:25.37gregjthanks
09:28.04AndyMillargregj: nscd or sssd will be caching
09:28.15AndyMillarother than that, round robin dns doesn't work as you expact
09:28.17AndyMillarexpect*
09:28.24AndyMillarand can't be relied on for load balancing
09:32.35gregjthat's what most people use for postgresql cluster balancing
09:32.42gregjor any other db for that matter
09:32.52gregjit seems to work fine, but only when the client is goign through a hop
09:32.59gregjwhich ti me is just bonkers
09:36.36mindbendergregj: not sure how a hop can be related tbh
09:37.48gregjneither do I
09:38.21gregjthere are some differences in the strace tho
09:38.31gregjbut they might have to do with timing, as I fired strace -f
09:39.04gregjthe 'remote' one is doing few more operations
09:39.08gregjlike gettimeofday(), etc
09:39.14gregjnothng that should ahve anything to do with this
09:39.29mindbendergregj: you can filter it for the network calls
09:40.01gregjtrue!
09:41.17gregjno differences there whatsoever
09:41.19gregj£$%T"£$^"£^
09:42.35gregjbut if I try multiple processes, it does not cache between them
09:42.58gregjI dont' know what the difference is, but it is some subtle difference in the logic somewhere, and has nothing to do directly with networking
09:44.07gregjspoke too soon, the scaling one is the remote one, not local
09:44.08gregjdoh
09:44.48mindbendergregj: # strace -s 500 -e connect ping google.com
09:45.07mindbendergregj: connect(4, {sa_family=AF_FILE, path="/var/run/nscd/socket"...}, 110) = 0
09:45.18mindbenderthat's what I get, maybe a similar approach?
09:45.37gregjyeah, I've seen that
09:45.42gregjthat gives -ENOTFOUND
09:45.45gregjso...
09:46.05mindbenderhuh
09:46.36mindbendergregj: not sure why it'd say that, try -e open
09:46.52gregjstrace -e open ?
09:47.40gregjexactly the same
09:48.44mindbendergregj: that can't be right, can you paste your line?
09:49.48gregjwhich one
09:50.01gregjwhen I run -e open, the nscd line doesn't appear there
09:50.23gregjconnect(3, {sa_family=AF_FILE, path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory)
09:50.43gregjit tries it twice
09:51.00mindbendergregj: i think that's fine
09:51.17gregjyeah
09:51.26mindbendergregj: whatabout open() ?
09:51.48gregjfunny thing is, if I run pgbench multiple times on the host that is listed in dns records -it always picks its own IP
09:51.53gregjso I think there is some funny logic there
09:52.08gregjnothing interesting there with open()
09:52.08mindbendergregj: i'm sure they didn't include their own networking stack :D
09:52.33mindbendergregj: not even open("/etc/resolv.conf", O_RDONLY)      = 4
09:53.25gregjhttp://pastebin.com/WtABpXFa
09:53.39gregjof course they did
09:53.43gregjbut thing is
09:53.50gregjit sends a dns request
09:53.54gregjgets back bunch of records
09:54.02gregjand somehow always picks its own IP from that list
09:54.06gregji.e. - favours it
09:54.13gregjthus breaking the RR
09:54.53gregj"The DNS protocol imposes no ordering on the resource records within a resource record set. When handling a response datagram, a DNS client (including the DNS client that forms the back-end of a proxy DNS server) is under no obligation to preserve the ordering of the resource record set as it is received. "
09:56.05mindbendergregj: that's true but on linux, it works fine (at least on centos5)
09:56.23mindbendergregj: ie. you get the other A record everytime you query something, with windows, that's not the case
09:56.45gregjthing is, bind will send different order fo A records every time
09:57.10gregjhowever, I think the code itself does some more there, and picks and favours certain IPs, such as - same subnet, or local ip
09:57.17mindbendergregj: you can try this
09:57.19mindbenderoptions timeout:1
09:57.20mindbenderoptions rotate
09:57.20mindbenderoptions attempts:1
09:57.28mindbenderthat goes into your resolv.conf
09:57.32gregjhttp://www.tek-tips.com/viewthread.cfm?qid=1307952
09:57.56mindbenderbut i suppose it only affects the dns server pick
09:57.59mindbenderneeds coffee
09:58.22gregjthat isn't fixing it mindbender
09:58.42gregjI think the extra logic is in the code itself, and does shit regardless of etc/resolv.conf
09:58.45gregjtime to dive into code
09:59.04gregjthis is where being software dev and doign some admin stuff comes handy
09:59.13gregjfrankly, I should get paid $100 ph for that
09:59.24mindbendergregj: must be using libresolv.so
10:05.06gregjAndyMillar: alternative to RR is something called pgpool2, which I tried and characterised - it doesn't scale as nicely as simple round robin
10:09.23gregj..
10:09.41gregj<PROTECTED>
10:09.44gregj..
10:09.46gregjno comment.
10:10.22mindbenderlol
10:10.30mindbenderwhere did you see that
10:10.54gregjit is fscking hidden like hell
10:11.03gregjwhoever wrote glibc is just plain evil
10:11.35mindbenderwell, it's a good thought
10:11.37mindbender:D
10:11.41gregjres_hconf.c
10:11.51gregjit gets called via macro
10:12.00gregjinnocently named POSTPROCESS
10:12.04mindbenderhow did you even get to whereabouts it is
10:12.26gregj#ifdef POSTPROCESS POSTPROCESS;
10:12.27gregj#endif
10:12.34gregjbit of tracing
10:12.39gregjand grepping
10:13.27mindbenderin glibc's source?
10:13.40gregjaye
10:14.16gregjfscking urlich drepper
10:14.56mindbenderfascinating
10:15.08mindbenderone more thing to worry about when doing RR DNS
10:15.08gregjand of course there is no option to turn it off
10:15.10gregj£$%^£&^£^"£%^
10:16.07mindbenderwhy do you have one int. and one ext. network for the job?
10:16.54gregjdiffernet networks with VMs and servers
10:17.09gregjI had only 2 test serevrs previously, so tests had to run on a vm
10:17.25gregjnow that I have more servers, I'm trying on the same subnet as the actual servers I'm testing
10:17.32gregjand ... doh!
10:18.48mindbendergregj: so, it's not a big deal then
10:19.06mindbenderi was thinking who would have the DBs on 2 different networks
10:19.18mindbenderat least for RR purposes
10:19.30mindbenderbut that'd be you. :p
10:20.56gregj<PROTECTED>
10:20.58gregjscore gold:
10:21.00gregj|#define ENV_HOSTCONF>-"RESOLV_HOST_CONF"
10:21.00gregj|#define ENV_SPOOF>"RESOLV_SPOOF_CHECK"
10:21.00gregj|#define ENV_TRIM_OVERR>-"RESOLV_OVERRIDE_TRIM_DOMAINS"
10:21.01gregj|#define ENV_TRIM_ADD>-"RESOLV_ADD_TRIM_DOMAINS"
10:21.03gregj|#define ENV_MULTI>"RESOLV_MULTI"
10:21.05gregj#define ENV_REORDER>"RESOLV_REORDER"
10:21.08gregjon/off settings !
10:21.10gregjyay
10:22.11mindbenderwhat, you're gonna re-compile it? mad.
10:22.31gregjnope, enviroment variables !
10:22.50gregjif they work
10:23.06mindbenderah didn't read that part lol
10:23.25mindbenderthat's what happens when you paste multi lines and why it's being prevented by your favorite IRC client.
10:23.29mindbenderpeople don't read this shite.
10:23.44gregjhowever turning it all off doesn't ehlp
10:23.46gregjdoh :(
10:24.08mindbendergregj: let me know if that ENV works
10:24.26gregjthey don't
10:24.31gregjat least turning all off
10:24.32gregjdoesn't
10:24.38mindbendergregj: how did you figure?
10:24.58mindbendergregj: passing the env on the same line with the binary?
10:25.08gregjI exported them one by one =off, and ran my process
10:25.16gregjexport FOO=off
10:25.22gregjsets it in the env
10:25.41mindbendergregj: hmm it doesn't spawn its own shell or such i hope
10:26.03gregjthat's why export works then, cos you inherit enviroment
10:26.05mindbendergregj: i'd be keen to try it with env + strace ping
10:26.22ChoHag<Boss> I'm going to add your private email address to the thing that spews the 1000s of alerts everybody ignored.
10:26.31ChoHag<Me> No you're fucking not.
10:27.44gregjthats' what we have filters for
10:27.46gregjand subfolders
10:28.03gregjmindbender: strace won't show you the extra logic in the code
10:28.45ChoHagAnd designated spam mailboxes.
10:29.25gregjyup
10:29.33gregjthat's where all corporate shit goes in my case
10:29.46gregjall svp shuffling that happens
10:29.51gregjI couldnt' care elss
10:30.48ChoHagI have 4 boxes here. At 2300 the database machine got too busy to respond but apparently without any external cause.
10:31.26ChoHagThere was a bit of extra chatter between them at the time but no reason for it. Nothing which would trigger anything at 11 and nothing in any of the logs I've found to indicate that something got triggered.
10:31.26gregj<PROTECTED>
10:31.45gregjwhat sort of db ?
10:31.54ChoHagmysql
10:32.03gregjyeah, it does weird shit
10:32.11ChoHagThe support weenies kicked it and it was fine.
10:35.05mindbendergregj: host.conf is a sensible approach than env variables :D
10:36.28gregjyeah, but it still doesn't work however
10:36.30gregjreboot might help
10:36.32gregjwill see
10:37.22gregjnope
10:37.24gregjhopeless...
10:44.00mindbenderll
10:44.12mindbenderreboot it and grab a coffee
11:20.25*** join/#gllug coppull (~h@181.sub-70-215-11.myvzw.com)
11:56.22morsingcoppull!
11:56.54coppull?
11:57.04mindbendercoppull!
11:57.43coppullum
12:01.50coppullwhats the bang for ?
12:02.02coppullim guessing im dense
12:03.14Leedsexcitement...
12:09.06coppullok here i am thinking its some kind of special irc thing lol  ....yep im dense
12:09.21coppullin that  case  hello!
12:10.43Leedsgood evening!
12:12.07coppullDays just starting here :( Morning
12:12.41coppullwish I was in the right time zone then I could get a pint
12:18.11coppullInstead its coffee and AIX punishment for me
12:20.58*** join/#gllug dick_turpin (~peter@host217-34-163-30.in-addr.btopenworld.com)
12:33.36*** join/#gllug cityLights (~nivw@bzq-218-29-26.cablep.bezeqint.net)
12:38.43morsingdick_turpin!
12:38.45morsingcityLights!
12:38.57cityLightsyep
12:39.08morsingcoppull: Where do you work?
12:40.11coppullNY state
12:42.24dick_turpinmorsing: --+-O-+--
12:44.30morsingcoppull: Which company?
12:44.52morsingdick_turpin: How's Sharon?
12:44.56coppullGov
12:45.42coppullnothing 16 floors below ground mind you
12:47.00morsing?
12:50.23dick_turpinmorsing: Working in a Saudi brothel. :-(
12:51.13coppullbetter hide yer booze
12:57.42*** join/#gllug GNUdog (~GNUdog@sakura.voidcode.com)
13:00.27morsingGNUdog!
13:00.29morsingcoppull: ?
13:00.33gregjlol
13:13.47coppullmorsing: when I say below ground... usually when IT people in the US say X amount of floors below ground or in mountains they mean NSA DOD type of stuff...That I do not do ;)
13:21.44morsing...? NFA-what?
13:22.53coppullhttp://www.nsa.gov/
13:23.49coppullwhat im saying is i work for state gov not those guys ;)
13:27.04morsingOk
13:27.13morsingcoppull: Why GLLUG?
13:27.53morsingcoppull: And also, do you know Zach?
13:28.33coppullLinux fan and im a Brit
13:28.45coppullI do not know Zach
13:38.55*** join/#gllug funkyHat (~m@funkyhat.org)
13:39.50dick_turpinAre you Brit Eklnad?
14:00.40*** join/#gllug funkyHat (~m@funkyhat.org)
14:10.37*** join/#gllug funkyHat (~m@funkyhat.org)
14:16.50morsingfunkyHat: STOP IT!
14:17.40funkyHathrm
14:18.30funkyHatNot sure what's going on
14:19.06funkyHatDoesn't seem to be my usual irssi failure mode either (where my silly scripts manage to start 2 instances of it which then battle for my nick)
14:48.13morsingfunkyHat: Erm... Do you like doing things the hard way? A bit of S&M?
14:50.19funkyHatmorsing: mostly it's nice and good and easy (log in to special user account, irssi screen session automatically attaches or is spawned if it wasn't running)
14:51.10funkyHatI think today my issue may be transient network problems on the VPS host, though, rather than problems with my stuff
14:51.36funkyHatOr were you referring to irssi in general? ;P
14:54.25toxboiDid anyone experienced power cut in Central London like 15 minutes back?
14:56.04yaMattnot here, what part?
14:56.24toxboiOld Street/Moorgate
15:15.15morsingfunkyHat: I am asking because I was reading about your Apache -> lighthhtpd
15:16.04funkyHatmorsing: oh right!
15:17.00funkyHatmorsing: lighttpd's config is much cleaner than apache's, and my drupal config generation thing is really handy
15:17.12*** join/#gllug chrisp (~wibble@host86-180-144-97.range86-180.btcentralplus.com)
15:17.18morsingchrisp!
15:19.53funkyHatJust reloading lighttpd's config sets up new drupal vhosts based on which directories are in the sites directory
15:39.52*** join/#gllug coppull (~h@181.sub-70-215-11.myvzw.com)
15:41.40chrispmorsing af'noon
15:43.02yaMatt'noon
16:12.35*** join/#gllug {mikey} (~mikey@146.185.18.114)
16:13.01*** part/#gllug dick_turpin (~peter@host217-34-163-30.in-addr.btopenworld.com)
16:13.42{mikey}hi
16:16.26{mikey}I've just uploaded a script to my blog which I find extremely useful with my day to day Linux life if anyone's interested
16:27.00{mikey}http://www.green-spot.co.uk/2013/05/21/storing-and-searching-code-snippets-with-metadata/
17:52.47*** join/#gllug Armand (~martin@87-194-165-154.bethere.co.uk)
18:08.43*** join/#gllug funkyHat (~m@funkyhat.org)
18:29.43*** join/#gllug funkyHat (~m@funkyhat.org)
21:33.09*** join/#gllug {mikey} (~mikey@cpc3-dals17-2-0-cust54.hari.cable.virginmedia.com)
23:49.05*** join/#gllug Leeds (~richardc@42-3-157-009.static.netvigator.com)

Generated by irclog2html.pl Modified by Tim Riker to work with infobot.