| 00:40.33 | *** join/#gllug Leeds (n=richardc@n219073033121.netvigator.com) |
| 02:41.50 | *** join/#gllug Leeds (n=richardc@static-ip-251-82-134-202.rev.dyxnet.com) |
| 05:12.19 | *** join/#gllug shai (n=Shai@l192-117-110-233.cable.actcom.net.il) |
| 05:55.10 | *** join/#gllug sabinef72 (n=sabinef7@ns.popipo.fr) |
| 06:59.19 | *** join/#gllug mozrat (n=sm@194.203.40.135) |
| 07:42.28 | *** join/#gllug DiscordianUK (n=ch@89.243.188.25) |
| 10:33.17 | zeroXten | man, shipping only within US? =( |
| 10:33.24 | zeroXten | oops, wrong window |
| 10:33.46 | yaMatt | yes, i am |
| 10:35.10 | bilarh | bob the builder |
| 10:35.57 | zeroXten | heh |
| 11:02.27 | *** join/#gllug DiscordianUK (n=ch@89.243.188.25) |
| 11:05.40 | *** join/#gllug celesteh (n=celesteh@sblug/member/celesteh) |
| 11:32.35 | AndyMillar | cpufreak: what are the lead times on those hetzner servers? |
| 11:36.34 | cpufreak | a week or so usually |
| 11:36.40 | AndyMillar | kk |
| 11:38.16 | hali | hello |
| 11:40.16 | hali | fish&chips friday |
| 11:40.20 | cpufreak | yes |
| 11:40.32 | cpufreak | although its normally pub lunch friday here |
| 11:47.37 | bilarh | damn ergo keyboards |
| 12:09.39 | AndyMillar | ah cock |
| 12:09.45 | AndyMillar | do we have any redhat cluster fans here? |
| 12:17.08 | cpufreak | RAC or redhat cluster? |
| 12:17.31 | AndyMillar | redhat cluster |
| 12:25.53 | cpufreak | not for a long time |
| 12:26.14 | cpufreak | we use either vcs or rac here. |
| 12:33.21 | AndyMillar | meh, one of my clusters just rebooted |
| 12:33.22 | AndyMillar | :/ |
| 12:46.16 | *** join/#gllug thebrother (n=jon@cpc2-cmbg13-0-0-cust353.cmbg.cable.ntl.com) |
| 12:51.36 | AndyMillar | and nothing looks wrong, apart from it losing the quorum disk |
| 13:00.19 | *** join/#gllug DiscordianUK (n=ch@89.243.188.25) |
| 13:30.09 | hali | that can usually upset things quite a lot |
| 13:43.11 | AndyMillar | aye |
| 13:43.14 | AndyMillar | thing is, I can't see how |
| 13:43.37 | AndyMillar | the MSA1000 (with dual controller) is attached via 2 FC switches |
| 13:43.45 | AndyMillar | each node has diverse fibre to it |
| 13:43.58 | AndyMillar | both FC switches report no problems and have been up for weeks |
| 13:44.10 | AndyMillar | both controllers in the MSA have been up for 90+ days and have no complaints |
| 13:44.33 | AndyMillar | I've tested the redundancy extensively (randomly unplugging things and rerouting fibre) |
| 13:44.37 | AndyMillar | and nothing broke |
| 13:52.29 | cpufreak | have you got anything IBM there? |
| 13:52.35 | cpufreak | I blame IBM for all failures. |
| 13:54.57 | AndyMillar | no :( |
| 13:54.59 | AndyMillar | HP Servers |
| 13:55.05 | AndyMillar | HP FC Switches |
| 13:55.07 | AndyMillar | HP MSA1000 |
| 13:55.34 | hali | did both nodes loose the quorum disk? |
| 13:55.39 | AndyMillar | 3 nodes |
| 13:55.41 | AndyMillar | and yes |
| 13:55.43 | AndyMillar | all 3 did |
| 13:55.46 | AndyMillar | exactly the same time |
| 13:55.50 | AndyMillar | and got it back at the same time |
| 13:56.12 | hali | i'd probably blame the MSA |
| 13:56.30 | AndyMillar | qdiskd[5321]: <warning> qdisk cycle took more than 1 second to complete (4.880000) |
| 13:56.34 | AndyMillar | I get that a bit |
| 13:56.40 | AndyMillar | 22:40:12 |
| 13:56.50 | AndyMillar | 22:46:14 I lose quorum disk |
| 13:58.39 | AndyMillar | hmm, it looks like the quorum disk is timing out a *lot* |
| 13:58.49 | hali | https://bugzilla.redhat.com/show_bug.cgi?id=490147 |
| 13:59.18 | AndyMillar | nothing got fenced :/ |
| 13:59.24 | AndyMillar | all 3 nodes stayed up |
| 13:59.30 | AndyMillar | the cluster dissolved then came back up |
| 13:59.38 | AndyMillar | and rebooted all domUs |
| 14:00.05 | hali | do you have sar running? |
| 14:00.14 | hali | i'd look at general io stats for the time |
| 14:00.21 | AndyMillar | opennms monitors it |
| 14:00.25 | AndyMillar | nothing special happened then |
| 14:00.33 | AndyMillar | oho |
| 14:00.34 | AndyMillar | actually |
| 14:00.37 | AndyMillar | i might have been stupid |
| 14:00.38 | AndyMillar | 2s |
| 14:00.50 | AndyMillar | i/o will be per node on it, not for the entire msa |
| 14:01.06 | AndyMillar | hali++ |
| 14:02.47 | AndyMillar | ok, no |
| 14:04.23 | AndyMillar | https://www.andymillar.co.uk/temp/node1.png |
| 14:04.24 | AndyMillar | https://www.andymillar.co.uk/temp/node2.png |
| 14:04.26 | AndyMillar | https://www.andymillar.co.uk/temp/node3.png |
| 14:04.48 | AndyMillar | oh |
| 14:04.51 | AndyMillar | i'm being a retard again |
| 14:06.38 | AndyMillar | ok, yes, increased i/o at that time :/ |
| 14:07.23 | cpufreak | ooi which scheduler are you using? |
| 14:08.06 | AndyMillar | whatever's default |
| 14:08.43 | hali | cfq |
| 14:08.50 | hali | if you are on a redhat box, i think |
| 14:09.02 | cpufreak | what is the box doing? |
| 14:09.20 | hali | xens? |
| 14:09.36 | AndyMillar | idd |
| 14:09.37 | AndyMillar | xen |
| 14:09.54 | AndyMillar | ok, every machine kicked off something at around the same time |
| 14:10.06 | AndyMillar | and it hammered reads |
| 14:18.50 | AndyMillar | oh, no |
| 14:18.56 | AndyMillar | that's idiotic |
| 14:19.06 | AndyMillar | that spike i'm seeing will be the stupid things stating up |
| 14:19.08 | AndyMillar | starting* |
| 14:22.13 | *** join/#gllug [Discordian] (n=ch@92.24.81.149) |
| 14:22.15 | AndyMillar | grmbls |
| 14:42.26 | zeroXten | this is better than eastenders |
| 14:42.57 | zeroXten | good plot twists |
| 14:48.12 | AndyMillar | ;p |
| 14:48.59 | AndyMillar | so, I can't see anything that could cause a problem |
| 14:49.00 | AndyMillar | :/ |
| 14:51.44 | *** join/#gllug Leeds (n=richardc@n219073033121.netvigator.com) |
| 14:55.37 | AndyMillar | any of you miserable people going to this opentech thing tomorrow? |
| 14:56.06 | hali | im going to gay pride |
| 14:56.39 | bilarh | to throw stones and bottles? :S |
| 16:42.06 | Leeds | listening to the wimblington |
| 16:50.44 | zeroXten | is listening to people watching the wimblington |
| 16:51.08 | zeroXten | either that or there are a few orgies going on around here |
| 16:52.27 | Leeds | Oooh!!! Aaaah!!! |
| 17:00.49 | zeroXten | pints \o/ |
| 18:22.32 | *** join/#gllug celesteh (n=celesteh@sblug/member/celesteh) |
| 18:37.46 | *** join/#gllug shai (n=Shai@l192-117-110-233.cable.actcom.net.il) |
| 21:11.14 | *** join/#gllug celesteh (n=celesteh@sblug/member/celesteh) |
