IRC log for #tomcat on 20130416

00:23.07*** part/#tomcat alx- (~alx-@69.165.240.218)
00:53.36*** join/#tomcat revattar (~revattar@c-98-248-187-174.hsd1.ca.comcast.net)
01:35.04*** join/#tomcat acidjnk22 (havenone@p4FDFDA25.dip.t-dialin.net)
02:10.13*** join/#tomcat robottinosino_ (~robottino@255.Red-2-138-158.dynamicIP.rima-tde.net)
02:38.02*** join/#tomcat revattar1 (~revattar@c-98-248-187-174.hsd1.ca.comcast.net)
03:28.59*** join/#tomcat revattar (~revattar@c-98-248-187-174.hsd1.ca.comcast.net)
04:03.23*** join/#tomcat nevada-smith (~nevada-sm@S0106c8fb2641d848.vs.shawcable.net)
04:20.14*** join/#tomcat revattar (~revattar@c-98-248-187-174.hsd1.ca.comcast.net)
05:19.24*** join/#tomcat vigneshwaran (~vigneshwa@182.71.239.158)
05:40.05*** join/#tomcat rhonabwy (~rhonabwy@95.157.57.154.dynamic.cablesurf.de)
06:08.15*** join/#tomcat chkal (~chkal@217.194.235.13)
06:12.27*** join/#tomcat papegaaij (~papegaaij@topicus-leeuwenbrug-33-deventer.xs4all.nl)
06:15.28*** join/#tomcat andre2g (~andre2g@dslb-088-074-218-068.pools.arcor-ip.net)
06:28.36*** join/#tomcat mturk (~mturk@18-216.dsl.iskon.hr)
06:28.37*** join/#tomcat mturk (~mturk@redhat/jboss/mturk)
06:30.00*** join/#tomcat vigneshwaran (~vigneshwa@182.71.239.158)
07:08.55*** join/#tomcat alcir (~alcir@fw.ftgm.it)
07:34.14*** join/#tomcat _Qman (~Qman@213.177.160.238)
07:45.01*** join/#tomcat jnhghy (~jalexandr@86.123.191.6)
09:41.49*** join/#tomcat mturk (~mturk@18-216.dsl.iskon.hr)
09:41.50*** join/#tomcat mturk (~mturk@redhat/jboss/mturk)
09:44.09*** join/#tomcat Reddy (~laxman@118.102.128.225)
10:22.40_QmanHello, can we configure Tomcat 6.0.36 to authenticate against multiple different OU with LDPA?
10:22.54*** join/#tomcat hzilla (~hzilla@41.178.136.196)
10:23.49hzillahello. i'm wasting my time here trying to get something as basic as app authentication working in tomcat7. I've got a tomcat installation at /etc/tomcat7 and tomcat-users.xml in that dir along with server.xml
10:23.52_Qmanmeaning a user may be located in ou=Users,dc=my,dc=domain,dc=com when another is located in ou=italy,dc=my,dc=domain,dc=com
10:24.33hzillatomcat-users.xml has a role and user specified
10:25.42hzillai'll pastie it
10:31.49hzillahttp://pastie.org/7598242 - first problem i don't understand is taht it doesn;t look like the users file is loaded. i've garbled the xml in the file to test if the server restart fails but it starts fine
10:32.04hzillaplease help
12:15.34*** join/#tomcat Karel| (~Karel@94-226-237-238.access.telenet.be)
12:28.24*** join/#tomcat xortim (~tim@167.246.61.8)
12:30.15Karel|what is the best way to do a proper analysis of the performance of a tomcat servlet?
12:30.32Karel|I have a servlet that quickly fills up the available RAM, and then drops back to practically 0 afterwards, every few minutes
12:30.38Karel|I assume that's because of garbage collection
12:31.51xortimi'm not a tomcat dev, but that's very likely to be garbage collection
12:32.14xortimtry using JMeter or installing NewRelic (they have a trial) to get a better look at what's going on in there
12:33.27Karel|I use JMeter to stress test the servlet to get that behaviour
12:33.43xortimtry tossing newrelic into the mix
12:33.44Karel|the only thing it tells me is raw memory, load, thread numbers
12:33.51Karel|alright, I'll check it out
12:34.07xortimanother option is dynatrace
12:34.15xortimnot sure if there's a trial available for that on
12:34.17xortim*one
12:36.02xortimalso, depending on the JVM you're using you can turn on verbose garbage collection which should vomit out a lot more data to native_stderr.log
12:48.37*** join/#tomcat pradhand (~pradhand@142.177.89.39)
12:55.13*** join/#tomcat pradhand_ (~pradhand@142.177.122.31)
12:57.18Karel|wow xortim
12:57.22Karel|that newrelic program is like magic
12:57.26Karel|it tells me everything
12:57.51*** join/#tomcat demetan (~chatzilla@gw1.ciep.fr)
12:57.52*** join/#tomcat demetan (~chatzilla@unaffiliated/demetan)
13:01.45*** join/#tomcat pradhand_ (~pradhand@142.177.121.208)
13:02.51xortimglad it could help
13:03.07xortimlooks like the lite version helps mostly with triage
13:03.25xortimunfortunately there's no instrumentation profiling that's FOSS
13:06.45xortimand try out some of the oracle tools for java analysis http://docs.oracle.com/javase/6/docs/technotes/tools/share/jhat.html
13:07.04xortimhttp://docs.oracle.com/javase/7/docs/technotes/samples/hprof.html
13:07.29*** join/#tomcat pradhand_ (~pradhand@142.177.41.11)
13:10.51*** join/#tomcat demetan (~chatzilla@gw1.ciep.fr)
13:10.52*** join/#tomcat demetan (~chatzilla@unaffiliated/demetan)
13:11.15xortimhttp://visualvm.java.net/  might also be useful if you're worried about giving newrelic your data
13:11.30Karel|so far I haven't encountered any limitations to what I can do with the free version
13:11.48xortimdata retention and where the data is stored has been the biggest point of contention with my clients
13:25.20Karel|ah
13:32.03*** join/#tomcat pradhand (~pradhand@142.177.89.99)
13:50.01*** join/#tomcat _Qman (~Qman@213.177.160.238)
13:57.53*** join/#tomcat pradhand (~pradhand@142.177.90.114)
14:16.24LzrdKingok, i have two idle identical tomcats on two identical servers, so why is one almost continuously GCing?
14:17.06LzrdKingViaual GC just tells me "Last Cause: unknown GCCause"
14:17.21LzrdKingVisual
14:17.44xortimunder the same load? (check the load balancer) settings. Also check that the JVM settings are the same on both
14:18.34LzrdKingload is 1 on one and 1.5 on the other, identical JVM settings
14:19.01LzrdKingthe one with the 1.5 load is the one thats GCing so much
14:19.46xortimmight be a silly question but is the code the same on each?
14:20.01LzrdKingthere is no code
14:20.08LzrdKingoh wait, maybe one has the silly default apps
14:20.30xortimcode meaning the webapps, deployable artifacts, etc.
14:20.42xortimmdf5sum ${tomcat_home}/webapps/*.war
14:20.46LzrdKingno, that wasn;t it, there is only the root app
14:20.50LzrdKingno wars
14:21.05xortimyou have tomcat idling on both servers?
14:21.06LzrdKingthats what i meant by idle, empty
14:21.11LzrdKingyup
14:21.34LzrdKingthe one is gcing almost exactly once an hour
14:22.08xortimbest bet is to look at the garbage collection logs, heap and core dumps
14:22.24LzrdKingthe other is like every 30 seconds
14:22.26xortimhttp://wolfe.id.au/2011/10/16/monitoring-the-openjdk-from-the-cli/
14:22.39xortimthat has great resources
14:22.58xortimi believe one of the tools allows you to attach to a given pid and see what's going on
14:23.34LzrdKingi have the tools
14:25.06Karel|alright, so I pinpointed the major source of CPU usage in my tomcat to DriverManagerDataSource.getConnection(), which initializes the database connection
14:25.10xortimnot sure. if it's just a basic installation of tomcat on two identical servers (same OS level, Java Version, architecture, physical resource such a RAM, etc.) then check to make sure that there's nothing else running that's preventing java from allocating the same amount of ram
14:26.02xortimKarel| - that's seems weird. is it not reusing connections?
14:26.15Karel|hmm it probably is not
14:26.41Karel|you got experience with spring?
14:26.51xortimnot really, no
14:26.59Karel|Simple implementation of the standard JDBC DataSource interface, configuring a plain old JDBC Driver via bean properties, and returning a new Connection for every getConnection call.
14:26.59Karel|NOTE: This class is not an actual connection pool; it does not actually pool Connections. It just serves as simple replacement for a full-blown connection pool, implementing the same standard interface, but creating new Connections on every call.
14:27.04xortimmostly a middleware admin
14:27.04Karel|that might explain it
14:27.18LzrdKingok, the one that is GCing more frequently has committed 8MB to eden, the other has committed 62GB
14:27.31xortimeden?
14:27.39LzrdKingnew gen
14:28.35LzrdKingmax is 1.3GB on each, i'm letting it autotune the heaps so it seems it has done so very differently on each
14:28.47*** join/#tomcat jieryn (~jieryn@pool-173-77-215-221.nycmny.fios.verizon.net)
14:28.48*** join/#tomcat jieryn (~jieryn@jenkins/developer/jieryn)
15:04.42*** join/#tomcat revattar (~revattar@c-98-248-187-174.hsd1.ca.comcast.net)
15:14.13*** join/#tomcat gcristian (~havok@mail.bixti.com.ar)
15:14.50*** join/#tomcat AJC_Z0 (~AJ_Z0@pdpc/supporter/professional/ajz0)
15:23.55LzrdKingi restarted both tomcats with -Xmn64m and they both seem a lot better
15:33.03_QmanHello, anyone can assist with setting LDAP authentication in Tomcat where the users are spread around multiple OU?
15:39.37xortimmultiple OUs are usually tricky no matter what. set the search base to the lowest common denominator and come up with a reasonable LDAP query for the OUs  (|(ou=one)(ou=two))
15:39.40xortimso on and so forth
15:40.40xortimtest your query against LDAP with JXExplorer or ApacheDS
15:41.06xortimget the results you want and make them work with the tomcat config
15:48.25*** join/#tomcat pradhand (~pradhand@142.177.121.38)
15:48.34LzrdKingno, i take that back, the 01 server is STILL GCing more frequentrly than 2, but a lot less often than before
15:57.32*** join/#tomcat pradhand_ (~pradhand@142.177.17.199)
16:05.44LzrdKing7 collections in the time it took the other to have 3
16:09.59LzrdKingooh, MaxThreads is different on the http conenctor, could that be part of the difference?
16:10.44whartungis the traffic at all different between the two servers LzrdKing
16:11.34LzrdKingthere is no traffic
16:11.42LzrdKingexcept jmx
16:11.58whartungso this is just idle GCing?
16:12.02LzrdKingyup
16:12.22LzrdKingi can see eden filling up faster on the one in visualgc than the toher (and zabbix, too
16:12.27LzrdKingother*
16:15.42*** join/#tomcat dknox (~dknox@66.109.209.21)
16:16.55LzrdKingi set both to the same number of threads, lets see if that matetrs
16:16.58LzrdKingmatters*
16:17.42whartungit'll matter perhaps "a little", it depends on whether those threads wake up at all. If you had "twice as many" and they wake up regularly to "do nothing", then you'd have twice as much garbage.
16:18.01LzrdKingone was 200, the other was 450
16:18.35whartungthat could explain it. Mind, I think your thread numbers are ..um.. insane, but I don't know what kind of traffic you guys drive.
16:18.55LzrdKingisn't 200 the default?
16:19.06whartungI dunno -- seems awfully high to me.
16:20.10whartungMy T7 install has the setting commented out.
16:20.41LzrdKingwhat, MaxThreads?
16:20.46whartungyes
16:20.58LzrdKingthat is is using the default for sure :)
16:21.01LzrdKingit is*
16:21.30whartungwell the commented out Executor tag has 150 assigned. I doubt the default is more than that, but I can't say.
16:21.43whartungI'd assume it was more like 10
16:22.03LzrdKingblah, frist gc is about to happen on 01, 02 still has a way to go
16:22.15*** join/#tomcat revattar (~revattar@50-0-117-34.dedicated.static.sonic.net)
17:09.20LzrdKingi think its looking good now...
17:18.11LzrdKinghah! 01 just GCed, but 02 hasn;t, though its ABOUT to
17:20.04LzrdKingor... not
17:24.05LzrdKingok, it went
17:25.07xortimwhy are you so concerned about the appserver GCing? that's usual behavior for the JVM
17:25.38xortimif you're not geting out of memory exceptions and/or cpu starvation, it's suppose to GC
17:33.41LzrdKingi know, but why is one GCing more frequently than another identical one?
17:34.18LzrdKing01 just GCed again, its got 5 to 02's 3
17:34.22xortimone is busier than the other ;o)
17:34.36LzrdKingno, they are both equally idle
17:34.59xortimdiff the server.xml
17:35.06xortimand diff the configuration files
17:35.13LzrdKingthey are identival not; one had more threads before but i changed that
17:35.17LzrdKingidentical now*
17:36.08xortimwhat OS?
17:36.31LzrdKingrhel 5.5
17:37.33xortimusing the tomcat package from RHN or from the apache foundation? and also which Java versions are you using?
17:37.40xortimare the ulimits the same for each user?
17:37.52LzrdKingusing a tarball from apache
17:39.11LzrdKingsame jvm, same ulimits
17:39.25LzrdKingjava version "1.6.0_22"
17:39.26LzrdKingJava(TM) SE Runtime Environment (build 1.6.0_22-b04)
17:39.26LzrdKingJava HotSpot(TM) 64-Bit Server VM (build 17.1-b03, mixed mode)
17:40.57xortimsame kernel versions?
17:41.15LzrdKingindeed
17:44.16xortimwell. I don't think it's something to lose any sleep over. It sounds like you've done some settings changes between the two and are wondering why one is different than the other.
17:44.24xortimtry taking coredumps (kill -3)
17:44.33xortimand see what's running at the time of the GC
17:45.46LzrdKing01 has a few more threads
17:47.31xortimwhich threads?
17:47.55LzrdKinglive and daemon
17:49.38xortimwhat's the command being executed for each?
17:49.40xortimps -ef | grep tomcat
17:51.59LzrdKingsvcgeo   19960 19959  0 12:55 ?        00:00:06 jsvc.exec -user svcgeo -home /usr/java/default -Dcatalina.home=/opt/geocoding/tomcat-geocode -Dcatalina.base=/opt/geocoding/tomcat-geocode/ -Djava.io.tmpdir=/var/tmp -wait 10 -pidfile /var/run/tomcat.geocode.pid -outfile /opt/geocoding/tomcat-geocode/logs/catalina.out -errfile &1 -Xmx4096m -XX:MaxPermSize=512m -Dcom.sun.management.jmxremote -Djava.rmi.server.hostname=serverFQDN -Dcom.sun.management.
17:52.08LzrdKingdamn, i'm sure that got cut off
17:52.15LzrdKingbut in any case, they are identical
17:54.11LzrdKingtomcat-users.xml has a user on one, but no manager application for it to matter
17:55.19LzrdKingthe 02 server hasn't GCed since 1:15, but the 02 has GCed 2 more times since then
17:57.08LzrdKingoooh they both just full-GCed, maybe that will help the one
17:57.23LzrdKingdoesn;t look it
17:58.31LzrdKingwell, i will continue to monitor but i won;t "lose any sleep over" it
18:31.47LzrdKingwhere do i get jsvc from?
18:32.33xortimhttp://commons.apache.org/proper/commons-daemon/jsvc.html  ?
18:32.44whartungit's a source tar ball in the bin directory of the binary distribution
18:32.55LzrdKingyeah, i'm not seeing it
18:33.16LzrdKingcommons-daemon-native.tar.gz?
18:33.22whartungyes
18:33.31LzrdKingi'd prefer to not compile it
18:33.44whartungguess you're out of luck then
18:34.05whartunghave you tried it?
18:34.06LzrdKingis tomcat 7's very different than tomcat 6's? because i could just copy over the tomcat 6 version
18:34.13whartungI imagine they're identical
18:35.24LzrdKingalso, http://www.apache.org/dist/commons/daemon/binaries/
18:35.59whartungbinaries for what platforms?
18:36.32LzrdKinglinux
18:36.40LzrdKingbut yeah, thats not the right thing
18:46.23xortimdefine "thing"
18:47.07LzrdKingexecutable
18:48.11xortimhow so?
18:48.16xortimthey're executable jar files
18:48.57LzrdKingwhat i copied from a tomcat 6 box (jsvc and commons-daemon.jar) isn;t working
18:49.09LzrdKingjsvc.exec error: Service exit with a return value of 3
18:49.13xortimor it's a compiled library for you to use in your application
18:49.48xortimcheck the javadoc/usage for this particular project
18:50.08whartungjsvc is not java xortim
18:50.14LzrdKingi have a script that calls it
18:50.17whartungit's a c program to run java as a service
18:50.32xortimlooks like a bunch of jar files to me ;o)
18:50.33whartungI've never used it
18:50.48xortimwhy not use tanuki java service wrapper?
18:50.59*** join/#tomcat jahor (~jahor@89.185.248.105)
18:51.42LzrdKingwhy not do a lot of things?
18:54.00xortimyeah, it looks that commons-daemon that you linked to isn't jsvc
18:54.55xortimDaemon is made of 2 parts. One written in C that makes the interface to the operating system and the other in Java that provides the Daemon API.
18:55.13xortimso like i said. that commons-daemon is a series of jar files for you to use in your application
18:55.30xortimand the other part is procrun (win) or jsvc (*nix)
18:56.51LzrdKingso where do i get the jsvc part from?
18:57.23whartungwhy don't you just build it from the tar.gz file?
18:57.42xortimyou build it
18:58.26LzrdKingi don;t have a build env
18:58.44xortimdo you have a linux machine?
18:59.26LzrdKingyes, i suppose i could do it there
19:05.37LzrdKingfgah, have to install gcc and friends
19:06.59LzrdKingwhy don't jsvc binaries exist?
19:08.46xortimthey do. once they're built ;o)
19:09.22xortimmy speculation is that it wasn't designed to exist as a standalone product
19:09.22LzrdKing1) thats not what i mean, 2) you know what i mean, and 3) i can't build jsvc because the java path is different on the servers than on my workstation
19:09.43xortimrather as something that can be included with a particular product
19:10.14LzrdKingits pissing me off
19:10.28xortimdo you not have a valid RHN subscription?
19:12.16LzrdKingwho said?
19:13.14xortimwhy not install gcc?
19:13.29*** join/#tomcat yassine (~yassine@unaffiliated/yassine)
19:13.31LzrdKingon a server?
19:13.40LzrdKingnow you're just being silly
19:13.41xortimremove it when you're done
19:13.57*** join/#tomcat franck (~franck@tibco-5.tibco.com)
19:14.16LzrdKingi just made a link so i could specify the same location for java
19:16.10LzrdKing/usr/local/prod/server/tomcat-proxy/bin/jsvc: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by /usr/local/prod/server/tomcat-proxy/bin/jsvc)
19:16.18LzrdKingbugger
19:16.31xortimneed the dev libraries
19:17.10LzrdKingits already compiled
19:18.52LzrdKingPackage glibc-devel-2.5-81.el5_8.1.x86_64 already installed and latest version
19:19.18LzrdKingi guess i'll have to put gcc on the server
19:25.46LzrdKingwtf! the point of a binary distro was to avoid crap like "failed loading capabilities library -- /usr/lib/libcap.so: cannot open shared object file: No such file or directory."
19:26.20xortimnot really
19:26.33xortimyou just don't have all the libraries installed to compile it
19:26.37xortimsame as everything else
19:26.40LzrdKingno, it compiled fine
19:26.53LzrdKingtwice now, once on my local machine and once on the server; problems both times
19:26.58LzrdKingruntime problems, that is
19:27.03xortimldd /path/to/jsvc
19:27.12LzrdKingyum install jsvc
19:27.22xortimso there was a package for it all along?
19:27.38LzrdKingno, i wish
19:27.42LzrdKingwhy isn;t there?
19:28.00xortimbecause of situations like you're having
19:28.20xortimif it requires the path to java installed, which can vary between applications and distribution methods. you run into issues with it
19:28.29LzrdKingoh jeez
19:28.31xortimthat said. try tanuki java service wrapper
19:28.32LzrdKingare you kidding me?
19:28.39LzrdKingredhat has a standar java location
19:28.53xortimyet yours are different between the two servers?
19:29.01LzrdKingno
19:29.09LzrdKingfedora desktop, rhel server
19:29.31LzrdKingso ldd gave me 5 libraries each with an address
19:29.51xortimoh, i misunderstood your previous comment about java locations then
19:30.02xortimldconfig run recently?
19:30.10LzrdKingdunno
19:30.16xortimpermission to that library?
19:30.19LzrdKingits not even looking for libcap.so
19:30.26xortimhit the low hanging fruit first
19:30.32xortimso that's why it compiled properly
19:30.41xortimit wasn't required to building, but required during runtime
19:30.48LzrdKingno, i mean ldd isn;t looking for it
19:31.23xortimyeah, means that jsvc wasn't compiled with a link to that library, but it's trying to use it anyway
19:31.26LzrdKinggoddamn! its at /lib/libcap.so
19:31.35xortimbooom
19:31.49xortimthe ./configure should have options for specifying the library directory
19:32.14LzrdKingnice to see compiling code has gotten HARDER since 1996
19:32.23xortimit really hasn't ;o)
19:32.37LzrdKingi never had this much trouble back then
19:32.53xortimit's always a pain because the author probably don't know about or give a crap about where the various spots are, defaults will be for his/her system
19:33.13xortimhence the ./configure options
19:33.58LzrdKingwhy am i bothering? failed loading capabilities library -- /usr/lib/libcap.so: wrong ELF class: ELFCLASS32.
19:34.48xortim./configure --help
19:35.14xortimand see if it's even 64bit compatible, otherwise you'll need to install the ia32-libs and point it there
19:35.44xortimor vis-versa, not sure how your system is set up
19:36.56LzrdKingdoesn;t say
19:38.03xortimuname -p
19:38.46LzrdKingwell, we KNOW the sierver is 64 bit
19:38.48LzrdKingserver
19:39.10xortimso see if there's a ./configure option to compile it for 64bit
19:43.46LzrdKingthere is not
19:43.51LzrdKingthis is total BS
19:44.16LzrdKingtomcat will run on 8080, i don;t even need this
19:45.36xortimi believe they still recommend running apache infront of tomcat
19:46.55LzrdKingok, still no need to start as rioot and drop privs, like jsvc lets you; i can just start tomcat
20:07.58LzrdKingthere, now tomcat starts up; damn that jsvc crap
20:34.40LzrdKingis the only way to monitor two different JVMs on on server to use two JMX ports?
20:35.15LzrdKingon one server
20:36.26whartungI would imagine LzrdKing how else could it be done?
20:37.09LzrdKingsomething other than the port
20:37.20xortimO_o
20:37.46*** join/#tomcat namredips (~namredips@76-224-212-245.lightspeed.iplsin.sbcglobal.net)
20:37.54LzrdKingsince i had to submit a firewall request to have the port opened... i would like to be able to monitor an arbitrary number of jvms through the one port
20:38.37*** join/#tomcat revattar (~revattar@50-0-117-34.dedicated.static.sonic.net)
20:39.01xortimyou always use ssh tunnels
20:39.12xortim*could always
20:39.30*** join/#tomcat bkfitz (~bkfitz@218.sub-70-197-162.myvzw.com)
20:40.44LzrdKingwhat set up is better, to have multiple tomcats/JVMs eachrunning one or two apps, or to throw all the apps into one tomcat/JVM?
20:41.05LzrdKingwell, maybe more than 2 apps
20:41.06whartungdepends
20:41.15LzrdKingin what?
20:42.11xortimvertical scaling isn't really necessary with 64bit
20:42.13whartunghow often they change, what impact the necessity to reboot the server because of some change in one app, even if the others have not. Need to balance CPU loads between apps
20:42.41whartungbenefits are lower memory footprint.
20:43.07LzrdKingso basically more tomcats means greater flexibility, but more memory is needed?
20:43.58xortimthat a bit of an over simplification. it depends on your needs
20:44.33whartungwell you also have to deal with port conflicts and such, but you could mitigate that with extra IPs and interfaces if that was an issue.
20:44.35xortimif you have a front end app that gets a code deploy very frequently and a backend app that cannot go down outside of maintenance windows, then you'll want two tomcats
21:01.44LzrdKinggot it
21:03.03LzrdKingso here's whats going on with these two "identical" tomcats: one has done a full GC once per hour for the past 3 hours and thats it; the other has done a full gc at the same times, but also a scavenge 3-4 times in the hour BEFORE the full GC
21:03.22LzrdKingby full GC, i mean a Scavenge AND a MarkSweep
21:25.39*** join/#tomcat clajo04 (~clajo04@pool-96-224-200-185.nycmny.fios.verizon.net)
21:56.39*** join/#tomcat AJ_Z0 (~AJ_Z0@pdpc/supporter/professional/ajz0)
22:42.51*** join/#tomcat nevada-smith (~nevada-sm@S0106c8fb2641d848.vs.shawcable.net)
22:54.58*** join/#tomcat bzb (~bzb@108.175.226.248)

Generated by irclog2html.pl Modified by Tim Riker to work with infobot.