[Anthill-pro] What CPU usage is expected by Anthill server?
Ryan Smith
rws at urbancode.com
Thu May 22 12:50:26 CDT 2008
Peter,
What version of Java are you using? We can probably create a jsp snippet that will tell us which threads are using up the CPU and deduce it from that if you are running Java 5 or later.
Ryan
Peter Steele wrote:
> We are using the default Derby database. No other processes on the
> system have high CPU usage. As for disk I/O activity, I'm not sure how
> to check for this under Linux (I'm more of Windows guy). The server box
> is running CentOS 4.5 I believe. This server also runs our Subversion
> and JIRA services, but I don't think the box is anywhere near being
> overloaded. It's a quad CPU system.
>
> -----Original Message-----
> From: Ryan Smith [mailto:rws at urbancode.com]
> Sent: Thursday, May 22, 2008 8:11 AM
> To: Peter Steele
> Cc: AnthillPro user and support list.; Yanko, Curtis
> Subject: Re: [Anthill-pro] What CPU usage is expected by Anthill server?
>
> I agree, it is unusual. We are testing the idle CPU usage with a large
> number of agents today. Is the process doing a lot of disk IO? What
> database are you using, how is its CPU usage?
>
> Ryan
>
> Peter Steele wrote:
>> I just applied this change to my 3.5.0 install and restarted the
> server.
>> It's showing under 50MB memory usage right now, which is not
> surprising
>> since I just rebooted the server. It also shows that there are zero
>> checked out threads. However, the Anthill Java process still shows up
>
>> as the number 1 CPU consumer, even though no builds nor anything else
>> are going on right now. The process is essentially never truly idle;
>> it's always churning away doing something. This just doesn't seem
> right.
>> Unfortunately I had not noted what the CPU usage was prior to
> upgrading
>> to 3.5.0 so I don't know if this is something that was introduced by
> the
>> upgrade. Is high CPU usage normal, even when no real activity is going
> on?
>>
>>
>> -----Original Message-----
>> From: anthill-pro-bounces at lists.urbancode.com
>> [mailto:anthill-pro-bounces at lists.urbancode.com] On Behalf Of Ryan
> Smith
>> Sent: Thursday, May 22, 2008 6:41 AM
>> To: Yanko, Curtis
>> Cc: AnthillPro user and support list.
>> Subject: Re: [Anthill-pro] What CPU usage is expected by Anthill
> server?
>>
>>
>> Everyone should be aware that the memory display was fixed in 3.5.1.
>> Prior versions did not correctly display the memory.
>>
>>
>>
>> The bottom of
>> opt/tomcat/webapps/ROOT/WEB-INF/jsps/currentActivity/memory.jsp should
> be:
>>
>>
>>
>>
>>
>>
>> <br>
>>
>> <div class="note">
>>
>> <b>Memory:</b><br>
>>
>> <%
>>
>> Runtime runtime = Runtime.getRuntime();
>>
>> long tm = runtime.totalMemory();
>>
>> long fm = runtime.freeMemory();
>>
>> long mm = runtime.maxMemory();
>>
>> long um = tm - fm;
>>
>> %>
>>
>> Using <%= getNearestBytes((double) um) %> / <%=
>> getNearestBytes((double) tm) %> bytes<br>
>>
>> <%= getNearestBytes((double) fm) %> bytes free<br>
>>
>> <%= getNearestBytes((double) mm) %> max memory<br>
>>
>> </div>
>>
>>
>>
>>
>>
>> Ryan
>>
>>
>>
>>
>>
>> Yanko, Curtis wrote:
>>
>> > Take a look at the resources tab and see how you are doing on
> memory.
>> > When it gets fully consume AHP BMS does some odd things. We have
> taken
>> > to upping the JVM memory size. Looking at my own BMS system on
> Windows
>> > it is a good, idle app waiting for work. (3.4.11)
>>
>> >
>>
>> >
>>
>> > ===
>>
>> > -Curt
>>
>> > W: 860.702.9059
>>
>> > M: 860.881.2050
>>
>> >
>>
>> > -----Original Message-----
>>
>> > From: anthill-pro-bounces at lists.urbancode.com
>>
>> > [mailto:anthill-pro-bounces at lists.urbancode.com] On Behalf Of Peter
>>
>> > Steele
>>
>> > Sent: Wednesday, May 21, 2008 9:10 PM
>>
>> > To: rws at urbancode.com; AnthillPro user and support list.
>>
>> > Subject: RE: [Anthill-pro] What CPU usage is expected by Anthill
> server?
>> >
>>
>> > I agree of course that CPU usage is proportional to activity, but
> why
>> > would the Anthill server be consistently the highest CPU consumer,
> even
>> > at times when the system is essentially idle--no builds running, no
>>
>> > cleanup processes running, no codestation resolves (we don't use
> this
>> > feature) and no UI activity (beyond anyone with windows open to the
>>
>> > dashboard). I assume the server is constantly monitoring the
> agents,
>> > even if the agents are idle, but that should be the bulk of the
>>
>> > activity. Are there figures on how well the Anthill server scales?
> We
>> > have currently have 50 agents configured, and this number will grow
> to
>> > 200+ over the next few months. Can the Anthill server handle this
> many
>> > agents?
>>
>> >
>>
>> > -----Original Message-----
>>
>> > From: anthill-pro-bounces at lists.urbancode.com
>>
>> > [mailto:anthill-pro-bounces at lists.urbancode.com] On Behalf Of Ryan
> Smith
>> > Sent: Wednesday, May 21, 2008 5:56 PM
>>
>> > To: AnthillPro user and support list.
>>
>> > Subject: Re: [Anthill-pro] What CPU usage is expected by Anthill
> server?
>> >
>>
>> > Peter,
>>
>> >
>>
>> > The CPU usage is going to be proportional to the activity on the
> server.
>> > Activity includes running workflows, miscellaneous tasks like
> cleanup,
>> > UI activity and any remote Codestation resolves. There are other
> things
>> > that could cause high CPU usage such as memory thrashing. We are
>>
>> > investigating potential memory problems with 3.5. If we find one,
> that
>> > could be the cause.
>>
>> >
>>
>> >
>>
>> > Ryan Smith
>>
>> >
>>
>> > Peter Steele wrote:
>>
>> >> Here's an interesting snapshot: This is the first few lines from
> the
>> >> "top" command:
>>
>> >>
>>
>> >>
>>
>> >>
>>
>> >> top - 15:26:49 up 64 days, 2:14, 3 users, load average: 0.10,
> 0.20,
>> > 0.25
>>
>> >> Tasks: 143 total, 1 running, 142 sleeping, 0 stopped, 0
> zombie
>> >>
>>
>> >> Cpu(s): 19.8% us, 0.2% sy, 0.0% ni, 79.9% id, 0.0% wa, 0.0%
> hi,
>> > 0.2% si
>>
>> >> Mem: 4109408k total, 3990708k used, 118700k free, 89004k
>>
>> > buffers
>>
>> >> Swap: 8385920k total, 144k used, 8385776k free, 3012000k
>>
>> > cached
>>
>> >>
>>
>> >>
>>
>> >> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+
>>
>> >> COMMAND
>
>> >>
>>
>> >> * 7778 root 18 0 530m 300m 8972 S 40 7.5 238:02.05
>>
>> >> java
>>
>> > *
>>
>> >> 30397 root 16 0 7348 1068 772 R 0 0.0 0:29.29
>>
>> >> top
>
>> >>
>>
>> >> 1 root 16 0 4756 556 460 S 0 0.0 0:01.23 init
>
>> >>
>>
>> >>
>>
>> >> 2 root RT 0 0 0 0 S 0 0.0 0:01.91
>>
>> >> migration/0
>
>> >>
>>
>> >>
>>
>> >>
>>
>> >> As you can see, process 7778 is current the top CPU user. This is
> the
>> >> Anthill server. I rebooted the server just two days ago and it as
>>
>> >> already consumed over 238 minutes of CPU time. Nothing else comes
>>
>> > close
>>
>> >> to this, and the system has been up for 64 days. This seems to be
>>
>> >> pointing to some kind of problem. What should we expect as far as
> CPU
>> >> usage is concerned for the Anthill server process?
>>
>> >>
>>
>> >>
>>
>> >>
>>
>> >> *From:* anthill-pro-bounces at lists.urbancode.com
>>
>> >> [mailto:anthill-pro-bounces at lists.urbancode.com] *On Behalf Of
> *Peter
>> > Steele
>>
>> >> *Sent:* Wednesday, May 21, 2008 1:21 PM
>>
>> >> *To:* AnthillPro user and support list.
>>
>> >> *Subject:* RE: [Anthill-pro] What CPU usage is expected by Anthill
>>
>> > server?
>>
>> >>
>>
>> >>
>>
>> >> The server still seems to be a CPU hog, with it almost always
> being
>> > the
>>
>> >> number one process on the "top" list, even when the system is
> largely
>> >> idol. We've added 40+ agents recently, but I did not check the
>>
>> > server's
>>
>> >> CPU usage before adding all the new agents, and I also did not
> what
>> > the
>>
>> >> CPU usage was before upgrading to 3.5. Fortunately the command
>>
>> > failures
>>
>> >> we were seeing the other day appear to have stopped after the
> reboot,
>> >> although the GUI is still very slow to respond to many typical
>>
>> > commands.
>>
>> >> For example, it just took 30 seconds for Anthill to respond to a
> click
>> >
>>
>> >> on a project displayed in the "Your Active Projects" column of the
>>
>> >> Dashboard. This seems to be slower than what it was in 3.4.12, but
> as
>> > I
>>
>> >> indicated we have added a lot of agents lately. We'll be adding
> more
>> > as
>>
>> >> well, exceeding 100 agents in the next while. Is there anything we
>>
>> >> should do to improve performance?
>>
>> >>
>>
>> >>
>>
>> >>
>>
>> >> *From:* anthill-pro-bounces at lists.urbancode.com
>>
>> >> [mailto:anthill-pro-bounces at lists.urbancode.com] *On Behalf Of
> *Steve
>> > Boone
>>
>> >> *Sent:* Wednesday, May 21, 2008 9:39 AM
>>
>> >> *To:* AnthillPro user and support list.
>>
>> >> *Subject:* Re: [Anthill-pro] What CPU usage is expected by Anthill
>>
>> > server?
>>
>> >>
>>
>> >>
>>
>> >> Peter,
>>
>> >>
>>
>> >> Has this been happening in a regular basis?
>>
>> >>
>>
>> >> What happened after the restart?
>>
>> >>
>>
>> >> On Wed, May 14, 2008 at 6:44 PM, Peter Steele
> <psteele at maxiscale.com
>> >> <mailto:psteele at maxiscale.com>> wrote:
>>
>> >>
>>
>> >> We've noticed some very slow feedback from the Anthill server
> today,
>> >> even coming back with failures for some operations. The Unix top
>>
>> > command
>>
>> >> is showing the ah3server process as taking as much as 100% CPU at
>>
>> > times,
>>
>> >> and is consistently the top 1 or 2 processes on the system. There
> are
>> >> just two builds going on, and both of course are running on agent
>>
>> >> systems, not the server. I don't recall seeing this before. We
>>
>> > upgraded
>>
>> >> to 3.5 yesterday and I am wondering if that might be to blame.
> It's
>> >> certainly very slow right now; I'll restart the server as soon as
> the
>> >> builds are done...
>>
>> >>
>>
>> >>
>>
>> >>
>>
>> >>
>>
>> >> _______________________________________________
>>
>> >> Anthill-pro mailing list
>>
>> >> Anthill-pro at lists.urbancode.com
>>
>> > <mailto:Anthill-pro at lists.urbancode.com>
>>
>> >> http://lists.urbancode.com/mailman/listinfo/anthill-pro
>>
>> >>
>>
>> >>
>>
>> >>
>>
>> >>
>>
>> >>
>>
>> >
> ------------------------------------------------------------------------
>> >> _______________________________________________
>>
>> >> Anthill-pro mailing list
>>
>> >> Anthill-pro at lists.urbancode.com
>>
>> >> http://lists.urbancode.com/mailman/listinfo/anthill-pro
>>
>> >
>>
>> > --
>>
>> > ===========================================================
>>
>> > Ryan Smith. 2044 Euclid Ave., Suite 600
>>
>> > Developer Cleveland, Ohio 44115
>>
>> > Urbancode, Inc.
>>
>> > email: rws at urbancode.com
>>
>> > web: www.urbancode.com phone: 216-858-9000
>>
>> > web: www.anthillpro.com fax: 216-858-9602
>>
>> > ===========================================================
>>
>> > _______________________________________________
>>
>> > Anthill-pro mailing list
>>
>> > Anthill-pro at lists.urbancode.com
>>
>> > http://lists.urbancode.com/mailman/listinfo/anthill-pro
>>
>> > _______________________________________________
>>
>> > Anthill-pro mailing list
>>
>> > Anthill-pro at lists.urbancode.com
>>
>> > http://lists.urbancode.com/mailman/listinfo/anthill-pro
>>
>> >
>>
>> >
>>
>> > This e-mail, including attachments, may include confidential and/or
>>
>> > proprietary information, and may be used only by the person or
> entity to
>> > which it is addressed. If the reader of this e-mail is not the
> intended
>> > recipient or his or her authorized agent, the reader is hereby
> notified
>> > that any dissemination, distribution or copying of this e-mail is
>>
>> > prohibited. If you have received this e-mail in error, please
> notify the
>> > sender by replying to this message and delete this e-mail
> immediately.
>> >
>>
>> >
>>
>>
>>
>> --
>>
>> ===========================================================
>>
>> Ryan Smith. 2044 Euclid Ave., Suite 600
>>
>> Developer Cleveland, Ohio 44115
>>
>> Urbancode, Inc.
>>
>> email: rws at urbancode.com
>>
>> web: www.urbancode.com phone: 216-858-9000
>>
>> web: www.anthillpro.com fax: 216-858-9602
>>
>> ===========================================================
>>
>> _______________________________________________
>>
>> Anthill-pro mailing list
>>
>> Anthill-pro at lists.urbancode.com
>>
>> http://lists.urbancode.com/mailman/listinfo/anthill-pro
>>
>
--
===========================================================
Ryan Smith. 2044 Euclid Ave., Suite 600
Developer Cleveland, Ohio 44115
Urbancode, Inc.
email: rws at urbancode.com
web: www.urbancode.com phone: 216-858-9000
web: www.anthillpro.com fax: 216-858-9602
===========================================================
More information about the Anthill-pro
mailing list