[Anthill-pro] What CPU usage is expected by Anthill server?
Ryan Smith
rws at urbancode.com
Thu May 22 15:23:51 CDT 2008
Peter,
Here is a JSP snippet that you can put at the bottom of opt/tomcat/webapps/ROOT/WEB-INF/jsps/currentActivity/memory.jsp. You will see a break down of each thread in Anthill and how much CPU it has used on the Current Activity - Resources screen. Please send me its contents, thanks.
<%@page import="java.lang.management.*" %>
<%@page import="com.urbancode.commons.util.Duration" %>
<%@page import="java.util.*" %>
<div class="note">
<table><tr><td>
<%
ThreadMXBean threadMBean = ManagementFactory.getThreadMXBean();
if (threadMBean.isThreadCpuTimeEnabled()) {
long[] threadIds = threadMBean.getAllThreadIds();
ThreadInfo[] threadInfos = threadMBean.getThreadInfo(threadIds);
Arrays.sort(threadInfos, new Comparator() {
public int compare(Object obj1, Object obj2) {
return ((ThreadInfo) obj1).getThreadName().compareToIgnoreCase(((ThreadInfo) obj2).getThreadName());
}
});
for (int t=0; t<threadInfos.length; t++) {
pageContext.setAttribute("threadInfo", threadInfos[t]);
%>
<div style="float: right;">
<%= new Duration(threadMBean.getThreadCpuTime(threadInfos[t].getThreadId()) / 1000000).getLeastUnit(true, true) %>
</div>
<%= threadInfos[t].getThreadName() %> --
<br/>
<%
}
%>
<%
} else {
%>
Thread CPU monitoring not enabled in this JVM
<%
}
%>
</td></tr></table>
</div>
Peter Steele wrote:
> We're using 1.6. Here's the complete list of installed properties:
>
> #Tue May 13 14:38:19 PDT 2008
> install.db.schema=null
> install.db.url=jdbc\:derby\://localhost\:11366/data
> install.server.web.always.secure=N
> install.db.password=pbe{hXXSbndq6l4Qdv1RJ1AhSKIWmwG3Kk72}
> install.derby.port=11366
> install.db.driver=org.apache.derby.jdbc.ClientDriver
> install.db.user=anthill3
> install.server.web.port=8080
> install.server.port=4567
> install.server.web.ip=0.0.0.0
> install.java.home=/usr/java/jdk1.6.0_01
> install.db.type=derby
>
> Peter
>
> -----Original Message-----
> From: Ryan Smith [mailto:rws at urbancode.com]
> Sent: Thursday, May 22, 2008 10:50 AM
> To: Peter Steele
> Cc: AnthillPro user and support list.
> Subject: Re: [Anthill-pro] What CPU usage is expected by Anthill server?
>
> Peter,
>
> What version of Java are you using? We can probably create a jsp snippet
> that will tell us which threads are using up the CPU and deduce it from
> that if you are running Java 5 or later.
>
>
> Ryan
>
> Peter Steele wrote:
>> We are using the default Derby database. No other processes on the
>> system have high CPU usage. As for disk I/O activity, I'm not sure how
>> to check for this under Linux (I'm more of Windows guy). The server
> box
>> is running CentOS 4.5 I believe. This server also runs our Subversion
>> and JIRA services, but I don't think the box is anywhere near being
>> overloaded. It's a quad CPU system.
>>
>> -----Original Message-----
>> From: Ryan Smith [mailto:rws at urbancode.com]
>> Sent: Thursday, May 22, 2008 8:11 AM
>> To: Peter Steele
>> Cc: AnthillPro user and support list.; Yanko, Curtis
>> Subject: Re: [Anthill-pro] What CPU usage is expected by Anthill
> server?
>> I agree, it is unusual. We are testing the idle CPU usage with a large
>> number of agents today. Is the process doing a lot of disk IO? What
>> database are you using, how is its CPU usage?
>>
>> Ryan
>>
>> Peter Steele wrote:
>>> I just applied this change to my 3.5.0 install and restarted the
>> server.
>>> It's showing under 50MB memory usage right now, which is not
>> surprising
>>> since I just rebooted the server. It also shows that there are zero
>>> checked out threads. However, the Anthill Java process still shows
> up
>>> as the number 1 CPU consumer, even though no builds nor anything else
>
>>> are going on right now. The process is essentially never truly idle;
>>> it's always churning away doing something. This just doesn't seem
>> right.
>>> Unfortunately I had not noted what the CPU usage was prior to
>> upgrading
>>> to 3.5.0 so I don't know if this is something that was introduced by
>> the
>>> upgrade. Is high CPU usage normal, even when no real activity is
> going
>> on?
>>>
>>>
>>> -----Original Message-----
>>> From: anthill-pro-bounces at lists.urbancode.com
>>> [mailto:anthill-pro-bounces at lists.urbancode.com] On Behalf Of Ryan
>> Smith
>>> Sent: Thursday, May 22, 2008 6:41 AM
>>> To: Yanko, Curtis
>>> Cc: AnthillPro user and support list.
>>> Subject: Re: [Anthill-pro] What CPU usage is expected by Anthill
>> server?
>>>
>>>
>>> Everyone should be aware that the memory display was fixed in 3.5.1.
>>> Prior versions did not correctly display the memory.
>>>
>>>
>>>
>>> The bottom of
>>> opt/tomcat/webapps/ROOT/WEB-INF/jsps/currentActivity/memory.jsp
> should
>> be:
>>>
>>>
>>>
>>>
>>>
>>>
>>> <br>
>>>
>>> <div class="note">
>>>
>>> <b>Memory:</b><br>
>>>
>>> <%
>>>
>>> Runtime runtime = Runtime.getRuntime();
>>>
>>> long tm = runtime.totalMemory();
>>>
>>> long fm = runtime.freeMemory();
>>>
>>> long mm = runtime.maxMemory();
>>>
>>> long um = tm - fm;
>>>
>>> %>
>>>
>>> Using <%= getNearestBytes((double) um) %> / <%=
>>> getNearestBytes((double) tm) %> bytes<br>
>>>
>>> <%= getNearestBytes((double) fm) %> bytes free<br>
>>>
>>> <%= getNearestBytes((double) mm) %> max memory<br>
>>>
>>> </div>
>>>
>>>
>>>
>>>
>>>
>>> Ryan
>>>
>>>
>>>
>>>
>>>
>>> Yanko, Curtis wrote:
>>>
>>> > Take a look at the resources tab and see how you are doing on
>> memory.
>>> > When it gets fully consume AHP BMS does some odd things. We have
>> taken
>>> > to upping the JVM memory size. Looking at my own BMS system on
>> Windows
>>> > it is a good, idle app waiting for work. (3.4.11)
>>>
>>> >
>>>
>>> >
>>>
>>> > ===
>>>
>>> > -Curt
>>>
>>> > W: 860.702.9059
>>>
>>> > M: 860.881.2050
>>>
>>> >
>>>
>>> > -----Original Message-----
>>>
>>> > From: anthill-pro-bounces at lists.urbancode.com
>>>
>>> > [mailto:anthill-pro-bounces at lists.urbancode.com] On Behalf Of
> Peter
>>> > Steele
>>>
>>> > Sent: Wednesday, May 21, 2008 9:10 PM
>>>
>>> > To: rws at urbancode.com; AnthillPro user and support list.
>>>
>>> > Subject: RE: [Anthill-pro] What CPU usage is expected by Anthill
>> server?
>>> >
>>>
>>> > I agree of course that CPU usage is proportional to activity, but
>> why
>>> > would the Anthill server be consistently the highest CPU consumer,
>> even
>>> > at times when the system is essentially idle--no builds running,
> no
>>> > cleanup processes running, no codestation resolves (we don't use
>> this
>>> > feature) and no UI activity (beyond anyone with windows open to
> the
>>> > dashboard). I assume the server is constantly monitoring the
>> agents,
>>> > even if the agents are idle, but that should be the bulk of the
>>>
>>> > activity. Are there figures on how well the Anthill server scales?
>> We
>>> > have currently have 50 agents configured, and this number will
> grow
>> to
>>> > 200+ over the next few months. Can the Anthill server handle this
>> many
>>> > agents?
>>>
>>> >
>>>
>>> > -----Original Message-----
>>>
>>> > From: anthill-pro-bounces at lists.urbancode.com
>>>
>>> > [mailto:anthill-pro-bounces at lists.urbancode.com] On Behalf Of Ryan
>> Smith
>>> > Sent: Wednesday, May 21, 2008 5:56 PM
>>>
>>> > To: AnthillPro user and support list.
>>>
>>> > Subject: Re: [Anthill-pro] What CPU usage is expected by Anthill
>> server?
>>> >
>>>
>>> > Peter,
>>>
>>> >
>>>
>>> > The CPU usage is going to be proportional to the activity on the
>> server.
>>> > Activity includes running workflows, miscellaneous tasks like
>> cleanup,
>>> > UI activity and any remote Codestation resolves. There are other
>> things
>>> > that could cause high CPU usage such as memory thrashing. We are
>>>
>>> > investigating potential memory problems with 3.5. If we find one,
>> that
>>> > could be the cause.
>>>
>>> >
>>>
>>> >
>>>
>>> > Ryan Smith
>>>
>>> >
>>>
>>> > Peter Steele wrote:
>>>
>>> >> Here's an interesting snapshot: This is the first few lines from
>> the
>>> >> "top" command:
>>>
>>> >>
>>>
>>> >>
>>>
>>> >>
>>>
>>> >> top - 15:26:49 up 64 days, 2:14, 3 users, load average: 0.10,
>> 0.20,
>>> > 0.25
>>>
>>> >> Tasks: 143 total, 1 running, 142 sleeping, 0 stopped, 0
>> zombie
>>> >>
>>>
>>> >> Cpu(s): 19.8% us, 0.2% sy, 0.0% ni, 79.9% id, 0.0% wa, 0.0%
>> hi,
>>> > 0.2% si
>>>
>>> >> Mem: 4109408k total, 3990708k used, 118700k free, 89004k
>>>
>>> > buffers
>>>
>>> >> Swap: 8385920k total, 144k used, 8385776k free, 3012000k
>>>
>>> > cached
>>>
>>> >>
>>>
>>> >>
>>>
>>> >> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+
>>>
>>> >> COMMAND
>>> >>
>>>
>>> >> * 7778 root 18 0 530m 300m 8972 S 40 7.5 238:02.05
>>>
>>> >> java
>>>
>>> > *
>>>
>>> >> 30397 root 16 0 7348 1068 772 R 0 0.0 0:29.29
>>>
>>> >> top
>>> >>
>>>
>>> >> 1 root 16 0 4756 556 460 S 0 0.0 0:01.23 init
>>> >>
>>>
>>> >>
>>>
>>> >> 2 root RT 0 0 0 0 S 0 0.0 0:01.91
>>>
>>> >> migration/0
>>> >>
>>>
>>> >>
>>>
>>> >>
>>>
>>> >> As you can see, process 7778 is current the top CPU user. This is
>> the
>>> >> Anthill server. I rebooted the server just two days ago and it as
>>>
>>> >> already consumed over 238 minutes of CPU time. Nothing else comes
>>>
>>> > close
>>>
>>> >> to this, and the system has been up for 64 days. This seems to be
>>>
>>> >> pointing to some kind of problem. What should we expect as far as
>> CPU
>>> >> usage is concerned for the Anthill server process?
>>>
>>> >>
>>>
>>> >>
>>>
>>> >>
>>>
>>> >> *From:* anthill-pro-bounces at lists.urbancode.com
>>>
>>> >> [mailto:anthill-pro-bounces at lists.urbancode.com] *On Behalf Of
>> *Peter
>>> > Steele
>>>
>>> >> *Sent:* Wednesday, May 21, 2008 1:21 PM
>>>
>>> >> *To:* AnthillPro user and support list.
>>>
>>> >> *Subject:* RE: [Anthill-pro] What CPU usage is expected by
> Anthill
>>> > server?
>>>
>>> >>
>>>
>>> >>
>>>
>>> >> The server still seems to be a CPU hog, with it almost always
>> being
>>> > the
>>>
>>> >> number one process on the "top" list, even when the system is
>> largely
>>> >> idol. We've added 40+ agents recently, but I did not check the
>>>
>>> > server's
>>>
>>> >> CPU usage before adding all the new agents, and I also did not
>> what
>>> > the
>>>
>>> >> CPU usage was before upgrading to 3.5. Fortunately the command
>>>
>>> > failures
>>>
>>> >> we were seeing the other day appear to have stopped after the
>> reboot,
>>> >> although the GUI is still very slow to respond to many typical
>>>
>>> > commands.
>>>
>>> >> For example, it just took 30 seconds for Anthill to respond to a
>> click
>>> >
>>>
>>> >> on a project displayed in the "Your Active Projects" column of
> the
>>> >> Dashboard. This seems to be slower than what it was in 3.4.12,
> but
>> as
>>> > I
>>>
>>> >> indicated we have added a lot of agents lately. We'll be adding
>> more
>>> > as
>>>
>>> >> well, exceeding 100 agents in the next while. Is there anything
> we
>>> >> should do to improve performance?
>>>
>>> >>
>>>
>>> >>
>>>
>>> >>
>>>
>>> >> *From:* anthill-pro-bounces at lists.urbancode.com
>>>
>>> >> [mailto:anthill-pro-bounces at lists.urbancode.com] *On Behalf Of
>> *Steve
>>> > Boone
>>>
>>> >> *Sent:* Wednesday, May 21, 2008 9:39 AM
>>>
>>> >> *To:* AnthillPro user and support list.
>>>
>>> >> *Subject:* Re: [Anthill-pro] What CPU usage is expected by
> Anthill
>>> > server?
>>>
>>> >>
>>>
>>> >>
>>>
>>> >> Peter,
>>>
>>> >>
>>>
>>> >> Has this been happening in a regular basis?
>>>
>>> >>
>>>
>>> >> What happened after the restart?
>>>
>>> >>
>>>
>>> >> On Wed, May 14, 2008 at 6:44 PM, Peter Steele
>> <psteele at maxiscale.com
>>> >> <mailto:psteele at maxiscale.com>> wrote:
>>>
>>> >>
>>>
>>> >> We've noticed some very slow feedback from the Anthill server
>> today,
>>> >> even coming back with failures for some operations. The Unix top
>>>
>>> > command
>>>
>>> >> is showing the ah3server process as taking as much as 100% CPU at
>>>
>>> > times,
>>>
>>> >> and is consistently the top 1 or 2 processes on the system. There
>> are
>>> >> just two builds going on, and both of course are running on agent
>>>
>>> >> systems, not the server. I don't recall seeing this before. We
>>>
>>> > upgraded
>>>
>>> >> to 3.5 yesterday and I am wondering if that might be to blame.
>> It's
>>> >> certainly very slow right now; I'll restart the server as soon as
>> the
>>> >> builds are done...
>>>
>>> >>
>>>
>>> >>
>>>
>>> >>
>>>
>>> >>
>>>
>>> >> _______________________________________________
>>>
>>> >> Anthill-pro mailing list
>>>
>>> >> Anthill-pro at lists.urbancode.com
>>>
>>> > <mailto:Anthill-pro at lists.urbancode.com>
>>>
>>> >> http://lists.urbancode.com/mailman/listinfo/anthill-pro
>>>
>>> >>
>>>
>>> >>
>>>
>>> >>
>>>
>>> >>
>>>
>>> >>
>>>
>>> >
> ------------------------------------------------------------------------
>>> >> _______________________________________________
>>>
>>> >> Anthill-pro mailing list
>>>
>>> >> Anthill-pro at lists.urbancode.com
>>>
>>> >> http://lists.urbancode.com/mailman/listinfo/anthill-pro
>>>
>>> >
>>>
>>> > --
>>>
>>> > ===========================================================
>>>
>>> > Ryan Smith. 2044 Euclid Ave., Suite 600
>>>
>>> > Developer Cleveland, Ohio 44115
>>>
>>> > Urbancode, Inc.
>>>
>>> > email: rws at urbancode.com
>>>
>>> > web: www.urbancode.com phone: 216-858-9000
>>>
>>> > web: www.anthillpro.com fax: 216-858-9602
>>>
>>> > ===========================================================
>>>
>>> > _______________________________________________
>>>
>>> > Anthill-pro mailing list
>>>
>>> > Anthill-pro at lists.urbancode.com
>>>
>>> > http://lists.urbancode.com/mailman/listinfo/anthill-pro
>>>
>>> > _______________________________________________
>>>
>>> > Anthill-pro mailing list
>>>
>>> > Anthill-pro at lists.urbancode.com
>>>
>>> > http://lists.urbancode.com/mailman/listinfo/anthill-pro
>>>
>>> >
>>>
>>> >
>>>
>>> > This e-mail, including attachments, may include confidential
> and/or
>>> > proprietary information, and may be used only by the person or
>> entity to
>>> > which it is addressed. If the reader of this e-mail is not the
>> intended
>>> > recipient or his or her authorized agent, the reader is hereby
>> notified
>>> > that any dissemination, distribution or copying of this e-mail is
>>>
>>> > prohibited. If you have received this e-mail in error, please
>> notify the
>>> > sender by replying to this message and delete this e-mail
>> immediately.
>>> >
>>>
>>> >
>>>
>>>
>>>
>>> --
>>>
>>> ===========================================================
>>>
>>> Ryan Smith. 2044 Euclid Ave., Suite 600
>>>
>>> Developer Cleveland, Ohio 44115
>>>
>>> Urbancode, Inc.
>>>
>>> email: rws at urbancode.com
>>>
>>> web: www.urbancode.com phone: 216-858-9000
>>>
>>> web: www.anthillpro.com fax: 216-858-9602
>>>
>>> ===========================================================
>>>
>>> _______________________________________________
>>>
>>> Anthill-pro mailing list
>>>
>>> Anthill-pro at lists.urbancode.com
>>>
>>> http://lists.urbancode.com/mailman/listinfo/anthill-pro
>>>
>
--
===========================================================
Ryan Smith. 2044 Euclid Ave., Suite 600
Developer Cleveland, Ohio 44115
Urbancode, Inc.
email: rws at urbancode.com
web: www.urbancode.com phone: 216-858-9000
web: www.anthillpro.com fax: 216-858-9602
===========================================================
More information about the Anthill-pro
mailing list