[Anthill-pro] Failed to create
directory.../anthill3/server/var/log/br34171
Varban
vvv at urbancode.com
Fri Feb 1 09:43:03 CST 2008
Uldis,
I guess my first suggestion would be to have a more aggressive and fine
grained cleanup policy - just go trough you build lives and check which
ones you actually want to keep and for the rest use a special status
that would be used by the cleanup policy to remove them (and doing it
once per day will probably be just fine). Also do not forget the cleanup
the build requests as well (they produce the brxxx folders).
The problem with the folders is that we need to create them, whether
they are all under the same folder or in a more complicated tree is not
going to change matters since var/log directory is still going to have
the same amount of sub directories.
The stats for you drives look fine, but I guess you cleaned up the
folders already.
As far as the file system goes, you do not need to switch the whole
server. All you need to do is create a new partition (it can be on the
same drive or a different one) with a different file system and move the
anthill3 install to that partition (you should be able to copy the files
to the new partition and mount it at the original anthill3 install
directory). This way the var/log directory will end up on the new file
system without affecting you server in any way.
Let us know in a day or two how things are looking going so that we can
potentially look for a better solution.
Regards,
Varban
Uldis Karlovs-Karlovskis wrote:
> Varban,
>
> Yes You`re absolutely right in all of Your comments about my build system
> and its customizations. It were needed to get functionality running one
> Workflow in parallel many times. I have already configured my Clean-Up so it
> is deleting all buildlifes older than 3 days and all "not needed". It is
> running every 3 hours. Let`s see what will happen after 3 days.
> So the only suggestion is to change file system. My sysadmins have the same
> idea but it is big task to create new FS, test it, move all server to it
> etc. I can`t show You how many folders there were because I`ve cleaned up
> them to get server running. So I don`t know where is threshold. Imo it is
> just not right to keep all logs in this way. I`m pretty sure another
> customer will run into same problem from time to time.
>
>
> [root at anthill ~]# df -i
> Filesystem Inodes IUsed IFree IUse% Mounted on
> /dev/sda2 2.5M 75K 2.5M 3% /
> /dev/sda1 50K 51 49K 1% /boot
> tmpfs 493K 1 493K 1% /dev/shm
> /dev/sda6 113M 2.2M 111M 2% /export/storage0
> /dev/sda3 1.0M 2.0K 1023K 1% /var
>
> [root at anthill ~]# df -h
> Filesystem Size Used Avail Use% Mounted on
> /dev/sda2 9.7G 3.9G 5.4G 42% /
> /dev/sda1 190M 44M 138M 24% /boot
> tmpfs 2.0G 0 2.0G 0% /dev/shm
> /dev/sda6 436G 170G 244G 42% /export/storage0
> /dev/sda3 3.9G 661M 3.1G 18% /var
>
>
> Uldis Karlovs-Karlovskis, C. T. Co
> uldis.karlovs-karlovskis at ctco.lv
>
>
> -----Original Message-----
> From: anthill-pro-bounces at lists.urbancode.com
> [mailto:anthill-pro-bounces at lists.urbancode.com] On Behalf Of Varban
> Sent: piektdiena, 2008. gada 1. februārī 17:13
> To: AnthillPro user and support list.
> Cc: 'AnthillPro user and support list.'
> Subject: Re: [Anthill-pro] Failed to create
> directory.../anthill3/server/var/log/br34171
>
> Uldis,
>
> this sounds like a file system limitation rather than linux kernel one.
> What file system are you using?
> In any case, can you send me the output of the following commands:
> df -i
> df -h
> That will check the inode and space usage on your file system.
> Also run
> ls -al | wc -l
> in the var/log directory to find out how many directories there are.
> From a previous email I see that your workflows have a bunch of jobs
> that compile, build and create artifacts of separate parts of the
> project - that seems like a scenario for one job with multiple steps
> rather than separate jobs?
> I also see that you have a job at the beginning of a workflow that
> checks if a build is necessary - I would assume that in most cases that
> just creates a build life without actually doing anything - maybe
> assigning a special status to that buildlife and having a cleanup policy
> that would remove those buildlives every day (along with maybe cleaning
> up all but the latest 3-4 failed builds every day) would greatly reduce
> the number of subfolders?
> And as a last resort you could look into moving the AnthillPro install
> to a partition with a file system that allows for a larger number of
> subfolders (reiserfs?).
>
> Regards,
> Varban
> Uldis Karlovs-Karlovskis wrote:
>> Hello!
>>
>>
>>
>> I`m running into new problems with AntHill. I`ve been just waiting when
>> it will happen and now it is.
>>
>> The folder var/log/ is reaching Linux kernel folders per directory limit
>> so then all processes in anthill is crushing. Now I have configured my
>> Cleanup to delete all history older than 3 days so threshold is not
>> reached anymore, but imo it is not normal - I want at least 5 days
> history.
>>
>>
>> //Uldis Karlovs - Karlovskis//
>>
>> //Build System and Deployment Administrator//
>>
>> //System Configuration Management//
>>
>> //C.T.Co//
>>
>> //Mobile////: +371 29 345 210//
>>
>>
>>
>>
>> ------------------------------------------------------------------------
>>
>> _______________________________________________
>> Anthill-pro mailing list
>> Anthill-pro at lists.urbancode.com
>> http://lists.urbancode.com/mailman/listinfo/anthill-pro
> _______________________________________________
> Anthill-pro mailing list
> Anthill-pro at lists.urbancode.com
> http://lists.urbancode.com/mailman/listinfo/anthill-pro
>
>
> _______________________________________________
> Anthill-pro mailing list
> Anthill-pro at lists.urbancode.com
> http://lists.urbancode.com/mailman/listinfo/anthill-pro
More information about the Anthill-pro
mailing list