SGE priorities and job dependency

From: Jacob Barhak <no email>
Date: Sun, 1 Sep 2013 02:19:12 -0500

Hello,

Does someone have experience with the SGE scheduler that comes with StarCluster? Experienced enough to figure out how make a dependent job launch before other jobs once its dependencies are gone?

I have been trying to give the dependent job a high priority, yet it seems the scheduler ignores this and launches the jobs in FIFO order.

Here is a simplified description of my problem. Lets say I have 100k jobs. The first 50k are generating files in a shared directory and the last 50k are processing those files and deleting them.

Jobs 1-50000 are independent while job 50001 is dependent on job 50000 and job 50000+n is dependent on job n.

I tried lowering the priority of the first 50k jobs using qsub -p -100. I was hoping to get the effect of job 1 completing and since job 50001 dependency was satisfied and has the highest priority then it would be launched next. The idea is to perform cleanup of the file after each job - otherwise too many files can accumulate in a shared directory and may slow down the NFS I/O significantly - not to mention disk space.

However, I cannot get this behavior from SGE on a single test machine outside StarCluster. So I assume this needs some special configuration.

I am trying to avoid the I/O bottleneck I experienced on the cloud due to too many files in a shared directory.

Can someone help with this without changing the order if the jobs when being launched?

I hope there is a simple one line / qsub option for this.

       Jacob



Sent from my iPhone
Received on Sun Sep 01 2013 - 03:19:23 EDT