Re: SGE priorities and job dependency
This archive was generated by
Check out the -hold_jid option to qsub. This should do exactly what you
On Sun, Sep 1, 2013 at 3:19 AM, Jacob Barhak <jacob.barhak_at_gmail.com> wrote:
> Does someone have experience with the SGE scheduler that comes with
> StarCluster? Experienced enough to figure out how make a dependent job
> launch before other jobs once its dependencies are gone?
> I have been trying to give the dependent job a high priority, yet it seems
> the scheduler ignores this and launches the jobs in FIFO order.
> Here is a simplified description of my problem. Lets say I have 100k jobs.
> The first 50k are generating files in a shared directory and the last 50k
> are processing those files and deleting them.
> Jobs 1-50000 are independent while job 50001 is dependent on job 50000 and
> job 50000+n is dependent on job n.
> I tried lowering the priority of the first 50k jobs using qsub -p -100. I
> was hoping to get the effect of job 1 completing and since job 50001
> dependency was satisfied and has the highest priority then it would be
> launched next. The idea is to perform cleanup of the file after each job -
> otherwise too many files can accumulate in a shared directory and may slow
> down the NFS I/O significantly - not to mention disk space.
> However, I cannot get this behavior from SGE on a single test machine
> outside StarCluster. So I assume this needs some special configuration.
> I am trying to avoid the I/O bottleneck I experienced on the cloud due to
> too many files in a shared directory.
> Can someone help with this without changing the order if the jobs when
> being launched?
> I hope there is a simple one line / qsub option for this.
> Sent from my iPhone
> StarCluster mailing list
Received on Mon Sep 02 2013 - 21:45:46 EDT