1. Sge Allocation Rule Pe Slots For Pc
  2. Sge Allocation Rule Pe Slots 2
  3. Sge Allocation Rule Pe Slots 2017

Introduction

Sun Grid Engine is a job scheduling system that is widely used in computing clusters today. Users should use use qmon (configure SGE by GUI) and qconf (configure SEG by command line) to configure SGE.

Jun 22, 2012 I would like to submit this job to a SGE cluster but I am not sure which. Parallel Environment for a multithreaded java code. You need a PE with an allocation. Qstat -j gave me this: cannot run in PE 'smp' because it only offers - slots On which platform / OS / SGE version do you observe this?

Detailed Reference

With this rule, if a user requests 8 slots and a single machine has 8 slots available, that job will run entirely on one machine. If 5 slots are available on one host and 3 on another, it will take all 5 on that host, and all 3 on the other host. Sun Grid Engine is a job scheduling system that is widely used in computing clusters today. Users should use use qmon (configure SGE by GUI) and qconf (configure SEG by command line) to configure SGE. This is extremely important to set up for any cluster, to enable to limit the amount of memory that. The format of a sgepe file is defined as follows: pename The name of the parallel environment in the format for pename in sgetypes(1). To be used in the qsub(1) -pe switch. Slots The total number of slots (normally one per parallel process or thread) allowed to be filled concurrently under the parallel environment. Don't change the values in Defaults.pm, those should be passed in on the command line. If you really want to change the code, make the change in GridSGE.pm. We used to allow (in Celera Assembler) a dotfile to hold common settings. I have written a multi threaded java code, which when runs creates 8 threads and the computation continues on these threads. I would like to submit this job to a SGE cluster but I am not sure which parallel environment (pe) should I choose? Or should I create one?I am new to SGE.

The more authorative and comprehensive reference is http://gridscheduler.sourceforge.net/htmlman/htmlman1/qstat.html and http://gridscheduler.sourceforge.net/htmlman/htmlman1/qconf.html

A simpler configuration tutorial is given at http://ait.web.psi.ch/services/linux/hpc/merlin3/sge/admin/sge_queues.html

SGE cheat sheet

To view the basic cluster and host configuration use the qconf -sconf command:

To modify the basic cluster or host specific configuration use the following commands:

To view and change the setup of hosts use qconf -XY [file] with the following options:

Great reference on SGE configurationhttp://arc.liv.ac.uk/SGE/howto/sge-configs.html

Configuration of h_vmem for any new cluster

This is extremely important to set up for any cluster, to enable to limit the amount of memory that each job can request to avoid memory issues for user jobs. The key is to make h_vmem requestable.

Use qconf -mc to change the line to

So that h_vmem is consumable with default value of 4G.

Now run the example command below 20 times to confirm:'echo 'sleep 120' qsub -cwd -V -l hostname=compute-0-0If default is 4G, and if the resource is 48G at compute-0-0, then only 12 jobs can be run, and 8 jobs will be put into waiting list.

Configuration of parallel environment

check parameters of a pe

To add a new pe such as smpfirst edit a file pe.txt

Then run qconf -Ap pe.txt as root. Now smp can be used as a PE in the qsub argument.

Next thing is to add the new PE smp into the all.q queue by qconf -mq all.q, and add smp to the pe_list line in the file.

SGE complex value for host

Then use 'complex_values h_vmem=48G', to set 48G for a particular host as h_vmem value.

Extremely important: whenever adding a new host, one must use 'qconf -me' to set up complex_values. In combination with -l h_vmem=xG in the qsub command, this will eliminate the possibility of running out of memory when multiple jobs in the same host all request large chunks of memory at the same time.

Temporarily enable or disable a host

qmod -d all.q@compute-* should disable all the queue instances

job status

'au' simply means that Grid Engine is likely not running on the node. The 'a' means 'alarm' and the 'u' means unheard/unreachable. The combination of the two more often than not means that SGE is not running on the compute node.

E is a worse state to see. It means that there was a major problem on the compute node (with the system or the job itself). SGE intentionally marked the queue as state 'E' so that other jobs would not run into the same bad problem.

E states do not go away automatically, even if you reboot the cluster. Once you think the cluster is fine you can use the 'qmod' command to clear the E state.

##host status

  • 'au' - Host is in alarm and unreachable,
  • 'u' - Host is unreachable. Usually SGE is down or the machine is down. Check this out.
  • 'a' - Host is in alarm. It is normal on if the state of the node is full, it means, if on the node is using most of its resources.
  • 'aS' - Host is in alarm and Suspended. If the node is using most of its resources, SGE suspends this node to take any other job unless resources are available.
  • 'd' - Host is disabled,
  • 'E' - ERROR. This requires the command qmod -c to clear the error state.

When job is in dr state

When deleting a job, sometimes a 'dr' will show up, indicating that the job is not running correctly and cannot be easily deleted. In this case, log in as 'su', then 'qdel ' to delete the job forcefully. If it does not work, do 'qdel -f ' to delete the job.

Sge Allocation Rule Pe Slots For Pc

When node is in E state

re-install the node, then clear the Error log:

See more explanations here: http://www.gridengine.info/2008/01/20/understanding-queue-error-state-e/

When machine restarts yet nodes are still full, use qstat -u '*' to show whose jobs are in dr state, then qdel these jobs

If a node continues to be in E state after clearing the error multiple times by qmod, then it is likely that there is a hardware error. In this case, try rocks set host runaction compute-0-0 action=memtest and restart the node to check for potential issues.

Check error messages from SGE

less /opt/gridengine/default/spool/qmaster/messages

Restart SGE

You can find these in $SGE_ROOT//common/

If your cell is the usual 'default' then all you need to do is:

Before you restart the master, make sure you don't have any old
sge_qmaster or sge_schedd processes hanging around.

Fair share policy

There are two types of fair shares: share tree versus functional.

  1. Make 2 changes in the main SGE configuration ('qconf -mconf'):* enforce_user auto* auto_user_fshare 100

  2. Make 1 change in the SGE scheduler configuration ('qconf -msconf'):* weight_tickets_functional 10000

Very useful tricks

  • Restart a failed job

If you job fails in a node (the node should show up as 'au' status in qstat), you can restart the job in a different node. First, alter the job to be restartable, then submit it again.

You will see that the status of the job becomes 'Rq', and soon it will be submitted to a different node.

  • Clear error for a job

Sometimes you will see that a job is at 'Eqw' state in qstat. This is due to errors in running the job, usually due to NFS error in the node in my experience. If you fixed the error, you can clear the error message by qmod -cj <jobid>, and the job will be submitted again.

  • Change priority of a job

Use qlater -p <priority> <jobid> to change the priority of a job. The valid range is -1024 to 1023. Lower number means lower priority. Regular users can only lower the priority. This applies only to queued jobs, not running jobs.

Adding a new queue

We want to add a new queue using all.q as the template:

Then edit the bigmem.q file (change qname to bigmem, change hostlist to @bigmemhosts, change slots to something like 1,[dragon.local=32] where dragon.local is a host in bigmemhosts), then

to add this queue.

Later you can directly edit it qconf -mq bigmem and further change the hostlist and slots parameter there.

For example, to switch a host from the all.q to bigmem, we can do this: (1) first qconf -mhgrp @allhosts to remove it, then qconf -ahgrp @bigmemhosts to add it. Then add this hostgroup to the bigmem queue.

default submission parameters

If you have multiple queues, it makes sense to set up a default queue, since SGE rand

Edit the /opt/gridengine/default/common/sge_request file, add -q all.q as the default queue.

From user's perspective, they can also use a default SGE specification file .sge_request in their home directory. If they do not specify the parameters in command line, these defaults from the file will be used.

Change appliance type

Sge Allocation Rule Pe Slots 2

Some times you may want to chagne a NAS to a job execution host. This can be done by changing appliance type.

shows all appliance types.

change the appliance type. Then re-install the node, and it will show up in qhost.

  • Add Frontend as a SGE Execution Host in Rocks

General Information / Documentation

Tight MPICH2 Integration with SGE

  • For generic instructions, see Tight MPICH2 Integration in Grid Engine
  • For Rocks-specific instructions, see sge-tight-mpich2-integration

Tight MPICH Integration with SGE

Rocks comes with a default parallel environment for SGE named mpich that facilitates tight integration of MPICH1. Unfortunately it's not quite complete. For the ch_p4 mpich device, the environment variable MPICH_PROCESS_GROUP must be set to no on both the frontend and compute nodes in order for SGE to maintain itself as process group leader. These are the steps I did to get it to work in Rocks 4.1 (I opted for the 2nd solution described in Tight MPICH Integration in Grid Engine, Nmichaud@jhu.edu, 14:11, 15 February 2006 EST)

Sge Allocation Rule Pe Slots 2017

Min bet poker 66 casino albuquerque. 1. Edit /opt/gridengine/default/common/sge_request and add the following line at the end:

2. For a default Rocks setup, SGE calls /opt/gridengine/mpi/startmpi when starting an mpi job. This in turn calls /opt/gridengine/mpi/rsh. Both these files must be changed. However, each compute node has its own copy of these files. Instead of editing it on the frontend and copying to all the compute nodes, I found it easier to place my own copy in a subdirectory of /share/apps called mpi and then change the mpich parallel environment to call my own copies of startmpi and stopmpi (and by extension rsh). This way my one copy is exported to all the nodes and I don't have to worry about keeping them in sync.

3. Edit /share/apps/mpi/startmpi. Change the line:

to:

4. Edit /share/apps/mpi/rsh. Change the following lines:

to:

5. Finally, run qconf -mp mpich. Change it from:

to:

Prologue/Epilogue

This is a place for information about the prologue and epilogue scripts that run before and after a job, respectively.

I have found that SGE is not particularly good at cleaning up after MPI jobs; it does not keep track of which other nodes are being used and killing leftover user processes on those nodes. If anyone has a good solution for this, I'd love to see it.

(Note: MPI tight integration is suppose to fix this, see http://web.archive.org/web/20080916135356/http://gridengine.sunsource.net/howto/lam-integration/lam-integration.html)

Add Frontend as a SGE Execution Host in Rocks

To setup the frontend node to also be a SGE execution host which queued jobs can be run on (like the compute nodes), do the following:

Quick Setup

Detailed Setup

1. As root, make sure $SGE_ROOT, etc. are setup correctly on the frontend:

It should return back something like:

If not, source the file /etc/profile.d/sge-binaries.[c]sh or check if the SGE Roll is properly installed and enabled:

2. Run the install_execd script to setup the frontend as a SGE execution host:

Accept all of the default answers as suggested by the script.

  • NOTE: For the following examples below, the text Frontend should be substituted with the actual 'short hostname' of your frontend (as reported by the command hostname -s).
For example, if running the command hostname on your frontend returns back the 'FQDN long hostname' of:

then hostname -s should return back just:

3. Verify that the number of job slots for the frontend is equal to the number of physical processors/cores on your frontend that you wish to make available for queued jobs by checking the value of the slots parameter of the queue configuration for all.q:

The [Frontend.local=4] means that SGE can run up to 4 jobs on the frontend. Be aware that since the frontend is normally used for other tasks besides running compute jobs, it is recommended that not all the installed physical processors/cores on the frontend be available to be scheduled by SGE to avoid overloading the frontend.

For example, on a 4-core frontend, to configure SGE to use only up to 3 of the 4 cores, you can modify the slots for Frontend.local from 4 to 3 by typing:

If there are additional queues besides the default all.q one, repeat the above for each queue.

Read 'man queue_conf' for a list of resource limit parameters such as s_cpu, h_cpu, s_vmem, and h_vmem that can be adjusted to prevent jobs from overloading the frontend.

  • NOTE: For Rocks 5.2 or older, the frontend may have been default configured during installation with only 1 job slot ([Frontend.local=1]) in the default all.q queue, which will only allow up to 1 queued job to run on the frontend. To check the value of the slots parameter of the queue configuration for all.q, type:

If needed, modify the slots for <frontend>.local from 1 to 4 (or up to the maximum number of physical processors/cores on your frontend that you wish to use) by typing:

  • NOTE: For Rocks 5.3 or older, create the file /opt/gridengine/default/common/host_aliases to contain both the .local hostname and the FQDN long hostname of your frontend:
  • NOTE: For Rocks 5.3 or older, edit the file /opt/gridengine/default/common/act_qmaster to contain the .local hostname of your frontend:
  • NOTE: For Rocks 5.3 or older, edit the file /etc/init.d/sgemaster.<frontend>:
Rule

and comment out the line:

by inserting a # character at the beginning, so it becomes:

in order to prevent the file /opt/gridengine/default/common/act_qmaster from getting overwritten with incorrect data every time sgemaster.Frontend is run during bootup.

4. Restart both qmaster and execd for SGE on the frontend:

And everything will start working. :)

References: