{"id":238,"date":"2016-12-15T16:13:40","date_gmt":"2016-12-15T21:13:40","guid":{"rendered":"https:\/\/www.bates.edu\/research-resources\/?page_id=238"},"modified":"2026-03-20T13:46:40","modified_gmt":"2026-03-20T17:46:40","slug":"running-jobs","status":"publish","type":"page","link":"https:\/\/www.bates.edu\/research-resources\/leavitt-hpc-cluster\/running-jobs\/","title":{"rendered":"Running Jobs"},"content":{"rendered":"\n<h5 class=\"wp-block-heading\">&nbsp;<\/h5>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"1\">Quick Start Guide to Running Jobs on Leavitt<\/h4>\n\n\n\n<p><span style=\"font-weight: 400;\">Once you have gained access to the Leavitt cluster (<\/span><a href=\"https:\/\/www.bates.edu\/research-resources\/leavitt-hpc-cluster\/request-access-or-support\/\">see access request page<\/a><span style=\"font-weight: 400;\">), you will be able to access Leavitt via SSH using a suitable terminal application, such as Terminal (Mac machines) or PuTTY (Windows machines).&nbsp;<\/span><\/p>\n\n\n\n<p><span style=\"font-weight: 400;\">Jobs on Leavitt are typically run from the command line using the terminal application of your choosing and the bash shell, so it is a good idea to gain some basic familiarity with these tools if you plan to be using the cluster regularly (see <a href=\"https:\/\/www.bates.edu\/research-resources\/leavitt-hpc-cluster\/faq\/#What-resources-are-available-to-help-me-learn-more-about-the-Linux-command-line-and\/or-bash-scripting?\">FAQ page<\/a> for a list of guides and video tutorials). <\/span><\/p>\n\n\n\n<p><a href=\"https:\/\/www.bates.edu\/ils\/2011\/how-do-i\/network\/terminal-settings-remote-access-for-macintosh-ssh\/\">Mac Users &#8211; Terminal<\/a><\/p>\n\n\n\n<p><span style=\"font-weight: 400;\">To login to Leavitt with your terminal application, simply type: <\/span><code lang=\"bash\" style=\"background: #d3d3d3;\">ssh leavitt.bates.edu<\/code><\/p>\n\n\n\n<p><a href=\"https:\/\/www.bates.edu\/ils\/2011\/how-do-i\/network\/putty-install-and-configure-remote-access-for-windows-ssh\/\">Windows Users &#8211; PuTTY<\/a><\/p>\n\n\n\n<p><span style=\"font-weight: 400;\">To login to Leavitt using PuTTY, simply enter <\/span><code lang=\"bash\" style=\"background: #d3d3d3;\">leavitt.bates.edu<\/code><span style=\"font-weight: 400;\"> in the Host Name field, and make sure the Connection type is set to SSH.<\/span><\/p>\n\n\n\n<p><span style=\"font-weight: 400;\">Once you enter your standard Bates username and password, you will be brought to your home directory on the Leavitt cluster: <\/span><\/p>\n\n\n\n<figure class=\"wp-block-image\"><a href=\"https:\/\/www.bates.edu\/research-resources\/files\/2016\/12\/Screen-Shot-2016-11-22-at-10.15.56-AM.png\"><img loading=\"lazy\" decoding=\"async\" width=\"1182\" height=\"546\" src=\"https:\/\/www.bates.edu\/research-resources\/files\/2016\/12\/Screen-Shot-2016-11-22-at-10.15.56-AM.png\" alt=\"screen-shot-2016-11-22-at-10-15-56-am\" class=\"wp-image-259\" srcset=\"https:\/\/www.bates.edu\/research-resources\/files\/2016\/12\/Screen-Shot-2016-11-22-at-10.15.56-AM.png 1182w, https:\/\/www.bates.edu\/research-resources\/files\/2016\/12\/Screen-Shot-2016-11-22-at-10.15.56-AM-400x185.png 400w, https:\/\/www.bates.edu\/research-resources\/files\/2016\/12\/Screen-Shot-2016-11-22-at-10.15.56-AM-768x355.png 768w, https:\/\/www.bates.edu\/research-resources\/files\/2016\/12\/Screen-Shot-2016-11-22-at-10.15.56-AM-900x416.png 900w, https:\/\/www.bates.edu\/research-resources\/files\/2016\/12\/Screen-Shot-2016-11-22-at-10.15.56-AM-200x92.png 200w\" sizes=\"(max-width: 1182px) 100vw, 1182px\" \/><\/a><\/figure>\n\n\n\n<p><\/p>\n\n\n\n<p><span style=\"font-weight: 400;\">Running jobs on Leavitt typically involves a couple steps: first, the appropriate modules are added to your environment, giving you access to the software packages or compilers your job needs to run. Second, you submit the job to the job scheduling system, Slurm, which also allows you to monitor your submitted jobs. <\/span><\/p>\n\n\n\n<p><span style=\"font-weight: 400;\">We will review each of these steps in detail in the following sections. <\/span><\/p>\n\n\n\n<p>&nbsp;<\/p>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"2\"><a id=\"Modules\"><\/a>Modules<\/h4>\n\n\n\n<p><span style=\"font-weight: 400;\">The Modules package allows users to dynamically configure their environment to run specific software packages and libraries. It also allows us to support potentially thousands of applications in a single system. <\/span><\/p>\n\n\n\n<p><span style=\"font-weight: 400;\">However, it is important to note that<\/span><i><span style=\"font-weight: 400;\"> not all software packages use Modules. <\/span><\/i><span style=\"font-weight: 400;\">Common software packages that many users or core software will need (e.g. Python 2.7 and editors like vi\/vim or emacs) are simply available on the system. Modules are used to customize the computing environment for specific users, for example to load a particular GNU or Intel compiler. <\/span><\/p>\n\n\n\n<p><span style=\"font-weight: 400;\">More information about the Modules package can be found on the <\/span><a href=\"http:\/\/modules.sourceforge.net\/man\/module.html\"><span style=\"font-weight: 400;\">module man-page<\/span><\/a><span style=\"font-weight: 400;\"> (short for manual page), which you can access by typing and executing \u2018<\/span><i><span style=\"font-weight: 400;\">man module\u2019<\/span><\/i><span style=\"font-weight: 400;\"> at the command prompt once you\u2019ve logged into Leavitt.<\/span><\/p>\n\n\n\n<p><span style=\"font-weight: 400;\">Here are a few of the most important commands:<\/span><\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">module avail<\/pre>\n\n\n\n<p><span style=\"font-weight: 400;\">The <\/span><i><span style=\"font-weight: 400;\">module avail<\/span><\/i><span style=\"font-weight: 400;\"> command will print a list of the available modules, or <\/span><i><span style=\"font-weight: 400;\">modulefiles<\/span><\/i><span style=\"font-weight: 400;\">. You should use this command&nbsp;as a guide when you need to load a particular module, as some of the modules have similar names and the list of available modules will change periodically as we add new software to the system. <\/span><\/p>\n\n\n\n<figure class=\"wp-block-image\"><a href=\"https:\/\/www.bates.edu\/research-resources\/files\/2016\/12\/Screen-Shot-2016-11-22-at-11.36.46-AM.png\"><img loading=\"lazy\" decoding=\"async\" width=\"1188\" height=\"1368\" src=\"https:\/\/www.bates.edu\/research-resources\/files\/2016\/12\/Screen-Shot-2016-11-22-at-11.36.46-AM.png\" alt=\"screen-shot-2016-11-22-at-11-36-46-am\" class=\"wp-image-260\" srcset=\"https:\/\/www.bates.edu\/research-resources\/files\/2016\/12\/Screen-Shot-2016-11-22-at-11.36.46-AM.png 1188w, https:\/\/www.bates.edu\/research-resources\/files\/2016\/12\/Screen-Shot-2016-11-22-at-11.36.46-AM-261x300.png 261w, https:\/\/www.bates.edu\/research-resources\/files\/2016\/12\/Screen-Shot-2016-11-22-at-11.36.46-AM-768x884.png 768w, https:\/\/www.bates.edu\/research-resources\/files\/2016\/12\/Screen-Shot-2016-11-22-at-11.36.46-AM-782x900.png 782w, https:\/\/www.bates.edu\/research-resources\/files\/2016\/12\/Screen-Shot-2016-11-22-at-11.36.46-AM-174x200.png 174w\" sizes=\"(max-width: 1188px) 100vw, 1188px\" \/><\/a><\/figure>\n\n\n\n<p><\/p>\n\n\n\n<p>&nbsp;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">module add &lt;module&gt;<\/pre>\n\n\n\n<p><span style=\"font-weight: 400;\">The <\/span><i><span style=\"font-weight: 400;\">module add<\/span><\/i><span style=\"font-weight: 400;\"> command (where &lt;module&gt; is the name of the module to be added) allows you to load a particular module into your shell environment. For example, executing <\/span><i><span style=\"font-weight: 400;\">module add gcc<\/span><\/i><span style=\"font-weight: 400;\"> adds the 6.1.0 implementation of the GNU Compiler Collection to your environment. <\/span><\/p>\n\n\n\n<p><span style=\"font-weight: 400;\">It\u2019s worth noting that some software packages or standards have multiple implementations available on Leavitt via distinct modules, such as the multiple Open MPI implementations shown here: <\/span><\/p>\n\n\n\n<figure class=\"wp-block-image\"><a href=\"https:\/\/www.bates.edu\/research-resources\/files\/2016\/12\/Screen-Shot-2016-11-23-at-9.58.26-AM.png\"><img loading=\"lazy\" decoding=\"async\" width=\"894\" height=\"178\" src=\"https:\/\/www.bates.edu\/research-resources\/files\/2016\/12\/Screen-Shot-2016-11-23-at-9.58.26-AM.png\" alt=\"screen-shot-2016-11-23-at-9-58-26-am\" class=\"wp-image-261\" srcset=\"https:\/\/www.bates.edu\/research-resources\/files\/2016\/12\/Screen-Shot-2016-11-23-at-9.58.26-AM.png 894w, https:\/\/www.bates.edu\/research-resources\/files\/2016\/12\/Screen-Shot-2016-11-23-at-9.58.26-AM-400x80.png 400w, https:\/\/www.bates.edu\/research-resources\/files\/2016\/12\/Screen-Shot-2016-11-23-at-9.58.26-AM-768x153.png 768w, https:\/\/www.bates.edu\/research-resources\/files\/2016\/12\/Screen-Shot-2016-11-23-at-9.58.26-AM-200x40.png 200w\" sizes=\"(max-width: 894px) 100vw, 894px\" \/><\/a><\/figure>\n\n\n\n<p><\/p>\n\n\n\n<p>&nbsp;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">module list<\/pre>\n\n\n\n<p><span style=\"font-weight: 400;\">The <\/span><i><span style=\"font-weight: 400;\">module list<\/span><\/i><span style=\"font-weight: 400;\"> command will return a list of all the currently loaded modules:<\/span><\/p>\n\n\n\n<figure class=\"wp-block-image\"><a href=\"https:\/\/www.bates.edu\/research-resources\/files\/2016\/12\/Screen-Shot-2016-11-23-at-9.07.32-AM.png\"><img loading=\"lazy\" decoding=\"async\" width=\"728\" height=\"164\" src=\"https:\/\/www.bates.edu\/research-resources\/files\/2016\/12\/Screen-Shot-2016-11-23-at-9.07.32-AM.png\" alt=\"screen-shot-2016-11-23-at-9-07-32-am\" class=\"wp-image-262\" srcset=\"https:\/\/www.bates.edu\/research-resources\/files\/2016\/12\/Screen-Shot-2016-11-23-at-9.07.32-AM.png 728w, https:\/\/www.bates.edu\/research-resources\/files\/2016\/12\/Screen-Shot-2016-11-23-at-9.07.32-AM-400x90.png 400w, https:\/\/www.bates.edu\/research-resources\/files\/2016\/12\/Screen-Shot-2016-11-23-at-9.07.32-AM-200x45.png 200w\" sizes=\"(max-width: 728px) 100vw, 728px\" \/><\/a><\/figure>\n\n\n\n<p><\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">module rm &lt;module&gt;<\/pre>\n\n\n\n<p><span style=\"font-weight: 400;\">The <\/span><i><span style=\"font-weight: 400;\">module rm <\/span><\/i><span style=\"font-weight: 400;\">command (where &lt;module&gt; is the name of the module to be removed) will remove the specified module from your environment. This can be helpful if you need to, for example, switch compilers. <\/span><\/p>\n\n\n\n<figure class=\"wp-block-image\"><a href=\"https:\/\/www.bates.edu\/research-resources\/files\/2016\/12\/Screen-Shot-2016-11-23-at-9.11.09-AM.png\"><img loading=\"lazy\" decoding=\"async\" width=\"644\" height=\"268\" src=\"https:\/\/www.bates.edu\/research-resources\/files\/2016\/12\/Screen-Shot-2016-11-23-at-9.11.09-AM.png\" alt=\"screen-shot-2016-11-23-at-9-11-09-am\" class=\"wp-image-263\" srcset=\"https:\/\/www.bates.edu\/research-resources\/files\/2016\/12\/Screen-Shot-2016-11-23-at-9.11.09-AM.png 644w, https:\/\/www.bates.edu\/research-resources\/files\/2016\/12\/Screen-Shot-2016-11-23-at-9.11.09-AM-400x166.png 400w, https:\/\/www.bates.edu\/research-resources\/files\/2016\/12\/Screen-Shot-2016-11-23-at-9.11.09-AM-200x83.png 200w\" sizes=\"(max-width: 644px) 100vw, 644px\" \/><\/a><\/figure>\n\n\n\n<p><\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">module purge<\/pre>\n\n\n\n<p><span style=\"font-weight: 400;\">Not to be confused with the above, the <\/span><i><span style=\"font-weight: 400;\">module purge <\/span><\/i><span style=\"font-weight: 400;\">command will remove <\/span><span style=\"font-weight: 400;\">all<\/span><span style=\"font-weight: 400;\"> modules from your environment. <\/span><\/p>\n\n\n\n<figure class=\"wp-block-image\"><a href=\"https:\/\/www.bates.edu\/research-resources\/files\/2016\/12\/Screen-Shot-2016-11-23-at-9.13.51-AM.png\"><img loading=\"lazy\" decoding=\"async\" width=\"688\" height=\"236\" src=\"https:\/\/www.bates.edu\/research-resources\/files\/2016\/12\/Screen-Shot-2016-11-23-at-9.13.51-AM.png\" alt=\"screen-shot-2016-11-23-at-9-13-51-am\" class=\"wp-image-264\" srcset=\"https:\/\/www.bates.edu\/research-resources\/files\/2016\/12\/Screen-Shot-2016-11-23-at-9.13.51-AM.png 688w, https:\/\/www.bates.edu\/research-resources\/files\/2016\/12\/Screen-Shot-2016-11-23-at-9.13.51-AM-400x137.png 400w, https:\/\/www.bates.edu\/research-resources\/files\/2016\/12\/Screen-Shot-2016-11-23-at-9.13.51-AM-200x69.png 200w\" sizes=\"(max-width: 688px) 100vw, 688px\" \/><\/a><\/figure>\n\n\n\n<p><\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">module initadd &lt;module&gt;<\/pre>\n\n\n\n<p><span style=\"font-weight: 400;\">The <\/span><i><span style=\"font-weight: 400;\">module initadd <\/span><\/i><span style=\"font-weight: 400;\">command allows more advanced users to add modules to the shell\u2019s initialization file (.bashrc) in the user\u2019s home directory so that these modules will be automatically loaded in future sessions. If you need to make changes to your .bashrc file in the future, you can edit the file with an editor like vim or emacs, or use the <\/span><i><span style=\"font-weight: 400;\">module initrm &lt;module&gt; <\/span><\/i><span style=\"font-weight: 400;\">command.<\/span><\/p>\n\n\n\n<p><span style=\"font-weight: 400;\">Note that this command does not load modules in the current environment. To do this you need to either start a new session or reload your .bashrc file using one of the following commands: <\/span><\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">source ~\/.bashrc<\/pre>\n\n\n\n<p><span style=\"font-weight: 400;\">or <\/span><\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">. ~\/.bashrc<\/pre>\n\n\n\n<p><span style=\"font-weight: 400;\">One of the default modules, preloaded for all users, is Slurm.<\/span><\/p>\n\n\n\n<p>&nbsp;<\/p>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"3\"><a id=\"Slurm\"><\/a>Slurm<\/h4>\n\n\n\n<p><span style=\"font-weight: 400;\">Slurm, or <\/span><i><span style=\"font-weight: 400;\">Simple Linux Utility for Resource Management<\/span><\/i><span style=\"font-weight: 400;\">, is an open-source cluster management and job scheduling system widely used on high performance computing clusters like Leavitt. Slurm is one of the most important software packages on Leavitt, where it is used to (1) allocate access to compute resources for users, (2) provide a framework for running and monitoring jobs, and (3) manage a queue for submitted jobs. As such, it is important for Leavitt users to have some familiarity with Slurm. <\/span><\/p>\n\n\n\n<p><span style=\"font-weight: 400;\">This documentation will provide an introduction to some of the most important concepts and commands, but there are many resources available online through a simple Google search (<a href=\"http:\/\/slurm.schedmd.com\/\">http:\/\/slurm.schedmd.com\/<\/a> is a good place to start). Additionally, you can always find documentation on a particular command when logged into Leavitt via the manual page (man-page): <\/span><\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">man command<\/pre>\n\n\n\n<p><span style=\"font-weight: 400;\">(e.g. <\/span><i><span style=\"font-weight: 400;\">\u201cman sinfo\u201d <\/span><\/i><span style=\"font-weight: 400;\">to see manual information on the sinfo command<\/span><span style=\"font-weight: 400;\">).<\/span><\/p>\n\n\n\n<p>&nbsp;<\/p>\n\n\n\n<h5 class=\"wp-block-heading\"><b>Common Slurm Commands<\/b><\/h5>\n\n\n\n<p>&nbsp;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">sinfo<\/pre>\n\n\n\n<p><span style=\"font-weight: 400;\">Executing the <\/span><i><span style=\"font-weight: 400;\">sinfo<\/span><\/i><span style=\"font-weight: 400;\"> command will show you information about the available compute nodes and partitions on Leavitt, including information about job limitations on each of the partitions, such as time limits (for a more in-depth description of each of the partitions, see the <a href=\"https:\/\/www.bates.edu\/research-resources\/leavitt-hpc-cluster\/running-jobs\/#Partitions\">Slurm Partitions section<\/a>).<\/span><\/p>\n\n\n\n<p><span style=\"font-weight: 400;\">Like most commands, there are many options available for output (view the sinfo man-page for a list). For example, you can view more detailed information about the available partitions by using the the <em>&#8211;long<\/em> option (e.g. <\/span><code lang=\"bash\" style=\"background: #d3d3d3;\">sinfo --long<\/code><span style=\"font-weight: 400;\"> (<\/span><span style=\"font-weight: 400;\"><em>-l<\/em> for short):<\/span><\/p>\n\n\n\n<figure class=\"wp-block-image\"><a href=\"https:\/\/www.bates.edu\/research-resources\/files\/2016\/12\/Screen-Shot-2016-11-30-at-10.00.36-AM.png\"><img loading=\"lazy\" decoding=\"async\" width=\"1190\" height=\"178\" src=\"https:\/\/www.bates.edu\/research-resources\/files\/2016\/12\/Screen-Shot-2016-11-30-at-10.00.36-AM.png\" alt=\"screen-shot-2016-11-30-at-10-00-36-am\" class=\"wp-image-275\" srcset=\"https:\/\/www.bates.edu\/research-resources\/files\/2016\/12\/Screen-Shot-2016-11-30-at-10.00.36-AM.png 1190w, https:\/\/www.bates.edu\/research-resources\/files\/2016\/12\/Screen-Shot-2016-11-30-at-10.00.36-AM-400x60.png 400w, https:\/\/www.bates.edu\/research-resources\/files\/2016\/12\/Screen-Shot-2016-11-30-at-10.00.36-AM-768x115.png 768w, https:\/\/www.bates.edu\/research-resources\/files\/2016\/12\/Screen-Shot-2016-11-30-at-10.00.36-AM-900x135.png 900w, https:\/\/www.bates.edu\/research-resources\/files\/2016\/12\/Screen-Shot-2016-11-30-at-10.00.36-AM-200x30.png 200w\" sizes=\"(max-width: 1190px) 100vw, 1190px\" \/><\/a><\/figure>\n\n\n\n<p><\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">squeue<\/pre>\n\n\n\n<p><span style=\"font-weight: 400;\">Use the <\/span><i><span style=\"font-weight: 400;\">squeue<\/span><\/i><span style=\"font-weight: 400;\"> command to view a list of jobs that have been submitted to the Slurm scheduling queue. This is a useful way to monitor jobs that have been submitted, and also to retrieve job ID\u2019s, which are necessary to cancel or get statistics on jobs (see <\/span><i><span style=\"font-weight: 400;\">scancel <\/span><\/i><span style=\"font-weight: 400;\">and <\/span><i><span style=\"font-weight: 400;\">sacct<\/span><\/i><span style=\"font-weight: 400;\"> sections below). <\/span><\/p>\n\n\n\n<p><span style=\"font-weight: 400;\">To view only jobs submitted by a particular user, use the <em>-u<\/em><\/span> <span style=\"font-weight: 400;\">option, e.g. <\/span><code lang=\"bash\" style=\"background: #d3d3d3;\">squeue -u kevenson<\/code><\/p>\n\n\n\n<p>&nbsp;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">sbatch<\/pre>\n\n\n\n<p><span style=\"font-weight: 400;\">The <\/span><i><span style=\"font-weight: 400;\">sbatch<\/span><\/i><span style=\"font-weight: 400;\"> command is used with serial batch jobs, and typically includes a submission script. For example, executing <\/span><code lang=\"bash\" style=\"background: #d3d3d3;\">sbatch myscript.sh<\/code><span style=\"font-weight: 400;\"> would submit the <\/span><i><span style=\"font-weight: 400;\">myscript.sh<\/span><\/i><span style=\"font-weight: 400;\"> shell script to Slurm.<\/span><\/p>\n\n\n\n<p><span style=\"font-weight: 400;\">Submission scripts may also include options preceded by <\/span><code lang=\"bash\" style=\"background: #d3d3d3;\">#SBATCH<\/code><span style=\"font-weight: 400;\"> to set key job parameters, such as the partition to be used, memory, nodes, etc. <\/span><\/p>\n\n\n\n<p><span style=\"font-weight: 400;\">See the examples below or the sbatch man-page for more information. <\/span><\/p>\n\n\n\n<p>&nbsp;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">srun<\/pre>\n\n\n\n<p><span style=\"font-weight: 400;\">The <\/span><i><span style=\"font-weight: 400;\">srun<\/span><\/i><span style=\"font-weight: 400;\"> command can be used to run parallel and interactive jobs with Slurm. It can be used within an existing job (e.g. in a script submitted using sbatch<\/span><span style=\"font-weight: 400;\">), or as a way to create a new job allocation and spawn an application (e.g. for X11 forwarding to run a graphical application like Jupyter Notebook or Matlab). <\/span><\/p>\n\n\n\n<p>&nbsp;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">scancel<\/pre>\n\n\n\n<p><span style=\"font-weight: 400;\">The <\/span><i><span style=\"font-weight: 400;\">scancel<\/span><\/i><span style=\"font-weight: 400;\"> command provides a simple way to cancel Slurm jobs. There are a few ways this can done, but here are a few of the most frequently used options:<\/span><\/p>\n\n\n\n<p><span style=\"font-weight: 400;\">To cancel one job: <\/span><code lang=\"bash\" style=\"background: #d3d3d3;\">scancel &lt;jobid&gt; <\/code><\/p>\n\n\n\n<p><span style=\"font-weight: 400;\">To cancel all the jobs for a user: <\/span><code lang=\"bash\" style=\"background: #d3d3d3;\">scancel -u &lt;username&gt; <\/code><\/p>\n\n\n\n<p><span style=\"font-weight: 400;\">To cancel all the pending jobs for a user: <\/span><code lang=\"bash\" style=\"background: #d3d3d3;\">scancel -t PENDING -u &lt;username&gt;<\/code><\/p>\n\n\n\n<p><span style=\"font-weight: 400;\">The <\/span><i><span style=\"font-weight: 400;\">scancel<\/span><\/i><span style=\"font-weight: 400;\"> command can also be used to cancel Slurm job arrays and job steps&#8211;see the man-page for more information.<\/span><\/p>\n\n\n\n<p>&nbsp;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">sacct<\/pre>\n\n\n\n<p><i><span style=\"font-weight: 400;\">sacct<\/span><\/i><span style=\"font-weight: 400;\"> displays accounting information for jobs submitted to Slurm and saved in the accounting log or Slurm database. This is a good way to retrieve basic statistics about a particular job. <\/span><\/p>\n\n\n\n<p><span style=\"font-weight: 400;\">For example, to retrieve stats for a completed job: <\/span><code lang=\"bash\" style=\"background: #d3d3d3;\">sacct -j &lt;jobid&gt; --format=JobID,JobName,MaxRSS,Elapsed<\/code><\/p>\n\n\n\n<p>&nbsp;<\/p>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"4\"><b><a id=\"Submitting Batch Jobs\"><\/a>Submitting Batch Jobs<\/b><\/h4>\n\n\n\n<p><span style=\"font-weight: 400;\">The most common way to run jobs on Leavitt is to submit a script to Slurm using the <\/span><i><span style=\"font-weight: 400;\">sbatch<\/span><\/i><span style=\"font-weight: 400;\"> command. Submissions scripts will vary based on the specifics of the&nbsp;jobs being run, but there are a few common components, which we\u2019ll review in the example scripts below. <\/span><\/p>\n\n\n\n<p><b>Example 1: <\/b><i><span style=\"font-weight: 400;\">myfirstscript.sh<\/span><\/i><\/p>\n\n\n\n<figure class=\"wp-block-image\"><a href=\"https:\/\/www.bates.edu\/research-resources\/files\/2016\/12\/Screen-Shot-2016-12-20-at-11.31.32-AM.png\"><img loading=\"lazy\" decoding=\"async\" width=\"1486\" height=\"692\" src=\"https:\/\/www.bates.edu\/research-resources\/files\/2016\/12\/Screen-Shot-2016-12-20-at-11.31.32-AM.png\" alt=\"screen-shot-2016-12-20-at-11-31-32-am\" class=\"wp-image-276\" srcset=\"https:\/\/www.bates.edu\/research-resources\/files\/2016\/12\/Screen-Shot-2016-12-20-at-11.31.32-AM.png 1486w, https:\/\/www.bates.edu\/research-resources\/files\/2016\/12\/Screen-Shot-2016-12-20-at-11.31.32-AM-400x186.png 400w, https:\/\/www.bates.edu\/research-resources\/files\/2016\/12\/Screen-Shot-2016-12-20-at-11.31.32-AM-768x358.png 768w, https:\/\/www.bates.edu\/research-resources\/files\/2016\/12\/Screen-Shot-2016-12-20-at-11.31.32-AM-900x419.png 900w, https:\/\/www.bates.edu\/research-resources\/files\/2016\/12\/Screen-Shot-2016-12-20-at-11.31.32-AM-200x93.png 200w\" sizes=\"(max-width: 1486px) 100vw, 1486px\" \/><\/a><\/figure>\n\n\n\n<p><\/p>\n\n\n\n<p><span style=\"font-weight: 400;\">This is a very simple script, created on Leavitt using the vi editor and titled <\/span><i><span style=\"font-weight: 400;\">myfirstscript.sh.&nbsp;<\/span><\/i><span style=\"font-weight: 400;\">(To create the script, you can use the&nbsp;<em>touch<\/em> command, e.g.&nbsp;<code lang=\"bash\" style=\"background: #d3d3d3;\">touch myfirstscript.sh<\/code>; to then edit the script with the vi editor, you would execute&nbsp;<code lang=\"bash\" style=\"background: #d3d3d3;\">vi myfirstscript.sh<\/code>)<\/span><span style=\"font-weight: 400;\">. <\/span><\/p>\n\n\n\n<p><span style=\"font-weight: 400;\">To submit the script, we\u2019d simply execute <\/span><code lang=\"bash\" style=\"background: #d3d3d3;\">sbatch myfirstscript.sh<\/code><span style=\"font-weight: 400;\"> at the command prompt, and Slurm would give us our job ID:<\/span><\/p>\n\n\n\n<figure class=\"wp-block-image\"><a href=\"https:\/\/www.bates.edu\/research-resources\/files\/2016\/12\/Screen-Shot-2016-12-06-at-3.55.24-PM.png\"><img loading=\"lazy\" decoding=\"async\" width=\"814\" height=\"116\" src=\"https:\/\/www.bates.edu\/research-resources\/files\/2016\/12\/Screen-Shot-2016-12-06-at-3.55.24-PM.png\" alt=\"screen-shot-2016-12-06-at-3-55-24-pm\" class=\"wp-image-277\" srcset=\"https:\/\/www.bates.edu\/research-resources\/files\/2016\/12\/Screen-Shot-2016-12-06-at-3.55.24-PM.png 814w, https:\/\/www.bates.edu\/research-resources\/files\/2016\/12\/Screen-Shot-2016-12-06-at-3.55.24-PM-400x57.png 400w, https:\/\/www.bates.edu\/research-resources\/files\/2016\/12\/Screen-Shot-2016-12-06-at-3.55.24-PM-768x109.png 768w, https:\/\/www.bates.edu\/research-resources\/files\/2016\/12\/Screen-Shot-2016-12-06-at-3.55.24-PM-200x29.png 200w\" sizes=\"(max-width: 814px) 100vw, 814px\" \/><\/a><\/figure>\n\n\n\n<p><\/p>\n\n\n\n<p><span style=\"font-weight: 400;\">We\u2019ll go through the script line-by-line below, but it is worth noting the three main components first:<\/span><\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li><span style=\"font-weight: 400;\">The first line of this script tells the system which interpreter to use. This is sometimes called the <\/span><i><span style=\"font-weight: 400;\">shebang<\/span><\/i><span style=\"font-weight: 400;\"> or <\/span><i><span style=\"font-weight: 400;\">hashbang<\/span><\/i><span style=\"font-weight: 400;\"> line. The syntax is standard in Unix-like systems, with the number sign and an exclamation point (#!) followed by the path to the interpreter to be used: <\/span><code lang=\"bash\" style=\"background: #d3d3d3;\">#!\/path\/to\/interpreter<\/code><span style=\"font-weight: 400;\">. <\/span><\/li>\n\n\n\n<li><span style=\"font-weight: 400;\">The #SBATCH lines (e.g. <\/span><code lang=\"bash\" style=\"background: #d3d3d3;\">#SBATCH -p defq<\/code><span style=\"font-weight: 400;\">), which make up the bulk of this particular script, are used by Slurm to set key parameters for the job. The text that you see following \u201c#SBATCH\u201d are the options specifying these parameters. See <\/span><a href=\"https:\/\/slurm.schedmd.com\/sbatch.html\"><span style=\"font-weight: 400;\">https:\/\/slurm.schedmd.com\/sbatch.html<\/span><\/a><span style=\"font-weight: 400;\"> or the sbatch man-page for more information. <\/span><\/li>\n\n\n\n<li><span style=\"font-weight: 400;\">The last part of the script, in this trivial example beginning with \u201c<\/span><i><span style=\"font-weight: 400;\">for j in\u2026<\/span><\/i><span style=\"font-weight: 400;\">\u201d is the most important part&#8211;this is the code that comprises most of the job steps, or tasks that must be done for your job to complete, such as modules to load or algorithms to solve (another job step is the script itself). <\/span><\/li>\n<\/ol>\n\n\n\n<p>&nbsp;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">#!\/bin\/bash<\/pre>\n\n\n\n<p><span style=\"font-weight: 400;\">As noted above, this is the <\/span><i><span style=\"font-weight: 400;\">shebang<\/span><\/i><span style=\"font-weight: 400;\"> line that tells the system what interpreter to use. When the script above is run, the <\/span><code lang=\"bash\" style=\"background: #d3d3d3;\">\/bin\/bash<\/code><span style=\"font-weight: 400;\"> text tells the system to run the script using the bash shell found in the \/bin directory. <\/span><\/p>\n\n\n\n<p><span style=\"font-weight: 400;\">Another example of a shebang line, this one for Python 2.7 users, would be <\/span><code lang=\"bash\" style=\"background: #d3d3d3;\">#!\/usr\/bin\/python<\/code><\/p>\n\n\n\n<p>&nbsp;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">#SBATCH -p defq<\/pre>\n\n\n\n<p><span style=\"font-weight: 400;\">The<\/span><i><span style=\"font-weight: 400;\"> -p<\/span><\/i><span style=\"font-weight: 400;\"> option in this line tells Slurm what partition to use. In this case, it\u2019s telling Slurm to use the <\/span><i><span style=\"font-weight: 400;\">defq<\/span><\/i><span style=\"font-weight: 400;\"> partition, which also happens to be the default partition that all Leavitt users have access to. <\/span><\/p>\n\n\n\n<p><b>Note<\/b><span style=\"font-weight: 400;\">: the partition used for a job determines many of the options that will be available. For example, the <\/span><i><span style=\"font-weight: 400;\">defq <\/span><\/i><span style=\"font-weight: 400;\">has a maximum job size of 6 nodes. To see a list of partitions and their usage options, see the SECTION BELOW, or use the <\/span><i><span style=\"font-weight: 400;\">sinfo -la<\/span><\/i><span style=\"font-weight: 400;\"> command for a summary.<\/span><\/p>\n\n\n\n<p>&nbsp;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">#SBATCH -N 1<\/pre>\n\n\n\n<p><span style=\"font-weight: 400;\">The<\/span><i><span style=\"font-weight: 400;\"> -N<\/span><\/i><span style=\"font-weight: 400;\"> option tells Slurm how many nodes to allocate to this job. In this case, only one node is being requested. <\/span><\/p>\n\n\n\n<p>&nbsp;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">#SBATCH --ntasks-per-node=8<\/pre>\n\n\n\n<p><span style=\"font-weight: 400;\">The<\/span><i><span style=\"font-weight: 400;\"> &#8211;ntasks-per-node=&lt;ntasks&gt;<\/span><\/i><span style=\"font-weight: 400;\"> option tells Slurm how many tasks to invoke on each node, and will be treated as a maximum count of tasks per node. Note: this is meant to be used with the <\/span><i><span style=\"font-weight: 400;\">-N <\/span><\/i><span style=\"font-weight: 400;\">option, above. <\/span><\/p>\n\n\n\n<p><span style=\"font-weight: 400;\">maximum number of tasks for which it needs to allocate resources. The default is for one task per node, so this essentially tells Slurm how many cores to allocate to the job. (Remember, each node has 2 CPU\u2019s\/sockets, each with 14 cores.)<\/span><\/p>\n\n\n\n<p>&nbsp;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">#SBATCH --mem-1000<\/pre>\n\n\n\n<p><span style=\"font-weight: 400;\">The<\/span><i><span style=\"font-weight: 400;\"> &#8211;mem <\/span><\/i><span style=\"font-weight: 400;\">option tells Slurm how much memory you expect to be required per node in megabytes. In this particular case, 1000MB were requested <\/span><span style=\"font-weight: 400;\">(note that in the core Leavitt cluster, each of the compute nodes has 128GB of memory).<\/span><\/p>\n\n\n\n<p><b>Remember<\/b><span style=\"font-weight: 400;\">: Leavitt is a shared resource, so it is very important that you make accurate requests for the amount of memory you need. If insufficient memory is requested, your job may fail. On the flip side, if you request much more memory than you need for your job, you may be taking resources that another Leavitt user needs. <\/span><\/p>\n\n\n\n<p>&nbsp;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">#SBATCH -t 0-01:1:30<\/pre>\n\n\n\n<p><span style=\"font-weight: 400;\">The<\/span><i><span style=\"font-weight: 400;\"> -t<\/span><\/i><span style=\"font-weight: 400;\"> option allows you to set a limit on the total run time for your job. Multiple formats are acceptable (see the sbatch man-page for more information), but in this example, a time limit of 1 hour and 30 minutes was imposed. <\/span><\/p>\n\n\n\n<p><span style=\"font-weight: 400;\">Note that this option may have no real bearing on how long your job takes to actually run. In this example, the job completed in approximately 19 seconds. <\/span><\/p>\n\n\n\n<p>&nbsp;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">#SBATCH -o slurm-%u-%N-%j.out<\/pre>\n\n\n\n<p><span style=\"font-weight: 400;\">The<\/span><i><span style=\"font-weight: 400;\"> -o<\/span><\/i><span style=\"font-weight: 400;\"> option tells Slurm to direct its output to the filename specified. The filename pattern can include a number of replacement symbols. In the example above, the \u2018%u\u2019 is replaced with the username associated with the job, the \u2018%N\u2019 is replaced by the name of the node that runs the script and \u2018%j\u2019 is replaced by the job allocation number\u2014resulting in an output file called <\/span><i><span style=\"font-weight: 400;\">slurm-kevenson-node001-242.out<\/span><\/i><span style=\"font-weight: 400;\">.<\/span><\/p>\n\n\n\n<p>&nbsp;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">#SBATCH -e slurm-%u-%N-%j.err<\/pre>\n\n\n\n<p><span style=\"font-weight: 400;\">Similar to the above, the<\/span><i><span style=\"font-weight: 400;\"> -e<\/span><\/i><span style=\"font-weight: 400;\"> option tells Slurm to direct its standard error directly to the filename specified. The filename pattern options are the same, so this particular line results in an error file called <\/span><i><span style=\"font-weight: 400;\">slurm-kevenson-node001-242.err<\/span><\/i><span style=\"font-weight: 400;\">.<\/span><\/p>\n\n\n\n<p>&nbsp;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">#SBATCH --mail-type=END,FAIL\n#SBATCH --mail-user=kevenson@bates.edu<\/pre>\n\n\n\n<p><span style=\"font-weight: 400;\">The<\/span><i><span style=\"font-weight: 400;\"> &#8211;mail-type <\/span><\/i><span style=\"font-weight: 400;\">and<\/span><i><span style=\"font-weight: 400;\"> &#8211;mail-user<\/span><\/i><span style=\"font-weight: 400;\"> options tell Slurm (a) to notify a user by email when certain event types occur (here END and FAIL) and (b) what user should be notified when these events occur. <\/span><\/p>\n\n\n\n<p><span style=\"font-weight: 400;\">See the sbatch man-page for more type values. <\/span><\/p>\n\n\n\n<p>&nbsp;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">for j in {1..50000}; do \n   echo $RANDOM &gt;&gt; MyRandomNumbers.txt \ndone\n\nsort -n MyRandomNumbers.txt<\/pre>\n\n\n\n<p><span style=\"font-weight: 400;\">These remaining lines comprise the real work you want to do. The for loop generates a long, long list of random numbers, which are appended to the file MyRandomNumbers.txt. This is then sorted in ascending order in the output file. <\/span><\/p>\n\n\n\n<p><span style=\"font-weight: 400;\">So, nothing too interesting in this example but this is where you would instruct Slurm to load particular modules, open another script with your custom software or perform additional job steps with the <\/span><i><span style=\"font-weight: 400;\">srun<\/span><\/i><span style=\"font-weight: 400;\"> command.<\/span><\/p>\n\n\n\n<p>&nbsp;<\/p>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"5\"><b><a id=\"Available Partitions\"><\/a>Slurm Partitions<\/b><\/h4>\n\n\n\n<p><span style=\"font-weight: 400;\">Partitions in Slurm are essentially queues for jobs. Partitions all have differing levels of access to the resources available on the Leavitt cluster. You may have access to different partitions based on your user type (e.g. if you are a faculty member or a student), and you may choose to use different partitions based on the types of jobs you need run. <\/span><\/p>\n\n\n\n<p><span style=\"font-weight: 400;\">As of this writing, the L<\/span><span style=\"font-weight: 400;\">eavitt cluster is composed of 11 Dell PowerEdge 430&#8217;s compute nodes and one Dell PowerEdge 730XD head node. Each of the compute nodes has 128GB of memory and dual Intel Xeon E5-2680 processors, each with 14 cores running at 2.40GHz, for a total of 308 cores and 1.4 terabytes of memory. The head node has an additional 48 terabytes of disk storage, which is shared amongst the compute nodes. The 12 nodes are also connected by an Intel Omni-Path switch, which provides very high speed (100 gigabits\/second) network access between each member in the cluster.<\/span><\/p>\n\n\n\n<p><span style=\"font-weight: 400;\">Below is an overview of the available partitions on Leavitt.<\/span><\/p>\n\n\n\n<h5 class=\"wp-block-heading\"><b><i>defq<\/i><\/b><\/h5>\n\n\n\n<p><span style=\"font-weight: 400;\">The defq or default queue is available to all Leavitt users, including students. This queue has access to a maximum of roughly half of Leavitt\u2019s resources for a maximum of 48 hours, making this an appropriate partition for classroom work. <\/span><\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><span style=\"font-weight: 400;\">Max compute nodes<\/span><span style=\"font-weight: 400;\">: 6 (12 sockets \/ 168 total cores)<\/span><\/li>\n\n\n\n<li><span style=\"font-weight: 400;\">Max job time<\/span><span style=\"font-weight: 400;\">: 48 hours<\/span><\/li>\n\n\n\n<li><span style=\"font-weight: 400;\">Default time<\/span><span style=\"font-weight: 400;\">: 2 hours<\/span><\/li>\n<\/ul>\n\n\n\n<p><i><span style=\"font-weight: 400;\">Note that if you do not specify a partition when you submit your job, it will automatically run on this partition. <\/span><\/i><\/p>\n\n\n\n<p>&nbsp;<\/p>\n\n\n\n<h5 class=\"wp-block-heading\"><b><i>faculty<\/i><\/b><\/h5>\n\n\n\n<p><span style=\"font-weight: 400;\">The faculty queue is available to all Leavitt faculty and staff users. This queue has access to all Leavitt\u2019s resources, but jobs submitted to this queue have a lower priority than jobs submitted to the contributor queue, detailed below. <\/span><\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><span style=\"font-weight: 400;\">Max compute nodes<\/span><span style=\"font-weight: 400;\">: 11 (22 sockets \/ 308 total cores)<\/span><\/li>\n\n\n\n<li><span style=\"font-weight: 400;\">Max job time<\/span><span style=\"font-weight: 400;\">: 15 days<\/span><\/li>\n\n\n\n<li><span style=\"font-weight: 400;\">Default time<\/span><span style=\"font-weight: 400;\">: 6 hours<\/span><\/li>\n<\/ul>\n\n\n\n<p>&nbsp;<\/p>\n\n\n\n<h5 class=\"wp-block-heading\"><b><i>contributor<\/i><\/b><\/h5>\n\n\n\n<p><span style=\"font-weight: 400;\">The contributor queue is restricted to faculty who have contributed to the expansion of the Leavitt cluster\u2019s resources, e.g. through grant applications or the purchasing of additional nodes through startup funds.<\/span><\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><span style=\"font-weight: 400;\">Max compute nodes<\/span><span style=\"font-weight: 400;\">: 11 (22 sockets \/ 308 total cores)<\/span><\/li>\n\n\n\n<li><span style=\"font-weight: 400;\">Max job time<\/span><span style=\"font-weight: 400;\">: unlimited<\/span><\/li>\n\n\n\n<li><span style=\"font-weight: 400;\">Default time<\/span><span style=\"font-weight: 400;\">: 10 days<\/span><\/li>\n<\/ul>\n\n\n\n<p>&nbsp;<\/p>\n\n\n\n<h5 class=\"wp-block-heading\"><b><i>interactive<\/i><\/b><\/h5>\n\n\n\n<p><span style=\"font-weight: 400;\">tbd<\/span><\/p>\n","protected":false},"excerpt":{"rendered":"<p>&nbsp; Quick Start Guide to Running Jobs on Leavitt Once you have&hellip;<\/p>\n","protected":false},"author":34,"featured_media":0,"parent":215,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"_hide_ai_chatbot":false,"_ai_chatbot_style":"","associated_faculty":[],"_Page_Specific_Css":"","_bates_restrict_mod":false,"_table_of_contents_display":true,"_table_of_contents_location":"sidebarBottom","_table_of_contents_disableSticky":false,"_is_featured":false,"footnotes":"","_bates_seo_meta_description":"","_bates_seo_block_robots":false,"_bates_seo_sharing_image_id":0,"_bates_seo_sharing_image_twitter_id":0,"_bates_seo_share_title":"","_bates_seo_canonical_overwrite":"","_bates_seo_twitter_template":""},"class_list":["post-238","page","type-page","status-publish","hentry","with-sidebar"],"_links":{"self":[{"href":"https:\/\/www.bates.edu\/research-resources\/wp-json\/wp\/v2\/pages\/238","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.bates.edu\/research-resources\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/www.bates.edu\/research-resources\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/www.bates.edu\/research-resources\/wp-json\/wp\/v2\/users\/34"}],"replies":[{"embeddable":true,"href":"https:\/\/www.bates.edu\/research-resources\/wp-json\/wp\/v2\/comments?post=238"}],"version-history":[{"count":5,"href":"https:\/\/www.bates.edu\/research-resources\/wp-json\/wp\/v2\/pages\/238\/revisions"}],"predecessor-version":[{"id":1022,"href":"https:\/\/www.bates.edu\/research-resources\/wp-json\/wp\/v2\/pages\/238\/revisions\/1022"}],"up":[{"embeddable":true,"href":"https:\/\/www.bates.edu\/research-resources\/wp-json\/wp\/v2\/pages\/215"}],"wp:attachment":[{"href":"https:\/\/www.bates.edu\/research-resources\/wp-json\/wp\/v2\/media?parent=238"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}