> For the complete documentation index, see [llms.txt](https://docs.idea-hpc.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.idea-hpc.com/modules/hpc-workloads/user-documentation/troubleshooting/my-job-is-not-starting.md).

# My job is not starting

Jobs can stay in the Q state for extended period of time because of license restrictions or other applicationS specific requirements. Follow this guide if you think all conditions are met but your job is still stuck in the Q state.

## 1 - Start with the job status in the web portal

Open **My jobs > Active** and select the job. A job that has not started yet reports:

* how long it has been waiting
* which provisioning attempt it is on, out of `scheduler.job_provisioning.max_provisioning_retries`
* the queue limit holding it, if one is

If the attempt count is climbing, provisioning is being retried and the reason is in the job's error message. If a queue limit is named, the job is waiting for the queue to drop below that limit and no action is needed.

If the status reads **held after N of N attempts**, the job has used all of `scheduler.job_provisioning.max_provisioning_retries`, 3 out of the box. Provisioning has stopped retrying and the job will not start on its own. Read the error message, fix what it names, then release the job with `qrls <job_id>` or delete it with `qdel <job_id>` and resubmit.

IDEA does not report a queue position or an estimated start time. Compute capacity is created on demand rather than drawn from a fixed pool, so neither can be computed from what the scheduler knows.

## 2 - Check whether the job is waiting on its project's AI model access

A job whose project has Bedrock enabled is held back until that project's model access is usable, so it never runs without the access the portal lists for it. The job's error message names which:

* **its instance profile has not been provisioned yet**, or **none of its models have been provisioned yet** - the cluster is still reconciling the project. Nothing to do; the job starts on the next provisioning cycle, usually within a minute of the project edit.
* **the scheduler module is not permitted to launch compute nodes under a project role** - the module was not deployed with Bedrock enabled for jobs. An administrator has to redeploy it.
* **not authorized for this queue** - the queue profile's `allowed_instance_profiles` does not include the project's instance profile. An administrator has to add it.
* **no model allowed on project ... could be provisioned** - every model the project allows was rejected, and the reason per model is on the project in the web portal.

The last three do not resolve on their own, and jobs submitted while they hold are rejected at submission rather than queued.

## 3 - Verify if the capacity is being provisioned

Open **My jobs > Active**, select the job, and inspect **Status Reason** and **Compute Stack**. A named queue limit or capacity wait means the job is still eligible but blocked. A provisioning stack in the details means capacity creation has started. A held disposition means automatic retries have stopped.

{% hint style="info" %}
You can login to AWS Console and navigate to the CloudFormation console to verify the CloudFormation stack associated to your job is in `CREATE_COMPLETE` state. If not, verify any potential errors via the `Events` tab.
{% endhint %}

## 4 - Verify the bootstrap logs for the compute node(s) being provisioned

If the capacity is being provisioned, the next thing to check is if there is no errors during the bootstrap sequence on the compute node(s) provisioned to run your job.

To verify that, review logs located under `/apps/<CLUSTER>/scheduler/jobs/<JOB_ID>/`

You will find the bootstrap & compute\_node logs for all EC2 instances being provisioned for your job:

<pre><code><strong>Bootstrap:
</strong><strong>/apps/&#x3C;CLUSTER>/scheduler/jobs/&#x3C;JOB_ID>/bootstrap/&#x3C;COMPUTE_NODE_JOB_ID>
</strong>
<strong>Compute Node Startup Logs:
</strong>/apps/&#x3C;CLUSTER>/scheduler/jobs/&#x3C;JOB_ID>/logs/&#x3C;INSTANCE_HOSTNAME>
</code></pre>

## 5 - Check if the compute node(s) is/are being registered on the scheduler

Verify if the compute node(s) are being registered correctly to the scheduler.

Run `pbsnodes -a` and find the section specific to your job id (see example below for job 103)

```bash
p-10-110-6-141
     Mom = ip-10-110-6-141.us-east-2.compute.internal
     Port = 15002
     pbs_version = unavailable
     ntype = PBS
     state = state-unknown,down
     resources_available.anonymous_metrics = True
     resources_available.auto_scaling_group = idea-demo-compute-ondemand-103-AutoScalingComputeGroup-YXWJFA4XLGKQ
     resources_available.availability_zone = us-east-2c
     resources_available.base_os = rocky9
     resources_available.cluster_name = idea-demo
     resources_available.compute_node = idea-demo-compute-ondemand-103
     resources_available.efa_support = True
     resources_available.force_ri = False
     resources_available.fsx_lustre = False
     resources_available.host = ip-10-110-6-141
     resources_available.ht_support = False
     resources_available.instance_ami = ami-0c1c3220d0b1716d2
     resources_available.instance_id = i-02447d465fbe84723
     resources_available.instance_profile = AIPA6ERFY3V55DSVSI2OZ
     resources_available.instance_type = c5n.9xlarge
     resources_available.job_group = gf4ea4e11
     resources_available.job_id = 103
     resources_available.keep_ebs = False
     resources_available.keep_forever = False
     resources_available.launch_time = 1671017843
     resources_available.placement_group = False
     resources_available.provisioning_time = 1671017862089
     resources_available.queue_type = compute
     resources_available.root_size = 10gb
     resources_available.scaling_mode = single-job
     resources_available.scratch_iops = 0
     resources_available.scratch_size = 0gb
     resources_available.stack_id = arn:aws:cloudformation:us-east-2:<REDACTED>:stack/idea-demo-compute-ondemand-103/9caa5e10-7ba3-11ed-837d-060b0d4719a4
     resources_available.subnet_id = subnet-064c905368b057b68
     resources_available.system_metrics = False
     resources_available.tenancy = default
     resources_available.terminate_when_idle = 0
     resources_available.vnode = ip-10-110-6-141
     resources_assigned.accelerator_memory = 0kb
     resources_assigned.hbmem = 0kb
     resources_assigned.mem = 0kb
     resources_assigned.naccelerators = 0
     resources_assigned.ncpus = 0
     resources_assigned.vmem = 0kb
     queue = normal
     resv_enable = True
     sharing = default_shared
```

In this example, the host is still being configured as state is `state = state-unknown,down.` If that's the case, wait a little longer. Your host will be ready to accept job when `state = free` (see below)

```
ip-10-110-6-141
     Mom = ip-10-110-6-141.us-east-2.compute.internal
     ntype = PBS
     state = free
     pcpus = 18
     resources_available.anonymous_metrics = True
     resources_available.arch = linux
     resources_available.auto_scaling_group = idea-demo-compute-ondemand-103-AutoScalingComputeGroup-YXWJFA4XLGKQ
     resources_available.availability_zone = us-east-2c
     resources_available.base_os = rocky9
     resources_available.cluster_name = idea-demo
     resources_available.compute_node = idea-demo-compute-ondemand-103
     resources_available.efa_support = True
     resources_available.force_ri = False
     resources_available.fsx_lustre = False
     resources_available.host = ip-10-110-6-141
     resources_available.ht_support = False
     resources_available.instance_ami = ami-0c1c3220d0b1716d2
     resources_available.instance_id = i-02447d465fbe84723
     resources_available.instance_profile = AIPA6ERFY3V55DSVSI2OZ
     resources_available.instance_type = c5n.9xlarge
     resources_available.job_group = gf4ea4e11
     resources_available.job_id = 103
     resources_available.keep_ebs = False
     resources_available.keep_forever = False
     resources_available.launch_time = 1671017843
     resources_available.mem = 96640680kb
     resources_available.ncpus = 18
     resources_available.placement_group = False
     resources_available.provisioning_time = 1671017862089
     resources_available.queue_type = compute
     resources_available.root_size = 10gb
     resources_available.scaling_mode = single-job
     resources_available.scratch_iops = 0
     resources_available.scratch_size = 0gb
     resources_available.stack_id = arn:aws:cloudformation:us-east-2:<REDACTED>:stack/idea-demo-compute-ondemand-103/9caa5e10-7ba3-11ed-837d-060b0d4719a4
     resources_available.subnet_id = subnet-064c905368b057b68
     resources_available.system_metrics = False
     resources_available.tenancy = default
     resources_available.terminate_when_idle = 0
     resources_available.vnode = ip-10-110-6-141
     resources_assigned.accelerator_memory = 0kb
     resources_assigned.hbmem = 0kb
     resources_assigned.mem = 0kb
     resources_assigned.naccelerators = 0
     resources_assigned.ncpus = 0
     resources_assigned.vmem = 0kb
     queue = normal
     resv_enable = True
     sharing = default_shared
     license = l
     last_state_change_time = Wed Dec 14 11:49:10 2022
```

## 6 - Inspect and restart the scheduler task

The scheduler runs as one ECS task. Connect to its application container using the [ECS Exec procedure](https://docs.idea-hpc.com/help-and-support/faq.md#cluster-management) and inspect `/opt/idea/app/logs`, `/var/spool/pbs/sched_logs` and `/var/spool/pbs/server_logs`. Correct the reported cause before replacing the task.

To restart it, force a new deployment of the scheduler ECS service:

```bash
aws --profile <PROFILE> --region <REGION> ecs update-service \
  --cluster <ECS_CLUSTER> --service <SCHEDULER_SERVICE> --force-new-deployment
```

Wait for the service to stabilize and confirm that the application logs show a healthy startup. The scheduler stops before its replacement starts, so submissions and the scheduler API are briefly unavailable. PBS state persists across task replacement.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://docs.idea-hpc.com/modules/hpc-workloads/user-documentation/troubleshooting/my-job-is-not-starting.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
