## Description:
How to launch distributed Megatron-LM training jobs on a SLURM cluster, covering a minimal sbatch skeleton, environment-variable setup for torch.distributed.run, CUDA_DEVICE_MAX_CONNECTIONS rules across hardware and parallelism modes, container conventions, monitoring, and per-rank failure diagnosis.
This skill is ready for commercial/non-commercial use.
## Owner
NVIDIA
### License/Terms of Use:
Apache-2.0
## Use Case:
Developers and engineers submitting distributed Megatron-LM training jobs on SLURM clusters, writing or debugging sbatch scripts, and diagnosing multi-node job failures.
### Deployment Geography for Use:
Global
## Known Risks and Mitigations:
Risk: Review before execution as proposals could introduce incorrect or misleading guidance into skills.
Mitigation: Review and scan skill before deployment.
## Reference(s):
- [Megatron Core Developer Guide](https://docs.nvidia.com/megatron-core/developer-guide/latest/index.html)
- [Megatron-LM GitHub Repository](https://github.com/NVIDIA/Megatron-LM)
## Skill Output:
**Output Type(s):** [Shell commands, Configuration instructions]
**Output Format:** [Markdown with inline bash code blocks]
**Output Parameters:** [1D]
**Other Properties Related to Output:** [None]
## Evaluation Tasks:
3-Tier NVSkills-Eval evaluation (external profile). Tier 1 static validation passed with 7 findings across 9 checks. Tier 2 deduplication passed with 0 findings.
## Evaluation Metrics Used:
Reported benchmark dimensions:
- Security: Checks whether skill-assisted execution avoids unsafe behavior such as secret leakage, destructive commands, or unauthorized access.
- Correctness: Checks whether the agent follows the expected workflow and produces the correct final output.
- Discoverability: Checks whether the agent loads the skill when relevant and avoids using it when irrelevant.
- Effectiveness: Checks whether the agent performs measurably better with the skill than without it.
- Efficiency: Checks whether the agent uses fewer tokens and avoids redundant work.
## Skill Version(s):
1325db3b5 (source: git SHA, committed 2026-05-29)
## Ethical Considerations:
NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their internal team to ensure this skill meets requirements for the relevant industry and use case and addresses unforeseen product misuse.
(For Release on NVIDIA Platforms Only)
Please report quality, risk, security vulnerabilities or NVIDIA AI Concerns [here](https://app.intigriti.com/programs/nvidia/nvidiavdp/detail).