This repository was archived by the owner on Jan 15, 2026. It is now read-only.
Description Referencing this tutorial: https://docs.nvidia.com/nemo-framework/user-guide/latest/playbooks/kubernetes.html#nemo-framework-on-kubernetes-playbook
NeMo Container image: nvcr.io/nvidia/nemo:24.03.01.framework
ENV PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python in the container dose not work.
I fixed this issue by adding the environment variable in the mpirun command.
mpirun -x PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python
- ' {{- range tuple "download" "extract" "preprocess" }} mpirun --allow-run-as-root -np {{ $config.totalProcesses }} -npernode {{ $config.procsPerNode }} -bind-to none -map-by slot --oversubscribe -x PYTHONPATH -mca pml ob1 -mca btl ^openib python3 /opt/NeMo-Megatron-Launcher/launcher_scripts/nemo_launcher/collections/dataprep_scripts/pile_dataprep/{{ . }}.py --config-path=/config --config-name=config.yaml && {{- end}} echo Data preparation complete'
Reactions are currently unavailable
Referencing this tutorial: https://docs.nvidia.com/nemo-framework/user-guide/latest/playbooks/kubernetes.html#nemo-framework-on-kubernetes-playbook
an error occurred during the Data Preparation stage.
NeMo Container image: nvcr.io/nvidia/nemo:24.03.01.framework
ENV PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python in the container dose not work.
I fixed this issue by adding the environment variable in the mpirun command.
NeMo-Framework-Launcher/launcher_scripts/nemo_launcher/core/k8s_templates/data_preparation/data-prep.yaml
Line 20 in d14e267