[Bug Report] Multi-GPU: setting n_devices leads to "ValueError: `size` auto is not in a valid format."

[sjgerstner]

Describe the bug
I'm trying to spread parameters of a big model over more than one GPU. I assumed the canonical way to do this is setting n_devices in boot_transformers(). But this leads to "ValueError: size auto is not in a valid format." (See full error message below.)

Code example

from transformer_lens.model_bridge import TransformerBridge
#I'm just showing a very small model here as an example, the bug appears with all models I tested
model = TransformerBridge.boot_transformers('SimpleStories/SimpleStories-1.25M', n_devices=2)

Error message:

  File "/mounts/Users/cisintern/sgerstner/test_multi_gpu.py", line 3, in <module>
    model = TransformerBridge.boot_transformers('SimpleStories/SimpleStories-1.25M', n_devices=2)
  File "/mounts/work/sgerstner/.conda/envs/tl3var/lib/python3.10/site-packages/transformer_lens/model_bridge/sources/transformers.py", line 649, in boot
    hf_model = model_class.from_pretrained(model_name, **model_kwargs)
  File "/mounts/work/sgerstner/.conda/envs/tl3var/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py", line 402, in from_pretrained
    return model_class.from_pretrained(
  File "/mounts/work/sgerstner/.conda/envs/tl3var/lib/python3.10/site-packages/transformers/modeling_utils.py", line 4348, in from_pretrained
    device_map = _get_device_map(model, device_map, max_memory, hf_quantizer)
  File "/mounts/work/sgerstner/.conda/envs/tl3var/lib/python3.10/site-packages/transformers/integrations/accelerate.py", line 353, in _get_device_map
    inferred_max_memory = get_balanced_memory(
  File "/mounts/work/sgerstner/.conda/envs/tl3var/lib/python3.10/site-packages/transformers/integrations/accelerate.py", line 269, in get_balanced_memory
    max_memory = get_max_memory(max_memory)
  File "/mounts/work/sgerstner/.conda/envs/tl3var/lib/python3.10/site-packages/transformers/integrations/accelerate.py", line 206, in get_max_memory
    final_max_memory = accelerate_max_memory(max_memory)
  File "/mounts/work/sgerstner/.conda/envs/tl3var/lib/python3.10/site-packages/accelerate/utils/modeling.py", line 831, in get_max_memory
    max_memory[key] = convert_file_size_to_int(max_memory[key])
  File "/mounts/work/sgerstner/.conda/envs/tl3var/lib/python3.10/site-packages/accelerate/utils/modeling.py", line 149, in convert_file_size_to_int
    raise ValueError(err_msg)
ValueError: `size` auto is not in a valid format. Use an integer for bytes, or a string with an unit (like '5.0GB').

System Info
Describe the characteristic of your environment:

• Describe how transformer_lens was installed (pip, docker, source, ...): installed via pip in a conda environment.
• What OS are you using? (Linux, MacOS, Windows): Linux
• Python version (We support 3.7--3.10 currently): the bug appears in both Python versions I tested: 3.12.9 and 3.10.20.

Additional context
transformer_lens 3.5.1, transformers 5.13.0, accelerate 1.14.0.

The problem seems to be due to line 186 in utilities/multi_gpu.py: accelerate doesn't know what to do with the keyword "auto" that is set here.

Note that I did not try using device_map instead of n_devices, I guess this is the next thing I will check out. But in any case I would wish for n_devices to work as well.

Checklist

• I have checked that there is no similar issue in the repo (required)