Training Arguments resets accelerator

[lcxrocks]

System Info

Related package versions:
transformers=4.51.3
accelerate==1.3.0

Who can help?

No response

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

Here is a minimal reproducible example:

from accelerate import Accelerator
accelerator = Accelerator()
print("(L3) AcceleratorState has distributed_type:", hasattr(accelerator.state, 'distributed_type'))
print("(L4) distributed_type value:", getattr(accelerator.state, 'distributed_type', 'NOT_FOUND'))
from transformers import TrainingArguments
training_args = TrainingArguments()
print("(L7) AcceleratorState has distributed_type:", hasattr(accelerator.state, 'distributed_type'))
print("(L8) distributed_type value:", getattr(accelerator.state, 'distributed_type', 'NOT_FOUND'))

And this returns:

(L3) AcceleratorState has distributed_type: True
(L4) distributed_type value: DistributedType.MULTI_GPU
...
(L7) AcceleratorState has distributed_type: False
(L8) distributed_type value: NOT_FOUND

This behavior of silently resetting the accelerator can be frustrating for people who have already defined it in an outer scope. I’m wondering how this could be resolved, and I’d be glad to help work on a fix.

Expected behavior

I personally suppose that initializing a dictionary parameter in TrainingArguments should not alter the global behavior of accelerator. Therefore, the snippet above should return True in both cases.

[ArkVex]

@lcxrocks I agree, the reset is a bit unintuitive. A better approach might be for TrainingArguments to reuse an existing Accelerator if one is already initialized, and only create a new one when none exists. Another option could be adding an explicit flag to control this behavior so it’s clearer to users. I’d be glad to help work on a fix in that direction.

[PrathmeshAdsod]

We have to be careful for hyper-parameter tuning. Suppose while creating accelerator user assigned some parameters which we are going to reuse (the state) but during hyper-parameter it gave different types of parameters then it can cause issues. So we have to be careful and give proper messages to the user. Consider these snippets -

# Custom accelerator with FP16
accelerator = Accelerator(mixed_precision="fp16")

# Hyperparameter tuning wants to test different precisions
def objective(trial):
    mixed_precision = trial.suggest_categorical('mixed_precision', ['fp16', 'bf16', None])
    training_args = TrainingArguments(
        fp16=(mixed_precision == "fp16"),
        bf16=(mixed_precision == "bf16")
    )
    # CONFLICT: Accelerator says FP16, TrainingArguments might say BF16!

accelerator = Accelerator(gradient_accumulation_steps=4)

def objective(trial):
    grad_accum = trial.suggest_int('gradient_accumulation_steps', 2, 16)
    training_args = TrainingArguments(gradient_accumulation_steps=grad_accum)
    # TWO different gradient accumulation settings!

For this we can give messages to user to make them aware of this issue.

If we want to solve this internally then one solution can be to keep user other configurations in Accelerator 
but prioritize and replace the parameters of Accelerator with required parameters of tuning 

If you guys not started working then I would like to fix these things...