Warning: find_unused_parameters=True 해결방법 (DDP programming)

Problem

DDP 코드를 만들어 학습하다가 갑자기 아래 warning이 발생했다.

[W reducer.cpp:1050] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters, consider turning this flag off. Note that this warning may be a false positive your model has flow control causing later iterations to have unused parameters. (function operator())

찾아보니 별로 심각한 문제는 아니었지만 해당 옵션을 False로 해주면 학습할 때 속도가 빨라진다는 말이 있어서 코드를 약간 수정했다.

Solution

방법은 간단하다.

DDPPlugin을 import해서Trainer를 init할때 해당 옵션을 False로 넘겨주면 된다.

trainer = Trainer(
		max_epochs=args.num_epochs, 
		gpus=args.num_gpus, 
		accelerator="ddp",
		plugins=DDPPlugin(find_unused_parameters=False),
		logger = wandb_logger
)

실제 적용된 코드는 깃헙에 업로드 되어있다.

GitHub - wlsdml1114/ddp_test: distributed data parallel test

- Reference

Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters. This flag results