Skip to content

Default use one thread in fluid#7208

Merged
reyoung merged 1 commit intoPaddlePaddle:developfrom
reyoung:feature/default_omp_num_threads_1
Jan 4, 2018
Merged

Default use one thread in fluid#7208
reyoung merged 1 commit intoPaddlePaddle:developfrom
reyoung:feature/default_omp_num_threads_1

Conversation

@reyoung
Copy link
Collaborator

@reyoung reyoung commented Jan 4, 2018

No description provided.

tonyyang-svail
tonyyang-svail approved these changes Jan 4, 2018
View reviewed changes
Copy link

@tonyyang-svail tonyyang-svail left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Excellent!
luotao1
luotao1 reviewed Jan 4, 2018
View reviewed changes
python/paddle/v2/fluid/__init__.py
'speed will not be optimized if you use data parallel. It will '
'fail if this PaddlePaddle binary is compiled with OpenBlas since'
' OpenBlas does not support multi-threads.'.format(num_threads),
file=sys.stderr)
Copy link
Contributor

@luotao1 luotao1 Jan 4, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这段注释没太看明白。openblas的时候可以用OPENBLAS_NUM_THREADS来设置，不知道和这个PR有没有关系。
Copy link
Collaborator Author

@reyoung reyoung Jan 4, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这个PR非常简单，就是要把OMP_NUM_THREADS默认设置成1。因为确实在数据并行��，这个环境变量设置成其他的值并不合适。
Copy link
Contributor

@luotao1 luotao1 Jan 5, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

当cpu核数大于trainer_count时，如果设成1，会导致cpu核数没用满。
OMP_NUM_THREADS = cpu核数/trainer_count

Paddle/python/paddle/v2/__init__.py

Lines 112 to 115 in 7508d52

threads = num_processors / trainer_count
threads = '1' if threads < 1 else str(threads)
set_env("OMP_NUM_THREADS", threads)
set_env("MKL_NUM_THREADS", threads)

@qingqing01
Copy link
Contributor

qingqing01 commented Jan 4, 2018

Could you add an issue to describe the problem?
@reyoung reyoung merged commit 2b259bf into PaddlePaddle:develop Jan 4, 2018
@tonyyang-svail tonyyang-svail mentioned this pull request Jan 5, 2018
Closed
@tensor-tang
Copy link
Contributor

tensor-tang commented Jan 5, 2018 •
edited
Loading

Two concern I should point out :

  1. OMP_NUM_THREADS will also impact MKL, not only OpenBLAS mentioned in OpenBLAS doesn't support multi thread application #7234.

  2. Force setting it to 1 will limit MKLML and MKLDNN multi-processing, which means use only 1 core to compute. I suggest it could be calculated by settings like threads_pre_trainer=available_threads/trainer_count.
@tonyyang-svail
Copy link

tonyyang-svail commented Jan 5, 2018

I see. Change OPENBLAS_NUM_THREADS is a better approach.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

5 participants

@reyoung @qingqing01 @tensor-tang @tonyyang-svail @luotao1