| | |
| | |
Stat |
Members: 3643 Articles: 2'487'895 Articles rated: 2609
28 March 2024 |
|
| | | |
|
Article overview
| |
|
Gradient Scheduling with Global Momentum for Non-IID Data Distributed Asynchronous Training | Chengjie Li
; Ruixuan Li
; Pan Zhou
; Haozhao Wang
; Yuhua Li
; Song Guo
; Keqin Li
; | Date: |
21 Feb 2019 | Abstract: | Distributed asynchronous offline training has received widespread attention
in recent years because of its high performance on large-scale data and complex
models. As data are processed from cloud-centric positions to edge locations, a
big challenge for distributed systems is how to handle native and natural
non-independent and identically distributed (non-IID) data for training.
Previous asynchronous training methods do not have a satisfying performance on
non-IID data because it would result in that the training process fluctuates
greatly which leads to an abnormal convergence. We propose a gradient
scheduling algorithm with global momentum (GSGM) for non-IID data distributed
asynchronous training. Our key idea is to schedule the gradients contributed by
computing nodes based on a white list so that each training node’s update
frequency remains even. Furthermore, our new momentum method can solve the
biased gradient problem. GSGM can make model converge effectively, and maintain
high availability eventually. Experimental results show that for non-IID data
training under the same experimental conditions, GSGM on popular optimization
algorithms can achieve an 20% increase in training stability with a slight
improvement in accuracy on Fashion-Mnist and CIFAR-10 datasets. Meanwhile, when
expanding distributed scale on CIFAR-100 dataset that results in sparse data
distribution, GSGM can perform an 37% improvement on training stability.
Moreover, only GSGM can converge well when the number of computing nodes is 30,
compared to the state-of-the-art distributed asynchronous algorithms. | Source: | arXiv, 1902.7848 | Services: | Forum | Review | PDF | Favorites |
|
|
No review found.
Did you like this article?
Note: answers to reviews or questions about the article must be posted in the forum section.
Authors are not allowed to review their own article. They can use the forum section.
browser claudebot
|
| |
|
|
|
| News, job offers and information for researchers and scientists:
| |