Embedding-based deep neural networks (DNNs) are widely used in large-scale recommendation systems. Differentially-private stochastic gradient descent (DP-SGD) provides a way to enable personalized experiences while preserving users' privacy by injecting noise into every model parameter during the training process. However, it is challenging to apply DP-SGD to large-scale embedding-based DNNs due to the slow training speed. Large-scale models are usually trained distributively. As the noise needs to be added to all embedding variables, DP introduces a huge communication overhead between the workers and the center parameters servers and thus drastically slows down the training speed. This paper proposes embedding-aware noise addition (ENA) to mitigate the communication overhead, making training a large-scale embedding-based DNN possible. We examine the privacy benefit of ENA both analytically and empirically (with secret sharer). We demonstrated that training with ENA can achieve reasonable model precision while providing good practical privacy protection indicated by the secret sharer tests. Experiments on a real-world, large-scale dataset and model show that ENA is much faster than the standard DP, improving the training speed by 100X and unblocking the training of a large-scale embedding-based DNN with reduced privacy risk.