Data augmentations have been widely studied to improve the accuracy and robustness of classifiers. However, there exist very few papers that thoroughly investigated the potential of image augmentation in improving GAN models for image synthesis. In this work, we systematically study the effectiveness of various existing augmentation techniques for GAN training in multiple settings. We provide insights and guidelines on how to augment images for both vanilla GANs and GANs with regularizations, improving the fidelity of the generated images substantially. Surprisingly, we find merely augmenting real and generated images for GANs can result in generation quality on par with recent state-of-the-art results. We further compare this with some commonly used regularization methods where augmentations are the essential component. We observe adding regularization on top of augmentation can always improve image quality. We also achieve new state-of-the-art results for conditional generation on CIFAR-10 with consistency loss and contrastive loss as additional regularizations.