Jump to Content
Chun-Ta Lu

Chun-Ta Lu

Chun-Ta Lu is a Software Engineer at Google Research. Prior to joining Google, he received his PhD from University of Illinois at Chicago. His main research interests span various fields of Computer Vision, Data Mining, Machine Learning.
Authored Publications
Google Publications
Other Publications
Sort By
  • Title
  • Title, descending
  • Year
  • Year, descending
    Scaling Up LLM Reviews for Google Ads Content Moderation
    Ariel Fuxman
    Chih-Chun Chia
    Dongjin Kwon
    Enming Luo
    Mehmet Tek
    Ranjay Krishna
    Tiantian Fang
    Tushar Dogra
    Yu-Han Lyu
    (2024)
    Preview abstract Large language models (LLMs) are powerful tools for content moderation but LLM inference costs and latency on large volumes of data, such as the Google Ads repository, are prohibitive for their casual usage. This study is focused on scaling up LLM reviews for content moderation in Google Ads. First, we use heuristics to select candidates via filtering and duplicate removal, and create clusters of ads for which we select one representative ad per cluster. Then, LLMs are used to review only the representative ads. Finally we propagate the LLM decisions for representative ads back to their clusters. This method reduces the number of reviews by more than 3 orders of magnitude while achieving a 2x recall compared to a non-LLM model as a baseline. Note that, the success of this approach is a strong function of the representations used in clustering and label propagation; we observed that cross-modal similarity representations yield better results than uni-modal representations. View details
    Visual Program Tuning: Training Large Multimodal Models to Reason like Programs
    Yushi Hu
    Krishna Viswanathan
    Enming Luo
    Ranjay Krishna
    Ariel Fuxman
    Conference on Computer Vision and Pattern Recognition (2024)
    Preview abstract Solving complex visual tasks (e.g., “Who invented the musical instrument on the right?”) involves back-and-forth between visual processing and reasoning. Visual programming is a recent multimodal framework that has shown promise in conducting visual reasoning in an interpretable and compositional manner. However, this framework is error-prone—it can lead to a wrong answer whenever the program itself is wrong, or when any of the steps of the program are solved incorrectly, thus leading to worse overall performance than end-to-end systems trained with labeled data. Moreover, it is inefficient to involve multiple steps (i.e., generating and then running programs) during inference. Ideally, a single large multimodal model (LMM) should directly conduct similar reasoning and yield the correct answer. In this work, we propose Visual Program Tuning (VPT), which leverages visual programs for teaching LLMs to reason via instruction tuning. VPT rewrites the execution traces of visual programs as chain-of-thought reasoning steps, and tunes an LMM to output not only the label but its reasoning as well. Extensive experiments on complex vision tasks show that models trained with VPT achieve state-of-the-art accuracy while being able to produce interpretable and faithful reasoning steps. PaLI-X + VPT outperforms all existing LMMs on a wide range of visual tasks, improving performance on counting, spatial relations, and compositional reasoning tasks. VPT is also helpful for quick adaptation on new tasks. Our experiments on content moderation show that fine-tuning LMMs with program-augmented examples is more sample efficient than traditional supervised training. View details
    Benchmarking Robustness to Adversarial Image Obfuscations
    Florian Stimberg
    Yintao Liu
    Merve Kaya
    Cyrus Rashtchian
    Ariel Fuxman
    Mehmet Tek
    Advances in Neural Information Processing Systems (2023)
    Preview abstract Automated content filtering and moderation is an important tool that allows online platforms to build striving user communities that facilitate cooperation and prevent abuse. Unfortunately, resourceful actors try to bypass automated filters in a bid to post content that violate platform policies and codes of conduct. To reach this goal, these malicious actors obfuscate policy violating content to prevent machine learning models from reaching the correct decision. In this paper, we invite researchers to tackle this specific issue and present a new image benchmark. This benchmark, based on ImageNet, simulates the type of obfuscations created by malicious actors. It goes beyond ImageNet-C and ImageNet-C-Bar by proposing general, drastic, adversarial modifications that preserve the original content intent. It aims to tackle a more common adversarial threat than the one considered by Lp-norm bounded adversaries. Our hope is that this benchmark will encourage researchers to test their models and methods and try to find new approaches that are more robust to these obfuscations. View details
    Graph-RISE: Graph-Regularized Image Semantic Embedding
    Aleksei Timofeev
    Futang Peng
    Krishnamurthy Viswanathan
    Lucy Gao
    Sujith Ravi
    Yi-ting Chen
    Zhen Li
    The 12th International Conference on Web Search and Data Mining (2020) (to appear)
    Preview abstract Learning image representation to capture instance-based semantics has been a challenging and important task for enabling many applications such as image search and clustering. In this paper, we explore the limits of image embedding learning at unprecedented scale and granularity. We present Graph-RISE, an image embedding that captures very fine-grained, instance-level semantics. Graph-RISE is learned via a large-scale, neural graph learning framework that leverages graph structure to regularize the training of deep neural networks. To the best of our knowledge, this is the first work that can capture instance-level image semantics at million—O(40M)—scale. Experimental results show that Graph-RISE outperforms state-of-the-art image embedding algorithms on several evaluation tasks, including image classification and triplet ranking. We also provide case studies to demonstrate that, qualitatively, image retrieval based on Graph-RISE well captures the semantics and differentiates nuances at instance level. View details
    Preview abstract We present Neural Structured Learning (NSL) in TensorFlow, a new learning paradigm to train neural networks by leveraging structured signals in addition to feature inputs. Structure can be explicit as represented by a graph, or implicit, either induced by adversarial perturbation or inferred using techniques like embedding learning. NSL is open-sourced as part of the TensorFlow ecosystem and is widely used in Google across many products and services. In this tutorial, we provide an overview of the NSL framework including various libraries, tools, and APIs as well as demonstrate the practical use of NSL in different applications. The NSL website is hosted at www.tensorflow.org/neural_structured_learning, which includes details about the theoretical foundations of the technology, extensive API documentation, and hands-on tutorials. View details
    Inferring Context from Pixels for Multimodal Image Classification
    Manan Shah
    Krishnamurthy Viswanathan
    Ariel Fuxman
    Zhen Li
    Aleksei Timofeev
    Proceedings of the 28th ACM International Conference on Information and Knowledge Management, ACM (2019) (to appear)
    Preview abstract Image classification models take image pixels as input and predict labels in a predefined taxonomy. While contextual information (e.g. text surrounding an image) can provide valuable orthogonal signals to improve classification, the typical setting in literature assumes the unavailability of text and thus focuses on models that rely purely on pixels. In this work, we also focus on the setting where only pixels are available in the input. However, we demonstrate that if we predict textual information from pixels, we can subsequently use the predicted text to train models that improve overall performance. We propose a framework that consists of two main components: (1) a phrase generator that maps image pixels to a contextual phrase, and (2) a multimodal model that uses textual features from the phrase generator and visual features from the image pixels to produce labels in the output taxonomy. The phrase generator is trained using web-based query-image pairs to incorporate contextual information associated with each image and has a large output space. We evaluate our framework on diverse benchmark datasets (specifically, the WebVision dataset for evaluating multi-class classification and OpenImages dataset for evaluating multi-label classification), demonstrating performance improvements over approaches based exclusively on pixels and showcasing benefits in prediction interpretability. We additionally present results to demonstrate that our framework provides improvements in few-shot learning of minimally labeled concepts. We further demonstrate the unique benefits of the multimodal nature of our framework by utilizing intermediate image/text co-embeddings to perform baseline zero-shot learning on the ImageNet dataset. View details
    Learning from Multi-View Multi-Way Data via Structural Factorization Machines
    Lifang He
    Hao Ding
    Bokai Cao
    Philip S. Yu
    Proceedings of the 2018 World Wide Web Conference, International World Wide Web Conferences Steering Committee, Lyon, France, pp. 1593-1602
    Spectral Collaborative Filtering
    Lei Zheng
    Fei Jiang
    Jiawei Zhang
    Philip S. Yu
    Proceedings of the 12th ACM Conference on Recommender Systems, ACM, Vancouver, British Columbia (2018), pp. 311-319
    Multilinear Factorization Machines for Multi-Task Multi-View Learning
    Lifang He
    Weixiang Shao
    Bokai Cao
    Philip S. Yu
    Proceedings of the Tenth ACM International Conference on Web Search and Data Mining, ACM, Cambridge, United Kingdom (2017), pp. 701-709
    Kernelized Support Tensor Machines
    Lifang He
    Guixiang Ma
    Shen Wang
    Linlin Shen
    Philip S. Yu
    Ann B. Ragin.
    Proceedings of the 34th International Conference on Machine Learning, PMLR, Sydney, Australia (2017), pp. 1442-1451
    Multi-way Multi-level Kernel Modeling for Neuroimaging Classification
    Lifang He
    Hao Ding
    Shen Wang
    Linlin Shen
    Philip S. Yu
    Ann B. Ragin
    The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, Honolulu, HI, USA (2017), pp. 6846-6854
    Online Unsupervised Multi-view Feature Selection
    Weixiang Shao
    Lifang He
    Xiaokai Wei
    Philip S. Yu
    2016 IEEE 16th International Conference on Data Mining (ICDM), IEEE, Barcelona, Spain, pp. 1203-1208
    Joint Community and Structural Hole Spanner Detection via Harmonic Modularity
    Lifang He
    Jiaqi Ma
    Jianping Cao
    Linlin Shen
    Philip S. Yu
    Proceedings of the 22Nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, ACM, San Francisco, California, USA (2016), pp. 875-884
    Item Recommendation for Emerging Online Businesses
    Sihong Xie
    Weixiang Shao
    Lifang He
    Philip S. Yu
    Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence, AAAI Press, New York, New York, USA (2016), pp. 3797-3803
    Inferring the impacts of social media on crowdfunding
    Sihong Xie
    Xiangnan Kong
    Philip S Yu
    Proceedings of the 7th ACM International Conference on Web Search and Data Mining, ACM, New York, New York, USA (2014), pp. 573-582
    Identifying your customers in social networks
    Hong-Han Shuai
    Philip S. Yu
    Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management, ACM, Shanghai, China (2014), pp. 391-400