publications

2024

1. ASAP 2024

   ISP2DLA: Automated Deep Learning Accelerator Design for On-Sensor Image Signal Processing

   Dong-eon Won*, Yeeun Kim*, Janghwan Lee, Minjae Lee, Jonghyun Bae, Jongjoo Park, Jeongyong Song, and Jungwook Choi

   In 35th IEEE International Conference on Application-specific Systems, Architectures and Processors (ASAP, Poster), 2024

   Abs PDF

   Deep neural network-based image signal processing (ISP-DNN) improves image quality with techniques such as demosaicing, but these models pose substantial computational and memory challenges when implemented on CMOS image sensors, particularly due to the high-resolution inputs that increase memory requirements for activations. Layer fusion reduces memory usage by combining consecutive processing steps, yet it increases computational demands, a critical issue in resource-limited on-sensor environments. To address these challenges, we introduce ISP2DLA, an automated deep learning accelerator design framework that balances computational and memory demands for on-sensor ISP. This framework optimizes hardware designs by adjusting line buffer sizes and the number of MAC units, reducing gate counts by 14-79% across two ISP-DNN models, thus enabling efficient on-sensor ISP model inference within constrained resources.

2. ACL 2024

   Improving Conversational Abilities of Quantized Large Language Models via Direct Preference Alignment

   Janghwan Lee*, Seongmin Park*, Sukjin Hong, Minsoo Kim, Du-Seong Chang, and  Jungwook Choi

   In The 62nd Annual Meeting of the Association for Computational Linguistics (ACL, Oral), 2024

   Abs PDF

   The rapid advancement of large language models (LLMs) has facilitated their transformation into conversational chatbots that can grasp contextual nuances and generate pertinent sentences, closely mirroring human values through advanced techniques such as instruction tuning and reinforcement learning from human feedback (RLHF). However, the computational efficiency required for LLMs, achieved through techniques like post-training quantization (PTQ), presents challenges such as token-flipping that can impair chatbot performance. In response, we propose a novel preference alignment approach, quantization-aware direct preference optimization (QDPO), that aligns quantized LLMs with their full-precision counterparts, improving conversational abilities. Evaluated on two instruction-tuned LLMs in various languages, QDPO demonstrated superior performance in improving conversational abilities compared to established PTQ and knowledge-distillation fine-tuning techniques, marking a significant step forward in the development of efficient and effective conversational LLMs.

3. ICEIC 2024

   Searching Optimal Floating-Point Format for Sub-8-Bit Large Language Model Inference

   Youngdeok Hwang*, Janghwan Lee*, Jiwoong Park, Jieun Lim, and Jungwook Choi

   In International Conference on Electronics, Information, and Communication (ICEIC, Oral), 2024

   Abs PDF

   Large Language Models (LLMs) have shown re- markable success in various natural language processing tasks. However, their extensive parameter count leads to significant memory and computational demands. To tackle these challenges, there is growing interest in employing post-training quantization (PTQ) with reduced-precision floating-point (FP) operations. Yet, the optimal FP configuration remains a topic of debate. Existing studies often overlook a thorough analysis of the diverse data distributions found in LLMs and the crucial design choice, de- normal. In this paper, we conduct a comprehensive examination of the various data distributions within LLMs and the significance of denormal representation, presenting a mixed-format floating- point framework. Our proposed framework allows for sub-8- bit inference with minimal performance degradation in language modeling and reasoning tasks across a broad spectrum of LLMs.

4. HPCA 2024

   SPADE: Sparse Pillar-based 3D Object Detection Accelerator for Autonomous Driving

   Minjae Lee, Seongmin Park, Hyungmin Kim, Minyong Yoon, Janghwan Lee, Junwon Choi, Nam Sung Kim, Mingu Kang, and Jungwook Choi

   In 30th IEEE International Symposium on High-Performance Computer Architecture (HPCA), 2024

   Abs PDF

   3D object detection using point cloud (PC) data is essential for perception pipelines of autonomous driving, where efficient encoding is key to meeting stringent resource and latency requirements. PointPillars, a widely adopted bird’s-eye view (BEV) encoding, aggregates 3D point cloud data into 2D pillars for fast and accurate 3D object detection. However, the state- of-the-art methods employing PointPillars overlook the inherent sparsity of pillar encoding where only a valid pillar is encoded with a vector of channel elements, missing opportunities for significant computational reduction. Meanwhile, current sparse convolution accelerators are designed to handle only element- wise activation sparsity and do not effectively address the vector sparsity imposed by pillar encoding. In this paper, we propose SPADE, an algorithm-hardware co- design strategy to maximize vector sparsity in pillar-based 3D object detection and accelerate vector-sparse convolution com- mensurate with the improved sparsity. SPADE consists of three components: (1) a dynamic vector pruning algorithm balancing ac- curacy and computation savings from vector sparsity, (2) a sparse coordinate management hardware transforming 2D systolic array into a vector-sparse convolution accelerator, and (3) sparsity- aware dataflow optimization tailoring sparse convolution schedules for hardware efficiency. Taped-out with a commercial technology, SPADE saves the amount of computation by 36.3–89.2% for representative 3D object detection networks and benchmarks, leading to 1.3–10.9× speedup and 1.5–12.6× energy savings compared to the ideal dense accelerator design. These sparsity- proportional performance gains equate to 4.1–28.8× speedup and 90.2–372.3× energy savings compared to the counterpart server and edge platforms.

2023

1. EMNLP 2023

   Enhancing Computation Efficiency in Large Language Models through Weight and Activation Quantization

   Janghwan Lee*, Minsoo Kim*, Seungcheol Baek, Seokjoong Hwang, Wonyong Sung, and Jungwook Choi

   In The 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2023

   Abs PDF

   Large Language Models (LLMs) are proficient in natural language processing tasks, but their deployment is often restricted by extensive parameter sizes and computational demands. This paper focuses on post-training quantization (PTQ) in LLMs, specifically 4-bit weight and 8-bit activation (W4A8) quantization, to enhance computational efficiency—a topic less explored compared to weight-only quantization. We present two innovative techniques: activation-quantization-aware scaling (AQAS) and sequence-length-aware calibration (SLAC) to enhance PTQ by considering the combined effects on weights and activations and aligning calibration sequence lengths to target tasks. Moreover, we introduce dINT, a hybrid data format combining integer and denormal representations, to address the underflow issue in W4A8 quantization, where small values are rounded to zero. Through rigorous evaluations of LLMs, including OPT and LLaMA, we demonstrate that our techniques significantly boost task accuracies to levels comparable with full-precision models. By developing arithmetic units compatible with dINT, we further confirm that our methods yield a 2× hardware efficiency improvement compared to 8-bit integer MAC unit.

2. NeurIPS 2023

   Token-Scaled Logit Distillation for Ternary Weight Generative Language Models

   Minsoo Kim, Sihwa Lee, Janghwan Lee, Sukjin Hong, Duseong Chang, Wonyong Sung, and Jungwook Choi

   In Thirty-seventh Conference on Neural Information Processing Systems (NeurIPS), 2023

   Abs PDF

   Generative Language Models (GLMs) have shown impressive performance in tasks such as text generation, understanding, and reasoning. However, the large model size poses challenges for practical deployment. To solve this problem, Quantization-Aware Training (QAT) has become increasingly popular. However, current QAT methods for generative models have resulted in a noticeable loss of accuracy. To counteract this issue, we propose a novel knowledge distillation method specifically designed for GLMs. Our method, called token-scaled logit distillation, prevents overfitting and provides superior learning from the teacher model and ground truth. This research marks the first evaluation of ternary weight quantization-aware training of large-scale GLMs with less than 1.0 degradation in perplexity and achieves enhanced accuracy in tasks like common-sense QA and arithmetic reasoning as well as natural language understanding.

3. arXiv 2023

   PillarAcc: Sparse PointPillars Accelerator for Real-Time Point Cloud 3D Object Detection on Edge Devices

   Minjae Lee, Hyungmin Kim, Seongmin Park, Minyong Yoon, Janghwan Lee, Junwon Choi, Mingu Kang, and Jungwook Choi

   2023

   Abs PDF

   3D object detection using point cloud (PC) data is vital for autonomous driving perception pipelines, where efficient encoding is key to meeting stringent resource and latency requirements. PointPillars, a widely adopted bird’s-eye view (BEV) encoding, aggregates 3D point cloud data into 2D pillars for high-accuracy 3D object detection. However, most state-of-the-art methods employing PointPillar overlook the inherent sparsity of pillar encoding, missing opportunities for significant computational reduction. In this study, we propose a groundbreaking algorithm-hardware co-design that accelerates sparse convolution processing and maximizes sparsity utilization in pillar-based 3D object detection networks. We investigate sparsification opportunities using an advanced pillar-pruning method, achieving an optimal balance between accuracy and sparsity. We introduce PillarAcc, a state-of-the-art sparsity support mechanism that enhances sparse pillar convolution through linear complexity input-output mapping generation and conflict-free gather-scatter memory access. Additionally, we propose dataflow optimization techniques, dynamically adjusting the pillar processing schedule for optimal hardware utilization under diverse sparsity operations. We evaluate PillarAcc on various cutting-edge 3D object detection networks and benchmarks, achieving remarkable speedup and energy savings compared to representative edge platforms, demonstrating record-breaking PointPillars speed of 500FPS with minimal compromise in accuracy.

4. DAC 2023

   Range-Invariant Approximation of Non-Linear Operations for Efficient BERT Fine-Tuning

   Janghyeon Kim, Janghwan Lee, JeongHo Han, Sangheon Lee, and Jungwook Choi

   In 2023 60th ACM/IEEE Design Automation Conference (DAC), 2023

   Abs PDF

   This paper proposes a range-invariant approximation of non-linear operations for training computations of Transformer-based large language models. The proposed method decomposes the approximation into the scaling and the range-invariant resolution for LUT approximation, covering diverse data ranges of non-linear operations with drastically reduced LUT entries during task-dependent BERT fine-tuning. We demonstrate that the proposed method robustly approximates all the non-linear operations of BERT without score degradation on challenging GLUE benchmarks using only a single-entry LUT, facilitating 52% area savings in hardware implementation.

5. ICASSP 2023

   Finding Optimal Numerical Format for Sub-8-Bit Post-Training Quantization of Vision Transformers

   Janghwan Lee, Youngdeok Hwang, and Jungwook Choi

   In 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2023

   Abs PDF

   Vision Transformers (ViTs) have gained significant attention for their exceptional model accuracies on computer vision applications, but their demanding memory requirements and computational complexity have hindered active deployment. Post-training quantization (PTQ) is a practical method to tackle this challenge by directly reducing ViT’s bit-precision. However, diverse data characteristics across different operations of ViT cannot be well captured solely by a single numerical format (fixed or floating-point). This work proposes an analytical framework that optimizes the numerical format of each matrix multiplication of ViTs for mixed-format sub-8bit quantization. The extensive evaluation demonstrates that the proposed method can reduce the PTQ error and achieve state-of-the-art accuracy for popular ViT models.

2022

1. AICAS 2022

   Optimizing Exponent Bias for Sub-8bit Floating-Point Inference of Fine-tuned Transformers

   Janghwan Lee, and Jungwook Choi

   In 2022 IEEE 4th International Conference on Artificial Intelligence Circuits and Systems (AICAS, Oral), 2022

   Abs PDF

   The Transformer-based fine-tuned neural networks have demonstrated remarkable success in natural language processing (NLP) at the cost of a substantial computational burden. Post-training quantization (PTQ) is a promising technique to reduce the computational cost without expensive re-training. But prior works either demand complex calibration or suffer noticeable accuracy degradation. This paper proposes a practical method for sub-8bit floating-point (FP) PTQ. The proposed method optimizes the exponent bias to minimize quantization error in terms of signal-to-quantization noise ratio (SQNR) progressively like stochastic gradient descent. We evaluate that the proposed method achieves close to full-precision model accuracy for 6 to 8 bit FP PTQ of fine-tuned BERT on GLUE and SQuAD tasks with negligible run-time overhead.