Back To Schedule
Tuesday, October 26 • 1:00pm - 1:15pm
ShiftAddNet: A Hardware-Inspired Deep Network

Sign up or log in to save this to your schedule, view media, leave feedback and see who's attending!

Feedback form is now closed.
Technical Presentations Group 3: Algorithms, Foundations, Visualizations, and Engineering Applications

Multiplication (e.g., convolution) is arguably a cornerstone of modern deep neural networks (DNNs). However, intensive multiplications cause expensive resource costs that challenge DNNs' deployment on resource-constrained edge devices, driving several attempts for multiplication-less deep networks. This paper presented ShiftAddNet, whose main inspiration is drawn from a common practice in energy-efficient hardware implementation, that is, multiplication can be instead performed with additions and logical bit-shifts. We leverage this idea to explicitly parameterize deep networks in this way, yielding a new type of deep network that involves only bit-shift and additive weight layers. This hardware-inspired ShiftAddNet immediately leads to both energy-efficient inference and training, without compromising the expressive capacity compared to standard DNNs. The two complementary operation types (bit-shift and add) additionally enable finer-grained control of the model's learning capacity, leading to more flexible trade-off between accuracy and efficiency, as well as improved robustness to quantization and pruning. We conduct extensive experiments and ablation studies, all backed up by our FPGA-based ShiftAddNet implementation and energy measurements. Compared to existing DNNs or other multiplication-less models, ShiftAddNet aggressively reduces over 80% hardware-quantified energy cost of DNNs training and inference, while offering comparable or better accuracies.

Authors: Haoran You (Rice University), Xiaohan Chen (The University of Texas at Austin), Yongan Zhang (Rice University), Chaojian Li (Rice University), Sicheng Li (Alibaba DAMO Academy), Zihao Liu (Alibaba DAMO Academy), Zhangyang Wang (The University of Texas at Austin), and Yingyan Lin (Rice University)


Haoran You

Rice University

Tuesday October 26, 2021 1:00pm - 1:15pm CDT

Attendees (3)