In this work, we compress convolutional neural network (CNN) weights post-training via transform quantization. Previous CNN quantization techniques tend to ignore the joint statistics of weights and activations, producing sub-optimal CNN performance at a given quantization bit-rate, or consider their joint statistics during training only and do not facilitate efficient compression of already trained CNN models. We optimally transform (decorrelate) and quantize the weights post-training using a rate-distortion framework to improve compression at any given quantization bit-rate. Transform quantization unifies quantization and dimensionality reduction (decorrelation) techniques in a single framework to facilitate low bit-rate compression of CNNs and efficient inference in the transform domain. We first introduce a theory of rate and distortion for CNN quantization, and pose optimum quantization as a rate-distortion optimization problem. We then show that this problem can be solved using optimal bit-depth allocation following decorrelation by the optimal End-to-end Learned Transform (ELT) we derive in this paper. Experiments demonstrate that transform quantization advances the state of the art in CNN compression in both retrained and non-retrained quantization scenarios. In particular, we find that transform quantization with retraining is able to compress CNN models such as AlexNet, ResNet and DenseNet to very low bit-rates (1-2 bits).
This talk is based on joint published work with Zhe Wang, David Taubman and Bernd Girod. Preprint is available at https://arxiv.org/abs/2009.01174.
The Vancouver Section of the IEEE promotes the exchange of technical information and awareness of the local engineering community by:
Hosting evening, half-day, or full-day technical presentations and seminars on topics of current interest
Organizing tours of local companies, research labs, and industrial sites
Arranging for local support and participation in engineering conferences and symposia held in Vancouver.
Other important activities include:
Recognizing local members by nominating them for advancement in IEEE membership grade and/or IEEE technical and service awards,
Arranging for industry support of projects organized by IEEE student branches and chapters, and,
Organizing events or activities concerning either technology or public policy for the general public.
The Vancouver Section was established on August 22, 1911. The Section encompasses the geographical area of the province of BC, except for Vancouver Island (which is part of the Victoria Section). The Section currently has about 2200 active members, including 325 students at BCIT, SFU, and UBC. It is the fourth largest Section in Canada.
The Section operates on a fiscal year that runs from May - April of each year. The Annual General Meeting and election of new officers are held on the second Monday of each May. Meetings and social events are generally held from September - April.
The Section's nine active technical chapters represent 15 of the 37 technical societies and councils of IEEE and are listed on the menu bar to the left. Each chapter holds 5-6 technical meetings or tours each year. Some chapters organize a few half or full day seminars, and a distinguished lecture or two.
The Section also has active committees responsible for:
GOLD (Graduates of the Last Decade)
IEEE Continuing education courses are offered online at http://www.ieee.org/web/education/home/index.html
Local courses are also offered at the Department of Electrical and Computer Engineering at UBC, the School of Engineering Science at SFU, and at the British Columbia Institute of Technology.
The Section publishes a newsletter, "IEEE CONTACT", 9 times each year (from September - May) that is emailed to members.