# Quantization API Reference ## torch.ao.quantization This module contains Eager mode quantization APIs. ```{eval-rst} .. currentmodule:: torch.ao.quantization ``` ### Top level APIs ```{eval-rst} .. autosummary:: :toctree: generated :nosignatures: :template: classtemplate.rst quantize quantize_dynamic quantize_qat prepare prepare_qat convert ``` ### Preparing model for quantization ```{eval-rst} .. autosummary:: :toctree: generated :nosignatures: :template: classtemplate.rst fuse_modules.fuse_modules QuantStub DeQuantStub QuantWrapper add_quant_dequant ``` ### Utility functions ```{eval-rst} .. autosummary:: :toctree: generated :nosignatures: :template: classtemplate.rst ObserverOrFakeQuantize swap_module propagate_qconfig_ default_eval_fn ``` ## torch.ao.quantization.utils ```{eval-rst} .. automodule:: torch.ao.quantization.utils .. autosummary:: :toctree: generated :nosignatures: :template: classtemplate.rst activation_is_dynamically_quantized activation_is_int32_quantized activation_is_int8_quantized activation_is_statically_quantized calculate_qmin_qmax check_min_max_valid determine_qparams get_combined_dict get_fqn_to_example_inputs get_qconfig_dtypes get_qparam_dict get_quant_type get_swapped_custom_module_class getattr_from_fqn NodePattern Pattern validate_qmin_qmax ``` ## torch.ao.quantization.quantize_fx This module contains FX graph mode quantization APIs (prototype). ```{eval-rst} .. currentmodule:: torch.ao.quantization.quantize_fx .. autofunction:: attach_preserved_attrs_to_model .. autofunction:: convert_to_reference_fx ``` ```{eval-rst} .. autosummary:: :toctree: generated :nosignatures: :template: classtemplate.rst prepare_fx prepare_qat_fx convert_fx fuse_fx ``` ## torch.ao.quantization.qconfig_mapping This module contains QConfigMapping for configuring FX graph mode quantization. ```{eval-rst} .. currentmodule:: torch.ao.quantization.qconfig_mapping ``` ```{eval-rst} .. autosummary:: :toctree: generated :nosignatures: :template: classtemplate.rst QConfigMapping get_default_qconfig_mapping get_default_qat_qconfig_mapping ``` ## torch.ao.quantization.backend_config This module contains BackendConfig, a config object that defines how quantization is supported in a backend. Currently only used by FX Graph Mode Quantization, but we may extend Eager Mode Quantization to work with this as well. ```{eval-rst} .. currentmodule:: torch.ao.quantization.backend_config .. autosummary:: :toctree: generated :nosignatures: :template: classtemplate.rst BackendConfig BackendPatternConfig DTypeConfig DTypeWithConstraints ObservationType .. currentmodule:: torch.ao.quantization.backend_config.executorch .. autofunction:: get_executorch_backend_config .. currentmodule:: torch.ao.quantization.backend_config.fbgemm .. autofunction:: get_fbgemm_backend_config .. currentmodule:: torch.ao.quantization.backend_config.onednn .. autofunction:: get_onednn_backend_config ``` ## torch.ao.quantization.backend_config.utils ```{eval-rst} .. currentmodule:: torch.ao.quantization.backend_config.utils ``` ```{eval-rst} .. autosummary:: :toctree: generated :nosignatures: :template: classtemplate.rst entry_to_pretty_str get_fused_module_classes get_fuser_method_mapping get_fusion_pattern_to_extra_inputs_getter get_fusion_pattern_to_root_node_getter get_module_to_qat_module get_pattern_to_dtype_configs get_pattern_to_input_type_to_index get_qat_module_classes get_root_module_to_quantized_reference_module pattern_to_human_readable remove_boolean_dispatch_from_name ``` ## torch.ao.quantization.fx.custom_config This module contains a few CustomConfig classes that's used in both eager mode and FX graph mode quantization ```{eval-rst} .. currentmodule:: torch.ao.quantization.fx.custom_config ``` ```{eval-rst} .. autosummary:: :toctree: generated :nosignatures: :template: classtemplate.rst FuseCustomConfig PrepareCustomConfig ConvertCustomConfig StandaloneModuleConfigEntry ``` ## torch.ao.quantization.fx.graph_module ```{eval-rst} .. currentmodule:: torch.ao.quantization.fx.graph_module ``` ```{eval-rst} .. autosummary:: :toctree: generated :nosignatures: :template: classtemplate.rst QuantizedGraphModule ``` ## torch.ao.quantization.fx.utils ```{eval-rst} .. currentmodule:: torch.ao.quantization.fx.utils ``` ```{eval-rst} .. autosummary:: :toctree: generated :nosignatures: :template: classtemplate.rst all_node_args_except_first all_node_args_have_no_tensors assert_and_get_unique_device collect_producer_nodes create_getattr_from_value create_node_from_old_node_preserve_meta get_custom_module_class_keys get_linear_prepack_op_for_dtype get_new_attr_name_with_prefix get_non_observable_arg_indexes_and_types get_qconv_prepack_op get_skipped_module_name_and_classes graph_module_from_producer_nodes maybe_get_next_module node_arg_is_bias node_arg_is_weight NodeInfo return_arg_list ``` ## torch (quantization related functions) This describes the quantization related functions of the `torch` namespace. ```{eval-rst} .. currentmodule:: torch ``` ```{eval-rst} .. autosummary:: :toctree: generated :nosignatures: :template: classtemplate.rst quantize_per_tensor quantize_per_channel dequantize ``` ## torch.Tensor (quantization related methods) Quantized Tensors support a limited subset of data manipulation methods of the regular full-precision tensor. ```{eval-rst} .. currentmodule:: torch.Tensor ``` ```{eval-rst} .. autosummary:: :toctree: generated :nosignatures: view as_strided expand flatten select ne eq ge le gt lt copy_ clone dequantize equal int_repr max mean min q_scale q_zero_point q_per_channel_scales q_per_channel_zero_points q_per_channel_axis resize_ sort topk ``` ## torch.ao.quantization.observer This module contains observers which are used to collect statistics about the values observed during calibration (PTQ) or training (QAT). ```{eval-rst} .. currentmodule:: torch.ao.quantization.observer ``` ```{eval-rst} .. autosummary:: :toctree: generated :nosignatures: :template: classtemplate.rst ObserverBase MinMaxObserver MovingAverageMinMaxObserver PerChannelMinMaxObserver MovingAveragePerChannelMinMaxObserver HistogramObserver PlaceholderObserver RecordingObserver NoopObserver get_observer_state_dict load_observer_state_dict default_affine_fixed_qparams_observer default_observer default_placeholder_observer default_debug_observer default_weight_observer default_histogram_observer default_per_channel_weight_observer default_dynamic_quant_observer default_fixed_qparams_range_0to1_observer default_fixed_qparams_range_neg1to1_observer default_float_qparams_observer default_float_qparams_observer_4bit default_symmetric_fixed_qparams_observer per_channel_weight_observer_range_neg_127_to_127 weight_observer_range_neg_127_to_127 AffineQuantizedObserverBase Granularity MappingType PerAxis PerBlock PerGroup PerRow PerTensor PerToken TorchAODType ZeroPointDomain get_block_size ``` ## torch.ao.quantization.fake_quantize This module implements modules which are used to perform fake quantization during QAT. ```{eval-rst} .. currentmodule:: torch.ao.quantization.fake_quantize ``` ```{eval-rst} .. autosummary:: :toctree: generated :nosignatures: :template: classtemplate.rst FakeQuantizeBase FakeQuantize FixedQParamsFakeQuantize FusedMovingAvgObsFakeQuantize default_affine_fixed_qparams_fake_quant default_dynamic_fake_quant default_embedding_fake_quant default_embedding_fake_quant_4bit default_fake_quant default_fixed_qparams_range_0to1_fake_quant default_fixed_qparams_range_neg1to1_fake_quant default_fused_act_fake_quant default_fused_per_channel_wt_fake_quant default_fused_wt_fake_quant default_histogram_fake_quant default_per_channel_weight_fake_quant default_symmetric_fixed_qparams_fake_quant default_weight_fake_quant disable_fake_quant disable_observer enable_fake_quant enable_observer fused_per_channel_wt_fake_quant_range_neg_127_to_127 fused_wt_fake_quant_range_neg_127_to_127 ``` ## torch.ao.quantization.qconfig This module defines `QConfig` objects which are used to configure quantization settings for individual ops. ```{eval-rst} .. currentmodule:: torch.ao.quantization.qconfig .. autofunction:: get_default_qat_qconfig .. autofunction:: get_default_qconfig .. autofunction:: qconfig_equals ``` ```{eval-rst} .. autosummary:: :toctree: generated :nosignatures: :template: classtemplate.rst QConfig QConfigAny default_qconfig default_debug_qconfig default_per_channel_qconfig default_dynamic_qconfig float16_dynamic_qconfig float16_static_qconfig per_channel_dynamic_qconfig float_qparams_weight_only_qconfig default_qat_qconfig default_weight_only_qconfig default_activation_only_qconfig default_qat_qconfig_v2 ``` ## torch.ao.quantization.quantization_mappings ```{eval-rst} .. automodule:: torch.ao.quantization.quantization_mappings ``` ```{eval-rst} .. currentmodule:: torch.ao.quantization.quantization_mappings ``` ```{eval-rst} .. autosummary:: :toctree: generated :nosignatures: :template: classtemplate.rst get_default_compare_output_module_list get_default_dynamic_quant_module_mappings get_default_dynamic_sparse_quant_module_mappings get_default_float_to_quantized_operator_mappings get_default_qat_module_mappings get_default_qconfig_propagation_list get_default_static_quant_module_mappings get_default_static_quant_reference_module_mappings get_default_static_sparse_quant_module_mappings get_dynamic_quant_module_class get_embedding_qat_module_mappings get_embedding_static_quant_module_mappings get_quantized_operator get_static_quant_module_class no_observer_set ``` ## torch.ao.nn.intrinsic ```{eval-rst} .. automodule:: torch.ao.nn.intrinsic .. automodule:: torch.ao.nn.intrinsic.modules ``` This module implements the combined (fused) modules conv + relu which can then be quantized. ```{eval-rst} .. currentmodule:: torch.ao.nn.intrinsic ``` ```{eval-rst} .. autosummary:: :toctree: generated :nosignatures: :template: classtemplate.rst ConvReLU1d ConvReLU2d ConvReLU3d LinearReLU ConvBn1d ConvBn2d ConvBn3d ConvBnReLU1d ConvBnReLU2d ConvBnReLU3d BNReLU2d BNReLU3d ``` ## torch.ao.nn.intrinsic.qat ```{eval-rst} .. automodule:: torch.ao.nn.intrinsic.qat .. automodule:: torch.ao.nn.intrinsic.qat.modules ``` This module implements the versions of those fused operations needed for quantization aware training. ```{eval-rst} .. currentmodule:: torch.ao.nn.intrinsic.qat ``` ```{eval-rst} .. autosummary:: :toctree: generated :nosignatures: :template: classtemplate.rst LinearReLU ConvBn1d ConvBnReLU1d ConvBn2d ConvBnReLU2d ConvReLU2d ConvBn3d ConvBnReLU3d ConvReLU3d update_bn_stats freeze_bn_stats ``` ## torch.ao.nn.intrinsic.quantized ```{eval-rst} .. automodule:: torch.ao.nn.intrinsic.quantized .. automodule:: torch.ao.nn.intrinsic.quantized.modules ``` This module implements the quantized implementations of fused operations like conv + relu. No BatchNorm variants as it's usually folded into convolution for inference. ```{eval-rst} .. currentmodule:: torch.ao.nn.intrinsic.quantized ``` ```{eval-rst} .. autosummary:: :toctree: generated :nosignatures: :template: classtemplate.rst BNReLU2d BNReLU3d ConvReLU1d ConvReLU2d ConvReLU3d LinearReLU ``` ## torch.ao.nn.intrinsic.quantized.dynamic ```{eval-rst} .. automodule:: torch.ao.nn.intrinsic.quantized.dynamic .. automodule:: torch.ao.nn.intrinsic.quantized.dynamic.modules ``` This module implements the quantized dynamic implementations of fused operations like linear + relu. ```{eval-rst} .. currentmodule:: torch.ao.nn.intrinsic.quantized.dynamic ``` ```{eval-rst} .. autosummary:: :toctree: generated :nosignatures: :template: classtemplate.rst LinearReLU ``` ## torch.ao.nn.qat ```{eval-rst} .. automodule:: torch.ao.nn.qat .. automodule:: torch.ao.nn.qat.modules ``` This module implements versions of the key nn modules **Conv2d()** and **Linear()** which run in FP32 but with rounding applied to simulate the effect of INT8 quantization. ```{eval-rst} .. currentmodule:: torch.ao.nn.qat ``` ```{eval-rst} .. autosummary:: :toctree: generated :nosignatures: :template: classtemplate.rst Conv2d Conv3d Linear ``` ## torch.ao.nn.qat.dynamic ```{eval-rst} .. automodule:: torch.ao.nn.qat.dynamic .. automodule:: torch.ao.nn.qat.dynamic.modules ``` This module implements versions of the key nn modules such as **Linear()** which run in FP32 but with rounding applied to simulate the effect of INT8 quantization and will be dynamically quantized during inference. ```{eval-rst} .. currentmodule:: torch.ao.nn.qat.dynamic ``` ```{eval-rst} .. autosummary:: :toctree: generated :nosignatures: :template: classtemplate.rst Linear ``` ## torch.ao.nn.quantized ```{eval-rst} .. automodule:: torch.ao.nn.quantized :noindex: .. automodule:: torch.ao.nn.quantized.modules ``` This module implements the quantized versions of the nn layers such as `~torch.nn.Conv2d` and `torch.nn.ReLU`. ```{eval-rst} .. currentmodule:: torch.ao.nn.quantized ``` ```{eval-rst} .. autosummary:: :toctree: generated :nosignatures: :template: classtemplate.rst ReLU6 Hardswish ELU LeakyReLU Sigmoid BatchNorm2d BatchNorm3d Conv1d Conv2d Conv3d ConvTranspose1d ConvTranspose2d ConvTranspose3d Embedding EmbeddingBag FloatFunctional FXFloatFunctional QFunctional Linear LayerNorm GroupNorm InstanceNorm1d InstanceNorm2d InstanceNorm3d ``` ## torch.ao.nn.quantized.functional ```{eval-rst} .. automodule:: torch.ao.nn.quantized.functional ``` ```{eval-rst} This module implements the quantized versions of the functional layers such as `~torch.nn.functional.conv2d` and `torch.nn.functional.relu`. Note: :math:`~torch.nn.functional.relu` supports quantized inputs. ``` ```{eval-rst} .. currentmodule:: torch.ao.nn.quantized.functional ``` ```{eval-rst} .. autosummary:: :toctree: generated :nosignatures: :template: classtemplate.rst avg_pool2d avg_pool3d adaptive_avg_pool2d adaptive_avg_pool3d conv1d conv2d conv3d interpolate linear max_pool1d max_pool2d celu leaky_relu hardtanh hardswish threshold elu hardsigmoid clamp upsample upsample_bilinear upsample_nearest ``` ## torch.ao.nn.quantizable This module implements the quantizable versions of some of the nn layers. These modules can be used in conjunction with the custom module mechanism, by providing the ``custom_module_config`` argument to both prepare and convert. ```{eval-rst} .. currentmodule:: torch.ao.nn.quantizable ``` ```{eval-rst} .. autosummary:: :toctree: generated :nosignatures: :template: classtemplate.rst LSTM MultiheadAttention ``` ## torch.ao.nn.quantized.dynamic ```{eval-rst} .. automodule:: torch.ao.nn.quantized.dynamic .. automodule:: torch.ao.nn.quantized.dynamic.modules ``` Dynamically quantized {class}`~torch.nn.Linear`, {class}`~torch.nn.LSTM`, {class}`~torch.nn.LSTMCell`, {class}`~torch.nn.GRUCell`, and {class}`~torch.nn.RNNCell`. ```{eval-rst} .. currentmodule:: torch.ao.nn.quantized.dynamic ``` ```{eval-rst} .. autosummary:: :toctree: generated :nosignatures: :template: classtemplate.rst Linear LSTM GRU RNNCell LSTMCell GRUCell ``` ## Quantized dtypes and quantization schemes Note that operator implementations currently only support per channel quantization for weights of the **conv** and **linear** operators. Furthermore, the input data is mapped linearly to the quantized data and vice versa as follows: ```{eval-rst} .. math:: \begin{aligned} \text{Quantization:}&\\ &Q_\text{out} = \text{clamp}(x_\text{input}/s+z, Q_\text{min}, Q_\text{max})\\ \text{Dequantization:}&\\ &x_\text{out} = (Q_\text{input}-z)*s \end{aligned} ``` ```{eval-rst} where :math:`\text{clamp}(.)` is the same as :func:`~torch.clamp` while the scale :math:`s` and zero point :math:`z` are then computed as described in :class:`~torch.ao.quantization.observer.MinMaxObserver`, specifically: ``` ```{eval-rst} .. math:: \begin{aligned} \text{if Symmetric:}&\\ &s = 2 \max(|x_\text{min}|, x_\text{max}) / \left( Q_\text{max} - Q_\text{min} \right) \\ &z = \begin{cases} 0 & \text{if dtype is qint8} \\ 128 & \text{otherwise} \end{cases}\\ \text{Otherwise:}&\\ &s = \left( x_\text{max} - x_\text{min} \right ) / \left( Q_\text{max} - Q_\text{min} \right ) \\ &z = Q_\text{min} - \text{round}(x_\text{min} / s) \end{aligned} ``` where :math:`[x_\text{min}, x_\text{max}]` denotes the range of the input data while :math:`Q_\text{min}` and :math:`Q_\text{max}` are respectively the minimum and maximum values of the quantized dtype. Note that the choice of :math:`s` and :math:`z` implies that zero is represented with no quantization error whenever zero is within the range of the input data or symmetric quantization is being used. Additional data types and quantization schemes can be implemented through the `custom operator mechanism `_. ```{eval-rst} * :attr:`torch.qscheme` — Type to describe the quantization scheme of a tensor. Supported types: * :attr:`torch.per_tensor_affine` — per tensor, asymmetric * :attr:`torch.per_channel_affine` — per channel, asymmetric * :attr:`torch.per_tensor_symmetric` — per tensor, symmetric * :attr:`torch.per_channel_symmetric` — per channel, symmetric * ``torch.dtype`` — Type to describe the data. Supported types: * :attr:`torch.quint8` — 8-bit unsigned integer * :attr:`torch.qint8` — 8-bit signed integer * :attr:`torch.qint32` — 32-bit signed integer ``` ```{eval-rst} .. These modules are missing docs. Adding them here only for tracking .. automodule:: torch.ao.nn.quantizable.modules :noindex: .. automodule:: torch.ao.nn.quantized.reference :noindex: .. automodule:: torch.ao.nn.quantized.reference.modules :noindex: .. automodule:: torch.nn.quantizable .. automodule:: torch.nn.qat.dynamic.modules .. automodule:: torch.nn.qat.modules .. automodule:: torch.nn.qat .. automodule:: torch.nn.intrinsic.qat.modules .. automodule:: torch.nn.quantized.dynamic .. automodule:: torch.nn.intrinsic .. automodule:: torch.nn.intrinsic.quantized.modules .. automodule:: torch.quantization.fx .. automodule:: torch.nn.intrinsic.quantized.dynamic .. automodule:: torch.nn.qat.dynamic .. automodule:: torch.nn.intrinsic.qat .. automodule:: torch.nn.quantized.modules .. automodule:: torch.nn.intrinsic.quantized .. automodule:: torch.nn.quantizable.modules .. automodule:: torch.nn.quantized .. automodule:: torch.nn.intrinsic.quantized.dynamic.modules .. automodule:: torch.nn.quantized.dynamic.modules .. automodule:: torch.quantization .. currentmodule:: torch.quantization .. autofunction:: default_eval_fn .. automodule:: torch.nn.intrinsic.modules ``` ```{eval-rst} .. toctree:: :hidden: quantization-support.aliases.md ```