Items tagged with “Quantization”.
An end-to-end look at deploying OpenAI GPT OSS models on SageMaker AI and Bedrock AgentCore to power a multi-agent stock analyzer with LangGraph, including 4-bit MXFP4 quantization, serverless orchestration, and scalable inference.
Explores quantization aware training (QAT) and distillation (QAD) as methods to recover accuracy in low-precision models, leveraging NVIDIA's TensorRT Model Optimizer and FP8/NVFP4/MXFP4 formats.
A report on using PyTorch Ahead-of-Time (AoT) compilation to accelerate ZeroGPU Spaces on Nvidia H200 hardware, including setup, quantization options, dynamic shapes, and practical speedups.
Learn how PyTorch AoT compilation speeds up ZeroGPU Spaces on Nvidia H200 GPUs, enabling instant reloads, FP8 quantization, dynamic shapes, and practical speedups (1.3x–1.8x) for Flux, Wan, and LTX models.
Learn how PyTorch AoT compilation speeds up ZeroGPU Spaces by exporting a compiled model once and reloading instantly, with FP8 quantization, dynamic shapes, and careful integration with the Spaces GPU workflow.
NVIDIA outlines a SFT + QAT workflow to recover FP4 accuracy for gpt-oss fine-tuning, comparing MXFP4 and NVFP4, and detailing deployment and performance gains with 98% pass rates on select tasks.
Guide to fine-tuning gpt-oss with SFT + QAT to recover FP4 accuracy while preserving efficiency, including upcasting to BF16, MXFP4, NVFP4, and deployment with TensorRT-LLM.
Jetson Thor delivers edge-scale AI, enabling fast, multi-model generative reasoning with a Blackwell GPU, MIG, FP4/FP8, and 128 GB memory for next-gen robotic platforms.
This work introduces a vector-quantization based approximation to cross-attention for contextual biasing in ASR, enabling scalable, memory-efficient use of large bias catalogs with notable accuracy gains.
OpenAI releases two open-weight, mixture-of-experts models—GPT OSS 120B and GPT OSS 20B—utilizing MXFP4 4-bit quantization. They’re licensed Apache 2.0 with a minimal usage policy and are accessible via Hugging Face Inference Providers for on-device, on-prem, and cloud deployments.
OpenAI releases GPT OSS, a pair of 120B and 20B mixture-of-experts models with MXFP4 4-bit quantization, licensed Apache 2.0, and delivered via Hugging Face Inference Providers for on-device and server deployments.
Two open-weight OpenAI models (gpt-oss-120b and gpt-oss-20b) using mixture-of-experts and MXFP4 quantization for fast, memory-efficient reasoning. Licensed Apache 2.0; designed for private/local deployments and on-device use, with Hugging Face Inference Providers integration.
This article explains how a Hugging Face optimization recipe speeds up LoRA inference on Flux using Diffusers and PEFT, delivering up to ~2.3x speedups and detailing hotswapping, FP8 quantization, and memory-management techniques.