Items tagged with “Inference”.
NVIDIA HGX B200 lowers embodied carbon intensity by 24% vs. HGX H100, while delivering higher AI performance and energy efficiency. This article reviews the PCF-backed improvements, new hardware features, and implications for developers and enterprises.
Scaleway is now a supported Inference Provider on the Hugging Face Hub, enabling serverless inference directly on model pages with JS and Python SDKs. Access popular open-weight models and enjoy scalable, low-latency AI workflows.
NVIDIA Dynamo offloads KV Cache from GPU memory to cost-efficient storage, enabling longer context windows, higher concurrency, and lower inference costs for large-scale LLMs and generative AI workloads.
Learn how to monitor and optimize Amazon Bedrock batch inference jobs with CloudWatch metrics, alarms, and dashboards to improve performance, cost efficiency, and operational oversight.
A detailed look at speculative decoding for AI inference, including draft-target and EAGLE-3 methods, how they reduce latency, and how to deploy on NVIDIA GPUs with TensorRT.
An end-to-end look at deploying OpenAI GPT OSS models on SageMaker AI and Bedrock AgentCore to power a multi-agent stock analyzer with LangGraph, including 4-bit MXFP4 quantization, serverless orchestration, and scalable inference.
A detailed look at how NVIDIA Run:ai Model Streamer lowers cold-start times for LLM inference by streaming weights into GPU memory, with benchmarks across GP3, IO2, and S3 storage.
Open-source Qwen3-Next 80B-A3B-Thinking and 80B-A3B-Instruct preview a hybrid MoE architecture designed for long context and efficient inference, with NVIDIA-backed deployment options and open tooling.
This post explains how msg automated data harmonization for msg.ProfileMap using Amazon Bedrock to power LLM-driven data enrichment, boosting HR concept matching accuracy, reducing manual workload, and aligning with EU AI Act and GDPR.
A structured guide to migrating from Anthropic Claude 3.5 Sonnet to Claude 4 Sonnet on Amazon Bedrock, covering model differences, access, APIs, extended thinking, evaluation, and rollout best practices.
Explores quantization aware training (QAT) and distillation (QAD) as methods to recover accuracy in low-precision models, leveraging NVIDIA's TensorRT Model Optimizer and FP8/NVFP4/MXFP4 formats.
NVIDIA’s RTX PRO 6000 Blackwell Server Edition dramatically speeds protein-structure inference, enabling end-to-end GPU-resident workflows with OpenFold and TensorRT—achieving up to 138x faster folding than AlphaFold2.
NVIDIA NIM Operator 3.0.0 expands scalable AI inference on Kubernetes, enabling multi-LLM and multi-node deployments, KServe integration, and DRA support in technology preview, with Red Hat collaboration and NeMo Guardrails.
LSEG uses Amazon Bedrock and Anthropic's Claude to automate market surveillance, analyzing 250,000 RNS articles from 2023 to assess price sensitivity and reduce false positives, boosting efficiency and accuracy.
Explores how NVIDIA Rivermax and NEIO FastSocket deliver ultra-low latency, high-throughput networking for financial services, leveraging kernel bypass, zero-copy, and GPUDirect to move data directly from NICs to GPUs.
AWS announces the Falcon-H1 instruction-tuned models from TII (0.5B–34B) on Amazon Bedrock Marketplace and SageMaker JumpStart, with multilingual support, a hybrid architecture, and deployment guidance.
NVIDIA Spectrum-XGS Ethernet enables scale-across networking to unify dispersed data centers into a single AI factory, delivering significant NCCL bandwidth gains and predictable performance over long distances.
A detailed look at using the HyperPod CLI and SDK to streamline training and deploying large AI models on SageMaker HyperPod, with practical steps for FSDP-based training and inference deployment.
A report on using PyTorch Ahead-of-Time (AoT) compilation to accelerate ZeroGPU Spaces on Nvidia H200 hardware, including setup, quantization options, dynamic shapes, and practical speedups.
SageMaker HyperPod introduces a one-click, validated cluster creation experience that automatically provisions prerequisite resources and prescriptive defaults, accelerating distributed training and inference with Slurm or EKS orchestration.
A scalable, serverless workflow for Bedrock batch inference orchestrated by AWS Step Functions. Demonstrated on a 2.2 million-row dataset with preprocessing, parallel execution, and postprocessing.
A new GPU memory swap (model hot-swapping) approach lets multiple models share GPUs beyond capacity, reducing costs while preserving responsiveness for large-scale LLM inference.
Learn how PyTorch AoT compilation speeds up ZeroGPU Spaces on Nvidia H200 GPUs, enabling instant reloads, FP8 quantization, dynamic shapes, and practical speedups (1.3x–1.8x) for Flux, Wan, and LTX models.
Leverage GPU memory swap (model hot-swapping) to share GPUs across multiple LLMs, reduce idle GPU costs, and improve autoscaling while meeting SLAs.
NVIDIA outlines a SFT + QAT workflow to recover FP4 accuracy for gpt-oss fine-tuning, comparing MXFP4 and NVFP4, and detailing deployment and performance gains with 98% pass rates on select tasks.
SageMaker HyperPod now offers managed node automatic scaling powered by Karpenter, enabling responsive, cost-aware scaling for inference and training workloads on HyperPod clusters.
Explains why small language models (SLMs) enable scalable, cost-efficient agentic AI, the role of heterogenous model ecosystems, and practical paths to adoption with NVIDIA NeMo and Nemotron Nano 2.
Guide to fine-tuning gpt-oss with SFT + QAT to recover FP4 accuracy while preserving efficiency, including upcasting to BF16, MXFP4, NVFP4, and deployment with TensorRT-LLM.
AWS announces Mercury and Mercury Coder foundation models from Inception Labs are now accessible via Amazon Bedrock Marketplace and SageMaker JumpStart, enabling ultra-fast diffusion-based generation for AWS generative AI workloads.
NVIDIA extends NVFP4 from inference to pretraining, enabling 4-bit precision to accelerate large-scale transformer pretraining while preserving FP8/BF16-like accuracy. Early results on a 12B model show stable convergence and strong downstream performance, with up to 7x GEMM speedups on Blackwell Ult
NVFP4 is a 4-bit data format delivering FP16-level accuracy with the throughput and memory efficiency of 4-bit precision, extended to pretraining for large language models. This profile covers 12B-scale experiments, stability, and industry collaborations.
SageMaker HyperPod adds continuous provisioning and custom AMIs to deliver flexible resource provisioning, security-aligned environments, and enterprise-grade control for large-scale foundation model training and inference on EKS.
NVIDIA Blackwell Ultra accelerates AI training and inference with dual-die integration, NVFP4 4-bit precision, 288 GB HBM3e, and high-bandwidth interconnects, enabling scalable AI factories at scale.
An in‑depth profile of NVIDIA Blackwell Ultra, its dual‑die NV‑HBI design, NVFP4 precision, 288 GB HBM3e per GPU, and system‑level interconnects powering AI factories and large‑scale inference.
NVIDIA explores how NVLink Fusion extends scale-up NVLink technologies to hyperscalers, enabling custom CPUs and XPUs with a rack-scale fabric for AI inference at scale.
Overview of NVLink Fusion, enabling scalable, high-bandwidth AI inference across custom CPU/XPU configurations and rack-scale NVLink fabrics.
The Nemotron Nano 2 9B is an open enterprise‑grade reasoning model with a Hybrid Transformer–Mamba backbone and a configurable thinking budget, delivering leading accuracy and 6x throughput for edge and hybrid deployments.
A step-by-step guide to authoring, building, versioning, and publishing production-ready CUDA kernels using the Hugging Face kernel-builder and reproducible Nix builds.
Salesforce and AWS collaborated to optimize GPU utilization, boost resource efficiency, and cut costs by deploying multiple foundation models on SageMaker AI endpoints using inference components. This approach enables granular per-model resource control and dynamic scaling.
Dynamo 0.4 introduces disaggregated serving, SLO-based autoscaling, AIConfigurator, and enhanced observability to accelerate large-model inference at scale with improved efficiency.
TextQuests is a benchmark testing LLM agents in 25 classic text-based Infocom games, emphasizing long-context reasoning and autonomous exploration.
NVIDIA Cosmos Reason is an open, fully customizable reasoning vision-language model for physical AI and robotics. It enables step-by-step multimodal reasoning and boosts robotics performance through post-training refinements.
AI Sheets is an open-source, no-code tool from Hugging Face for building, transforming and enriching datasets using thousands of open models or local models. Try it in Spaces or deploy locally.
NVIDIA and OpenAI optimize gpt-oss-120b and gpt-oss-20b for accelerated inference on Blackwell, achieving up to 1.5M tokens per second on GB200 NVL72, with Day 0 support across cloud to edge.
OpenAI releases two open-weight, mixture-of-experts models—GPT OSS 120B and GPT OSS 20B—utilizing MXFP4 4-bit quantization. They’re licensed Apache 2.0 with a minimal usage policy and are accessible via Hugging Face Inference Providers for on-device, on-prem, and cloud deployments.
OpenAI releases GPT OSS, a pair of 120B and 20B mixture-of-experts models with MXFP4 4-bit quantization, licensed Apache 2.0, and delivered via Hugging Face Inference Providers for on-device and server deployments.
Two open-weight OpenAI models (gpt-oss-120b and gpt-oss-20b) using mixture-of-experts and MXFP4 quantization for fast, memory-efficient reasoning. Licensed Apache 2.0; designed for private/local deployments and on-device use, with Hugging Face Inference Providers integration.
ExecuTorch is Meta’s open-source PyTorch inference framework for on-device ML across its family of apps, delivering lower latency, better privacy, and improved performance on Android and iOS.
This article explains how a Hugging Face optimization recipe speeds up LoRA inference on Flux using Diffusers and PEFT, delivering up to ~2.3x speedups and detailing hotswapping, FP8 quantization, and memory-management techniques.
TinyAgent shows small language models can be fine-tuned for reliable function calling and edge deployment, using curated synthetic data, an LLMCompiler planner, and a Tool RAG approach to power private, low-latency agentic workflows.
A study demonstrating how small language models can perform accurate function calling at the edge using a curated data pipeline, an LLMCompiler-based planner, and on-device execution with macOS integrations.
Mamba replaces Transformer attention with State Space Models (SSMs) to achieve linear scaling in sequence length, enabling fast inference and competitive language performance up to million-token contexts.
A deep dive into Mamba, a State Space Model (SSM) based backbone designed for long-context sequences, offering Transformer-like performance with improved efficiency.