NVIDIA-Certified Professional: AI Operations: key terms

Flashcards for this exam NVIDIA-Certified Professional: AI Operations guide Class dates

• All-ReduceA collective operation that combines values (such as gradients) from all processes and distributes the result back to all.Advanced
• Base Command Manager (BCM)NVIDIA software for provisioning, managing and monitoring AI clusters.Advanced
• Continuous BatchingAn LLM serving technique that adds and removes requests from the running batch at each generation step.Advanced
• CUDACompute Unified Device Architecture: NVIDIA's parallel computing platform and programming model for GPUs.Foundational
• Data Center GPU Manager (DCGM)NVIDIA tools for monitoring, health-checking and diagnosing GPUs in clusters.Advanced
• Data ParallelismDistributed training where each GPU holds a full model copy and processes a different slice of each batch.Intermediate
• FP3232-bit single-precision floating point, the traditional default for neural network math.Intermediate
• FP8An 8-bit floating-point format supported by NVIDIA Hopper and later Tensor Cores for faster training and inference.Advanced
• Fully Sharded Data Parallel (FSDP)A PyTorch training strategy that shards parameters, gradients and optimizer states across data-parallel GPUs.Advanced
• GPUDirect RDMANVIDIA technology that lets network adapters read and write GPU memory directly, bypassing host memory and CPU.Advanced
• Gradient AccumulationSumming gradients over several mini-batches before updating weights to simulate a larger batch.Advanced
• Graphics Processing Unit (GPU)A massively parallel processor, originally for graphics, that accelerates the matrix math behind AI training and inference.Foundational
• Inference ThroughputThe amount of work an inference system completes per unit time, such as tokens or requests per second.Intermediate
• Kubernetes GPU OperatorAn NVIDIA operator that automates deployment of drivers, container toolkit, device plugin and monitoring for GPUs in Kubernetes.Advanced
• KV CacheStored attention keys and values for previous tokens, reused so each new token does not recompute them.Advanced
• Mixed Precision TrainingTraining with lower-precision number formats (FP16/BF16/FP8) for most operations while keeping critical values in FP32.Intermediate
• Multi-Instance GPU (MIG)An NVIDIA feature that partitions one GPU into up to seven isolated instances with dedicated compute and memory.Intermediate
• NCCLNVIDIA Collective Communications Library: optimized multi-GPU and multi-node collectives such as all-reduce.Advanced
• NVIDIA AI Enterprise (NVAIE)NVIDIA's licensed, supported software suite for developing and deploying production AI, including NIM, NeMo and Triton.Intermediate
• NVIDIA DGXNVIDIA's integrated AI systems combining GPUs, NVLink, networking and an optimized software stack.Intermediate
• NVIDIA DGX SuperPODNVIDIA's reference architecture for scalable AI supercomputers built from DGX systems, InfiniBand or Ethernet fabrics and storage.Advanced
• NVIDIA DynamoNVIDIA's open-source distributed inference framework for serving reasoning and generative models across many GPUs.Advanced
• NVIDIA NGCNVIDIA's catalog of GPU-optimized containers, pretrained models, Helm charts and SDKs.Intermediate
• NVIDIA NIMNVIDIA Inference Microservices: prebuilt, optimized containers that serve AI models through industry-standard APIs.Intermediate
• nvidia-smiNVIDIA System Management Interface: a command-line tool to query and manage GPU state.Intermediate
• PagedAttentionA KV-cache memory technique, introduced by vLLM, that stores cache in fixed-size blocks like virtual memory pages.Advanced
• Pipeline ParallelismSplitting a model's layers into sequential stages placed on different GPUs, with micro-batches flowing through.Advanced
• Prefill and DecodeThe two phases of LLM inference: processing the whole prompt in parallel (prefill), then generating tokens sequentially (decode).Advanced
• SlurmAn open-source workload manager and job scheduler widely used for HPC and large AI training clusters.Intermediate
• Speculative DecodingSpeeding up generation by having a small draft model propose tokens that the large model verifies in parallel.Advanced
• Tensor CoreSpecialized NVIDIA GPU units that accelerate matrix multiply-accumulate operations used in deep learning.Intermediate
• Tensor ParallelismSplitting individual layers' weight matrices across GPUs so each computes part of every operation.Advanced
• TensorRT-LLMNVIDIA's open-source library for optimized LLM inference on NVIDIA GPUs.Advanced
• Triton Inference ServerNVIDIA's open-source inference server (now Dynamo-Triton) for serving models from many frameworks on GPUs and CPUs.Intermediate
• Unified Fabric Manager (UFM)NVIDIA's platform for provisioning, monitoring and managing InfiniBand fabrics.Advanced
• vLLMA popular open-source, high-throughput LLM serving engine built around PagedAttention.Advanced