NVIDIA-Certified Associate: AI Infrastructure and Operations: key terms

Flashcards for this exam NVIDIA-Certified Associate: AI Infrastructure and Operations guide Class dates

• Artificial Intelligence (AI)The field of building computer systems that perform tasks normally requiring human intelligence, such as reasoning, perception and language.Foundational
• Batch SizeThe number of training examples processed together before the model's parameters are updated.Foundational
• BF16Brain Floating Point 16: a 16-bit format with FP32's exponent range but less mantissa precision.Intermediate
• CUDACompute Unified Device Architecture: NVIDIA's parallel computing platform and programming model for GPUs.Foundational
• Data Center GPU Manager (DCGM)NVIDIA tools for monitoring, health-checking and diagnosing GPUs in clusters.Advanced
• Data ParallelismDistributed training where each GPU holds a full model copy and processes a different slice of each batch.Intermediate
• Data Processing Unit (DPU)A programmable processor that offloads networking, storage and security tasks from the host CPU.Intermediate
• Deep Learning (DL)A subset of machine learning that uses multi-layer neural networks to learn hierarchical representations directly from raw data.Foundational
• Edge AIRunning AI inference on devices near the data source, such as cameras, robots or phones, rather than in the cloud.Intermediate
• FP1616-bit half-precision floating point, halving memory versus FP32 with a narrower numeric range.Intermediate
• FP3232-bit single-precision floating point, the traditional default for neural network math.Intermediate
• FP4A 4-bit floating-point format accelerated by NVIDIA Blackwell Tensor Cores for high-throughput inference.Advanced
• FP8An 8-bit floating-point format supported by NVIDIA Hopper and later Tensor Cores for faster training and inference.Advanced
• GB200 NVL72A liquid-cooled rack-scale NVIDIA system linking 36 Grace CPUs and 72 Blackwell GPUs in a single NVLink domain.Advanced
• GPUDirect RDMANVIDIA technology that lets network adapters read and write GPU memory directly, bypassing host memory and CPU.Advanced
• Graphics Processing Unit (GPU)A massively parallel processor, originally for graphics, that accelerates the matrix math behind AI training and inference.Foundational
• High Bandwidth Memory (HBM)Stacked DRAM packaged alongside the GPU that provides very high memory bandwidth.Intermediate
• InferenceUsing a trained model to make predictions or generate outputs on new data.Foundational
• Inference ThroughputThe amount of work an inference system completes per unit time, such as tokens or requests per second.Intermediate
• InfiniBandA high-throughput, low-latency network fabric with native RDMA, widely used to connect GPU clusters.Intermediate
• Kubernetes GPU OperatorAn NVIDIA operator that automates deployment of drivers, container toolkit, device plugin and monitoring for GPUs in Kubernetes.Advanced
• KV CacheStored attention keys and values for previous tokens, reused so each new token does not recompute them.Advanced
• Mixed Precision TrainingTraining with lower-precision number formats (FP16/BF16/FP8) for most operations while keeping critical values in FP32.Intermediate
• Model ParametersThe internal values, such as weights and biases, that a model learns from training data.Foundational
• Model ServingHosting a trained model behind an interface so applications can request predictions.Foundational
• Model TrainingThe process of fitting a model's parameters to data by optimizing an objective function.Foundational
• Multi-Instance GPU (MIG)An NVIDIA feature that partitions one GPU into up to seven isolated instances with dedicated compute and memory.Intermediate
• NCCLNVIDIA Collective Communications Library: optimized multi-GPU and multi-node collectives such as all-reduce.Advanced
• Neural Network (NN)A model made of layers of interconnected artificial neurons that learn weighted transformations of input data.Foundational
• NVIDIA AI Enterprise (NVAIE)NVIDIA's licensed, supported software suite for developing and deploying production AI, including NIM, NeMo and Triton.Intermediate
• NVIDIA BlackwellNVIDIA's GPU architecture following Hopper, used in B200, B300 and GB200/GB300 systems, adding FP4 and fifth-generation NVLink.Intermediate
• NVIDIA DGXNVIDIA's integrated AI systems combining GPUs, NVLink, networking and an optimized software stack.Intermediate
• NVIDIA HopperNVIDIA's GPU architecture behind the H100 and H200, introducing FP8 Transformer Engine and fourth-generation NVLink.Intermediate
• NVIDIA NGCNVIDIA's catalog of GPU-optimized containers, pretrained models, Helm charts and SDKs.Intermediate
• NVIDIA NIMNVIDIA Inference Microservices: prebuilt, optimized containers that serve AI models through industry-standard APIs.Intermediate
• nvidia-smiNVIDIA System Management Interface: a command-line tool to query and manage GPU state.Intermediate
• NVLinkNVIDIA's high-bandwidth, low-latency GPU-to-GPU interconnect.Intermediate
• ONNXOpen Neural Network Exchange: an open format for representing models so they can move between frameworks and runtimes.Intermediate
• Prefill and DecodeThe two phases of LLM inference: processing the whole prompt in parallel (prefill), then generating tokens sequentially (decode).Advanced
• QuantizationReducing the numeric precision of model weights (and sometimes activations) to shrink memory and speed inference.Intermediate
• RAPIDSNVIDIA's suite of open-source GPU-accelerated data science libraries with familiar Python APIs.Intermediate
• SlurmAn open-source workload manager and job scheduler widely used for HPC and large AI training clusters.Intermediate
• Spectrum-XNVIDIA's Ethernet networking platform for AI, pairing Spectrum-4 switches with BlueField-3 SuperNICs.Advanced
• TensorA multi-dimensional array of numbers, the core data structure in deep learning frameworks.Foundational
• Tensor CoreSpecialized NVIDIA GPU units that accelerate matrix multiply-accumulate operations used in deep learning.Intermediate
• TensorRTNVIDIA's SDK for optimizing trained models into high-performance inference engines.Intermediate
• Time to First Token (TTFT)The latency between sending a request and receiving the first generated token.Intermediate
• Transformer EngineNVIDIA hardware and software that dynamically manages FP8/FP4 precision to accelerate transformer models.Advanced
• Triton Inference ServerNVIDIA's open-source inference server (now Dynamo-Triton) for serving models from many frameworks on GPUs and CPUs.Intermediate
• WeightsLearned numeric parameters that scale the strength of connections between neurons.Foundational