CUDA: Getting started on Windows

If nvidia is the pickaxe of the AI gold rush, CUDA is clearly the handle of that pickaxe; indeed, CUDA is the software layer which gives you direct access to the GPU massive multi-threading capabilities. In short, a nvidia GPU with no CUDA is same as having a car with no driver licence; it only looks good in a garage but there is no way you can fully enjoy it.

CUDA is available for many platforms, this blog is about how to get started on Windows (basically on a PC running Windows). Also keep in mind that the official documentation is available here.

1. Install Visual Studio
Visual Studio is required to have the CUDA toolkit to work (you can actually make it work without Visual Studio but this is more of an advanced way; we'll go with the easier and more beginner friendly way here; also Visual Studio Community Edition is free). So install the free Visual Studio (10 GB on disk required). If you do not install Visual Studio you'll get that warning when installing the CUDA toolkit:

Also note that you'll need this Desktop dev with C++ option checked to be installed with Visual Studio:

2. Download the CUDA toolkit
The Windows CUDA toolkit (3 GB for version 12.4)
3. Install the CUDA toolkit:
Double click on the downloaded file, next, next, etc...
4. Verify the CUDA toolkit version
Just run nvcc -V in a "cmd" window (nvcc being "NVidia CUDA Compiler"):

C:\Users\fredd> nvcc -V
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2024 NVIDIA Corporation
Built on Thu_Mar_28_02:30:10_Pacific_Daylight_Time_2024
Cuda compilation tools, release 12.4, V12.4.131
Build cuda_12.4.r12.4/compiler.34097967_0
C:\Users\fredd>

5. Run a sample
NVidia is nice enough to provide many CUDA samples in this git repository which you can easily clone on your local machine when starting Visual Studio:

Let's try deviceQuery which can be found in the below path:
• Samples => 1_Utilities => deviceQuery
In Visual Studio, right click on deviceQuery then build I could not have it executed from Visual Studio so start a cmd window and you'll find deviceQuery.exe under the bin\win64\Debug path (this is not clear at all in the official documentation); run it from there:

C:\Users\fredd\source\repos\cuda-samples\bin\win64\Debug> deviceQuery.exe  
deviceQuery.exe Starting...

 CUDA Device Query (Runtime API) version (CUDART static linking)

Detected 1 CUDA Capable device(s)

Device 0: "NVIDIA GeForce RTX 2070"
  CUDA Driver Version / Runtime Version          12.4 / 12.4
  CUDA Capability Major/Minor version number:    7.5
  Total amount of global memory:                 8192 MBytes (8589475840 bytes)
  (036) Multiprocessors, (064) CUDA Cores/MP:    2304 CUDA Cores
  GPU Max Clock rate:                            1620 MHz (1.62 GHz)
  Memory Clock rate:                             7001 Mhz
  Memory Bus Width:                              256-bit
  L2 Cache Size:                                 4194304 bytes
  Maximum Texture Dimension Size (x,y,z)         1D=(131072), 2D=(131072, 65536), 3D=(16384, 16384, 16384)
  Maximum Layered 1D Texture Size, (num) layers  1D=(32768), 2048 layers
  Maximum Layered 2D Texture Size, (num) layers  2D=(32768, 32768), 2048 layers
  Total amount of constant memory:               65536 bytes
  Total amount of shared memory per block:       49152 bytes
  Total shared memory per multiprocessor:        65536 bytes
  Total number of registers available per block: 65536
  Warp size:                                     32
  Maximum number of threads per multiprocessor:  1024
  Maximum number of threads per block:           1024
  Max dimension size of a thread block (x,y,z): (1024, 1024, 64)
  Max dimension size of a grid size    (x,y,z): (2147483647, 65535, 65535)
  Maximum memory pitch:                          2147483647 bytes
  Texture alignment:                             512 bytes
  Concurrent copy and kernel execution:          Yes with 6 copy engine(s)
  Run time limit on kernels:                     Yes
  Integrated GPU sharing Host Memory:            No
  Support host page-locked memory mapping:       Yes
  Alignment requirement for Surfaces:            Yes
  Device has ECC support:                        Disabled
  CUDA Device Driver Mode (TCC or WDDM):         WDDM (Windows Display Driver Model)
  Device supports Unified Addressing (UVA):      Yes
  Device supports Managed Memory:                Yes
  Device supports Compute Preemption:            Yes
  Supports Cooperative Kernel Launch:            Yes
  Supports MultiDevice Co-op Kernel Launch:      No
  Device PCI Domain ID / Bus ID / location ID:   0 / 7 / 0
  Compute Mode:
      < Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) >

deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 12.4, CUDA Runtime Version = 12.4, NumDevs = 1
Result = PASS
C:\Users\fredd\source\repos\cuda-samples\bin\win64\Debug>

That's it, you have a working CUDA dev environment on Windows!