Intel AMX: Understanding CPU-Based Matrix Acceleration

Dedicated GPUs get most of the attention for AI speed. Intel Advanced Matrix Extensions (AMX) offer another option. AMX puts matrix multiplication hardware inside the CPU. You can run AI workloads faster without a graphics card.

Each topic in this post has two parts. The first part uses a simple example. The second part gives the technical details. You can read the topics in any order.

What AMX is

Think of a grocery list. A traditional CPU adds the items one by one: apples, then oranges, then bread, then milk. This works, but it is slow.

AMX is like a calculator that adds the whole shopping cart at one time.

An AI model is a large set of multiplications and additions. AMX is hardware in newer Intel CPUs. It does many of these calculations at the same time. AI programs run faster, and you do not need a graphics card.

AMX uses tile-based matrix operations. A vector is a one-dimensional array of numbers. AMX does not work on vectors. It works on tiles. A tile is a two-dimensional block of a matrix. Each tile fits in a dedicated tile register.

The TMUL (Tile Matrix Multiply) unit does the math. TMUL is hardware made only for matrix operations. One TDPBF16PS instruction multiplies two full tiles. Without AMX, the CPU needs hundreds of traditional instructions for the same work.

BF16 number format

When you work out a tip at a restaurant, you do not need 15 decimal places. You round to the nearest cent.

BF16 (Brain Float 16) works the same way for AI. A traditional computer uses 32 bits to store a number. This gives very high precision. AI does not need that much detail. BF16 uses 16 bits, which is half the space. AI keeps enough accuracy to work correctly. The CPU moves less data, so it processes the data faster.

AMX processes BF16 natively in hardware. Without AMX, the CPU must emulate BF16. It converts BF16 to FP32, does the math, and converts the result back to BF16. This conversion removes the memory benefit of BF16.

With AMX, the data stays in BF16 for the whole computation. The TMUL unit multiplies and adds BF16 values directly. There is no conversion delay. The CPU makes full use of the memory bandwidth.

Running many tasks

Think of a car factory. If one worker builds a whole car alone, it takes a long time. On an assembly line, many workers build different cars at the same time. The factory makes more cars each hour.

Python is the language that most AI programs use. Python has a limit: only one thing can run at a time. It is like an assembly line where only one worker can move at a time.

AsyncInferQueue removes this limit. It moves the work to a separate factory floor. Many workers can work there at the same time, outside the Python limit.

The Python Global Interpreter Lock (GIL) stops Python from running threads in parallel. For AI inference, this is a bottleneck. Inference is the step where a trained model makes a prediction. With the GIL, Python can process only one request at a time, even on fast AMX hardware.

AsyncInferQueue from OpenVINO manages the requests at the C++ level. The inference runs in parallel threads outside the control of Python. This keeps the AMX hardware fully busy.

# Create queue with multiple parallel inference slots
async_queue = AsyncInferQueue(compiled_model, jobs=16)

# Submit requests (non-blocking, GIL released)
for frame in preprocessed_frames:
    async_queue.start_async({0: frame})

# Wait for all completions
async_queue.wait_all()

The oneDNN library

You do not need to know how a car engine works to drive a car. In the same way, you do not need to know your CPU to use oneDNN. oneDNN is a software library from Intel. It uses the best CPU features that are available.

When you run an AI model, oneDNN checks which features your CPU has. It then picks the fastest method. It is like a GPS that picks the fastest route for the traffic now. You say where to go, and it does the rest.

oneDNN hides the hardware details from you. When it compiles an operation, it asks the CPU which ISAs (Instruction Set Architectures) it supports. It then picks the best implementation.

The ONEDNN_MAX_CPU_ISA environment variable lets you set the limit yourself:

# Enable all instructions including AMX
export ONEDNN_MAX_CPU_ISA=DEFAULT

# Disable AMX, use only AVX-512
export ONEDNN_MAX_CPU_ISA=AVX512_CORE_VNNI

# Fallback to basic AVX2
export ONEDNN_MAX_CPU_ISA=AVX2

Key points

These are the four main points of this post:

1. AMX processes blocks of a matrix, not single numbers. This design lets AI workloads run many operations in parallel.
2. AMX uses BF16 natively. There is no conversion, so the CPU uses the full memory bandwidth.
3. AsyncInferQueue gives true parallelism. You must avoid the Python GIL to keep AMX fully busy.
4. oneDNN does the hard work for you. Frameworks use the best instructions without manual setup.

AMX makes modern Intel CPUs good tools for AI inference. It does not replace GPUs for training. It does change the cost of AI inference on CPUs.

If you run AI workloads on Intel hardware, AMX can give you better performance. You do not need to change your infrastructure.