Computer Vision Workload Analysis: Memory Leaks & System Debugging

Unit tests do not find some production failures. This post describes how we debugged a computer vision benchmark platform. The platform processes many video streams at the same time. It runs on CPU (AMX and non-AMX) and GPU backends. We found three linked bugs. They caused full disks, tasks that ran again and again, and too much memory use.

The system under test

The platform runs in Docker. A Celery coordinator sends benchmark scenarios to specialized workers.

┌─────────────────────────────────────────────────┐
│  API Server (FastAPI)                           │
│    └── Celery Coordinator                       │
│         ├── CV Worker (AMX) ─── OpenVINO Async  │
│         ├── CV Worker (CPU) ─── OpenVINO Direct │
│         └── CV Worker (GPU) ─── HTTP → GPU Srv  │
│                                                 │
│  Infrastructure: Redis │ PostgreSQL │ Prometheus │
└─────────────────────────────────────────────────┘

Each scenario runs at increasing concurrency levels: 1, 2, 4, 8, 16, 32, and 64 streams. Each stream processes 210 video frames through YOLO-based detection pipelines.

Bug 1: memory leak

The first symptom was a full disk at 64 concurrent streams. The root cause was not the disk. The root cause was memory.

Root cause

In openvino_async.py, the inference-only path copies frames. It does this to reach a minimum benchmark duration.

# openvino_async.py - The culprit
preprocessed = [self._preprocess_frame(frame) for frame in frames]

if inference_only:
    min_frames_for_benchmark = 5000
    iterations = max(1, min_frames_for_benchmark // len(preprocessed))
    if iterations > 1:
        preprocessed = preprocessed * iterations  # Replicate frames

Each preprocessed frame uses 4.9 MB (shape [1, 3, 640, 640] float32). A 300-frame video is copied 16 times to make 4800 frames. This uses 23.6 GB per stream.

The multi-stream CPU/AMX path starts num_streams threads. Each thread makes its own set of copies. At 2 streams, the system uses 47 GB. At 4 streams, it uses 94 GB. The numpy arrays need physical RAM. When the system swaps, it fills the disk.

The fix for the leak

We added explicit memory cleanup at four places.

# 1. benchmark.py - Between concurrency levels
del video_results
gc.collect()

# 2. openvino_async.py - After distributing to threads
del preprocessed    # Free 23GB before inference starts
gc.collect()

# 3. vvl_gpu_http.py - After HTTP POST
del payload         # Free serialized frames
del preprocessed

# 4. openvino_direct.py - After batch distribution
del all_preprocessed
gc.collect()

Python scoping error

After we deployed the fix, we got the error cannot access local variable 'gc'. Python treats a variable as local in the whole function if the function has an import gc anywhere. This is the cause of the error.

def _run_single_stream_level(self):
    gc.collect()       # Line 1465 - UnboundLocalError!
    # ... 50 lines later ...
    import gc          # Line 1521 - Python sees this and makes gc "local"
    gc.collect()       # This one works fine

The fix: we removed all 4 local import gc statements. The code now uses the top-level import.

Bug 2: full disk from oneDNN logs

The disk kept filling after the memory fix. Monitoring showed that RAM was now correct. The container logs filled the disk.

The Docker Compose file set ONEDNN_VERBOSE=1 for the workers. This setting writes a log line for every oneDNN kernel run. At 30 or more concurrent streams, with 210 frames and 50 or more layers per frame, it writes millions of log lines. Docker stores them in JSON log files.

The fix for the logs

# docker-compose.yaml
environment:
  - ONEDNN_VERBOSE=0    # Was 1

# Also added log rotation
logging:
  driver: "json-file"
  options:
    max-size: "50m"
    max-file: "3"

Bug 3: infinite loop from the Celery timeout

This bug was the hardest to find. A CV batch finished successfully. Then it started again immediately. It did this without end.

05:40:15 - Task succeeded (took ~2h 37m)
05:40:15 - Task received  ← redelivered copy!  
08:17:20 - Task succeeded (took ~2h 37m)
08:17:20 - Task received  ← redelivered AGAIN

Root cause of the loop

The Celery configuration used task_acks_late=True. This setting means a task is not acknowledged until it finishes. The Celery Redis broker has a default visibility_timeout of 3600 seconds (1 hour). A CV batch with 3 scenarios takes about 3 hours.

The steps were:

1. The coordinator task starts and runs 3 CV scenarios (about 3 hours).
2. After 1 hour, Redis decides the message is lost. The task has not sent an acknowledgment.
3. Redis sends the message again to the queue.
4. The original run finishes and sends its acknowledgment.
5. The worker takes the copy from the queue.
6. The cycle repeats without end.

The fix for the loop

# celery_app.py
broker_transport_options={"visibility_timeout": 14400},  # 4 hours

GPU use analysis

GPU use showed 0% most of the time. Monitoring over time showed the reason.

16 streams (3360 frames):
  Preprocessing: 31.8s (CPU) — GPU idle
  Inference:      3.7s (GPU at 97%)
  → GPU busy only ~10% of the time

32 streams (6720 frames):
  Preprocessing: 61.5s (CPU) — GPU idle
  Inference:      ~5s  (GPU at 99%)
  → GPU busy only ~7% of the time

The root cause is that preprocessing and serialization use only the CPU. They take most of the wall-clock time. The GPU model server gets frames after the worker finishes preprocessing and HTTP serialization. YOLO inference runs at about 3500 FPS combined. It finishes in seconds and then waits for the next batch.

Docker volume path bug

A separate problem made OVMS (OpenVINO Model Server) containers fail with the error is a directory: permission denied. Docker made empty directories. It did not mount the entrypoint script file.

The root cause: Docker resolves ./ paths in docker-compose.yaml from the directory where you run docker compose. If you run it from a different directory, Docker makes the bind mount targets as empty directories. It does not use the real files.

The fix: the code now reads the host project directory from the Docker socket API when no path is set. New servers need no manual path configuration.

Key takeaways

• Python does not always free large numpy or tensor arrays at once. For memory-critical work, use del and gc.collect().
• An import inside a function makes that name local in the whole function body. This includes the lines before the import.
• With task_acks_late=True, make the Celery Redis visibility_timeout longer than the longest task. Many people miss this rule.
• Always rotate container logs for verbose workloads. One ONEDNN_VERBOSE=1 flag can write gigabytes of logs each hour.
• GPU use metrics can mislead you. Low use can mean the CPU preprocessing is the bottleneck.
• Bind mount paths that start with ./ break easily. Use absolute paths or auto-detection for portable deployments.