Running Real Research on Rented GPUs

Much of the work shown on this site — the PENNANT port and optimization campaign across NVIDIA T4, H100, and AMD MI300X — was done self-funded on rented cloud GPUs, not on a standing cluster. That constraint is a feature: it forces the discipline that makes multi-vendor performance work trustworthy and affordable. This page is a candid operator's log of what that actually takes — which providers and access paths are reliable, where the sharp edges are, and the guardrails we now run by default.

Honest operational competence on rented infrastructure is itself a deliverable. A client paying for cross-vendor benchmarking should not also pay for forgotten instances, flaky tooling, or measurement artifacts.

Providers we have actually used

Provider GPUs used Access path Verdict
Hot Aisle AMD MI300X (gfx942, ROCm 7.2.4) direct ssh, prepaid, hotaisle CLI Most reliable. Clean SSH, fast bring-up, ~$2/hr. Zero lost runs.
Brev (NVIDIA / Hyperstack) T4 (sm_75), H100 PCIe (sm_90) portal + brev CLI / brev exec Excellent portal; the CLI has sharp edges (below).
AMD Developer Cloud MI300X web instance + SSH Good when available; capacity-constrained in practice.

The same kernels were built and profiled on all of these from one clean-room codebase (CUDA, HIP, OpenMP-target), with native vendor tooling on each — Nsight Compute on NVIDIA, rocprof / rocprof-compute on AMD.

What works, and what bites

Brev's portal is genuinely good. The GPU-type carousel with live $/hr, VRAM, provider, and an explicit "no stop/start" warning surfaces exactly the information that prevents a bad provisioning choice. First-time bring-up via the portal takes a few minutes.

Brev's CLI needs care. Over the campaign it produced a recurring set of surprises worth knowing before you depend on it:

  • brev exec forces a pseudo-TTY and an SSH control-master by default. For non-interactive commands that mangles captured output and can hang a long-running command's return entirely. Fix: drive instances with direct SSH (ssh -i ~/.brev/brev.pem -o RequestTTY=no -o ControlMaster=no shadeform@<ip>) — the same plain-SSH pattern that made the Hot Aisle MI300X work flawless. For long jobs, nohup to a file and poll with short SSH commands; never depend on one long-lived exec's stdout.
  • Several flags don't behave as documented (--disk is implicit in the machine type; the GPU string format varies per provider), and Codeberg-hosted git URLs could panic the workspace creator. Pull --help rather than trusting memory.
  • There is no automatic idle teardown. An instance left running bills until you delete it.

Profiler tooling is not always turnkey. rocprof-compute (omniperf) did not run out of the box on ROCm 7.2.4 — it needed a Python virtualenv for its dependencies and a one-line patch to its parser for modern pandas. Knowing the tool's failure modes is part of the job; we document the fix rather than abandon the measurement.

The guardrails we run by default

These are not optional niceties — they are what makes rented-GPU research safe and reproducible:

  1. Guaranteed teardown on every instance. The moment a paid instance is created, it gets an independent timer that force-deletes it (( sleep 1800; <provider> delete <name> ) &). It fires no matter what — a dropped connection, a hung job, a closed laptop. Teardown must never depend on anyone remembering. (We learned this the expensive way once; now it is automatic.)
  2. Direct SSH over wrapper CLIs for any command whose output matters — reliability beats convenience.
  3. Round-robin A/B timing on shared / virtualized GPUs. Cloud instances are often VF-partitioned and cold on first use; a naïve "config A then config B" comparison measures cold-vs-warm, not the configs. We interleave repetitions — it caught a phantom 7% "speedup" that was pure thermal drift.
  4. Profile small, profile cheap. A stall or occupancy signature needs one clean profile per kernel on a tiny mesh, not a full run on a big one — the difference between a 5-second capture and a hung 30-minute one.
  5. Reproducible bootstrap. Every box is brought up from a scripted build-and-validate, so a result can be regenerated on a fresh rental.

Why it matters for clients

Cross-vendor performance numbers are only worth anything if they were gathered competently: on the right hardware, with the right tools made to actually work, measured in a way that survives scrutiny, and without burning budget on idle silicon. We have done exactly this — NVIDIA and AMD, multiple providers, self-funded — and the practices above travel directly to a client engagement on their preferred cloud, at their budget.

See the PENNANT optimization campaign for the technical results those rented GPUs produced.