API Reference

This page is generated from the source docstrings. For task-oriented guides, see Units and Probe and Snapshot.

Machine

mainboard.Machine

Bases: Singleton

Singleton facade for the host and hardware units.

host cached property

Detected host CPU, memory, and disk.

cpu cached property

Detected host CPU.

gpus cached property

Detected GPUs across supported providers.

npus cached property

Detected neural processing units.

environment cached property

The host's execution context: user, group(s), and job scheduler on PATH.

board cached property

The host's motherboard and firmware identity.

toolchain cached property

C/C++/CUDA compilers and build systems found on the host PATH.

units cached property

All detected schedulable units.

compilers cached property

Detected host compilers with CMake build configuration.

nvcc_path cached property

Absolute path to the nvcc binary.

cuda_architecture cached property

CUDA compute capability without a dot, e.g. 89.

snapshot()

Probe the host's compute resources into one serializable model.

Provider detection is best-effort: a host with no accelerator yields empty gpus and npus rather than raising.

Units

mainboard.Unit

Bases: FrozenModel

Schedulable hardware execution resource.

A unit can be a CPU package or cluster, GPU, NPU, DSP, or other hardware engine that executes work over memory.

label cached property

Human-readable unit name.

architecture cached property

Human-readable architecture or generation.

memory property

Memory visible to this unit.

clock_readings property

Clock readings grouped by hardware domain.

utilization property

Normalized utilization where available.

energy property

Power and cumulative energy where available.

thermal property

Thermal state where available.

snapshot(name='')

Capture neutral telemetry for this unit.

mainboard.CPU

Bases: Unit

Host CPU package or SoC CPU cluster.

label cached property

CPU model name.

architecture cached property

CPU architecture string.

clock_readings property

CPU clock readings from the OS.

memory property

System memory visible to the CPU.

mainboard.GPU

Bases: Unit, Registry

GPU with telemetry and legacy profiling sensor accessors.

Registry root: concrete vendor providers self-register on import, and all fans out over them, concatenating each provider's own probe.

label cached property

Human-readable GPU name.

uuid cached property

Stable GPU identifier when the provider exposes one.

architecture cached property

Human-readable architecture or generation name.

arch_key cached property

A stable, machine-friendly architecture id for per-arch dispatch.

The key to look this device up in an arch-keyed table (see mainboard.profiling.arch_config): vendor backends return a precise, dot-free target such as sm_90 (NVIDIA) so tile sizes and kernel configs can be pinned per generation. The base falls back to the lowercased human architecture name.

peak_bandwidth_gbs cached property

Theoretical peak memory bandwidth in GB/s when known.

driver_version cached property

Driver or runtime version as (major, minor) when known.

memory property

Current accelerator memory state.

clocks property

Current compute and memory clocks.

clock_readings property

Clock readings grouped by hardware domain.

utilization property

Current compute and memory-controller utilization.

thermal property

Current thermal state.

energy property

Current power and cumulative energy reading.

pcie property

Current host interconnect throughput.

fan_speed_pct property

Fan speed percentage, or zero when unavailable.

temperature_c property

Current GPU temperature in Celsius.

gpu_util_pct property

Current compute utilization percentage.

processes property

Processes using this accelerator.

all() classmethod

Return GPUs visible across every registered provider.

Probing is best-effort per provider: a backend whose all raises (a binding that loads but then throws, an unexpected NVML error) is logged and skipped so one broken vendor never sinks the whole machine probe.

probe(provider) classmethod

One provider's devices, or an empty tuple when its probe fails.

snapshot(name='')

Point-in-time reading of all common sensor properties.

name: profiling region label to embed in the snapshot.

mainboard.NPU

Bases: Unit, Registry

Neural processing unit.

Registry root: concrete vendor providers self-register on import, and all fans out over them, concatenating each provider's own probe.

all() classmethod

Return NPUs visible across every registered provider.

Probing is best-effort per provider: a backend whose all raises is logged and skipped so one broken vendor never sinks the whole probe.

probe(provider) classmethod

One provider's devices, or an empty tuple when its probe fails.

Snapshots

mainboard.MachineSnapshot

Bases: FrozenModel

One-call JSON-serializable probe of a host's compute resources.

timestamp_ns: monotonic timestamp set at construction. hostname: network name of the probed host. cpu: host CPU identity and capacity. memory: system RAM usage at probe time. gpus: detected GPUs with per-device telemetry, empty when none are present. npus: detected neural processing units, empty when none are present. environment: the user, group(s), and job scheduler available on the host. board: the host's motherboard and firmware identity. toolchain: C/C++/CUDA compilers and build systems found on the host PATH. cgroup_memory: the enforced cgroup memory cap, the real OOM-kill ceiling for the job. scratch: the fastest writable node-local scratch tier with its free space.

unit_count property

Total schedulable units: CPU plus every GPU and NPU.

kinds property

Distinct unit kinds present on the host.

mainboard.UnitSnapshot

Bases: FrozenModel

Point-in-time neutral telemetry for one unit.

name: caller-provided label, e.g. a profiling region. unit_name: human-readable unit name. kind: unit category. vendor: hardware vendor. timestamp_ns: monotonic timestamp set at construction. clocks: clock readings by hardware domain. memory: memory visible to the unit. utilization: normalized utilization when available. energy: instantaneous power and cumulative energy when available. thermal: thermal reading when available.

mainboard.GPUSnapshot

Bases: UnitSnapshot

Point-in-time reading of all GPU sensors.

pcie: PCIe bus TX/RX throughput counters. fan_speed_pct: fan duty cycle as a percentage (0 if no fan or unsupported). processes: list of compute processes and their GPU memory usage.

Memory

mainboard.Memory

Bases: FrozenModel

Memory usage for a host, unit, or memory region.

total_bytes: total capacity. used_bytes: currently used bytes when known. free_bytes: currently free bytes when known. scope: region name, e.g. system, vram, unified. unified: whether CPU and accelerator share the memory pool. source: provider that produced the value. supported: whether this platform exposes the reading.

total_gb property

Total capacity in gibibytes.

used_gb property

Used capacity in gibibytes.

free_gb property

Free capacity in gibibytes.

percent_used property

Percentage of total memory currently used; 0 when total is 0.

system(scope='system', unified=False) classmethod

Live system RAM usage sampled from psutil.

scope: region name to record, e.g. system or unified. unified: whether CPU and accelerators share this pool.

mainboard.MemoryHardware

Bases: FrozenModel

Physical DIMM slots and swap space for the host.

All properties are lazily evaluated; no data is captured at construction.

cards cached property

DIMM slot details from dmidecode; empty when unavailable.

swap_total_bytes property

Total swap space in bytes.

swap_used_bytes property

Swap currently in use in bytes.

swap_total_gb property

Total swap space in gibibytes.

speed_mhz property

Maximum speed across populated slots; None when cards is unavailable.

slots_total property

Total DIMM slot count; 0 when cards is unavailable.

slots_used property

Number of populated DIMM slots.

Runtime metrics

mainboard.Meter(machine)

Times a region and tracks peak host and GPU memory across samples.

Memory is sampled from the live Machine snapshot at enter, at every explicit sample(), and at exit; peaks are the maximum used bytes over all samples. No background thread is used: callers drive sampling.

peak_host_gb property

Highest host memory in use across samples, in gibibytes.

peak_gpu_gb property

Highest total GPU memory in use across samples, in gibibytes.

host_delta_gb property

Host memory growth from the first to the last sample, in gibibytes.

sample()

Capture one host and GPU memory reading from the live machine.

mainboard.meter

MemorySource

Bases: Protocol

A unit or host that exposes a memory reading.

MeteredMachine

Bases: Protocol

The slice of a machine the meter samples: its host and GPUs.

Meter(machine)

Times a region and tracks peak host and GPU memory across samples.

Memory is sampled from the live Machine snapshot at enter, at every explicit sample(), and at exit; peaks are the maximum used bytes over all samples. No background thread is used: callers drive sampling.

peak_host_gb property

Highest host memory in use across samples, in gibibytes.

peak_gpu_gb property

Highest total GPU memory in use across samples, in gibibytes.

host_delta_gb property

Host memory growth from the first to the last sample, in gibibytes.

sample()

Capture one host and GPU memory reading from the live machine.

meter()

Open a runtime-metrics meter bound to the current machine.

Use as with meter() as m: ..., then read m.elapsed_s, m.peak_host_gb, m.peak_gpu_gb, and m.host_delta_gb.

Host

mainboard.Environment

Bases: FrozenModel

The host's execution environment: who is running and what scheduler is available.

Probed from the OS and PATH so a tool can route work without re-detecting the user, group, or job scheduler on its own.

user: login name of the current user. group: primary group name of the current user. groups: every group the current user belongs to. scheduler: the job scheduler found on PATH.

probe() classmethod

Detect the current user, group(s), and job scheduler.

mainboard.Board

Bases: FrozenModel

The host's motherboard and firmware identity.

Probed from the OS so a tool can record which physical system board and BIOS a snapshot came from. Unreadable on a host means empty strings, never an error.

vendor: motherboard manufacturer. model: motherboard product name. version: motherboard revision or model identifier. bios_vendor: firmware vendor. bios_version: firmware version string.

probe() classmethod

Detect the motherboard and BIOS identity for the current platform.

Toolchain

mainboard.Toolchain

Bases: FrozenModel

Build tools discovered on the host, grouped by category.

Each registered ToolProbe is run once at probe time; only tools found on PATH are kept, so the model reports the host's real build capability.

tools: every available tool, ordered as registered in TOOL_PROBES.

by_category property

Available tools bucketed by their toolchain category.

probe(probes=TOOL_PROBES) classmethod

Run every probe and keep the tools that are present on PATH.

mainboard.DetectedTool

Bases: FrozenModel

A build tool discovered on the host PATH.

name: display name of the tool, e.g. gcc or cmake. category: the toolchain group the tool belongs to. path: absolute path to the resolved binary, None when not on PATH. version: parsed version string, None when absent or unparseable. available: whether the binary was found on PATH.

mainboard.ToolProbe(name, category, binaries, version_args=('--version',), pattern=re.compile('(\\d+\\.\\d+(?:\\.\\d+)?)')) dataclass

Immutable recipe for discovering one build tool.

A probe is pure data: adding a tool to the host inventory means appending one ToolProbe to TOOL_PROBES, with no change to the discovery logic.

name: display name reported on the DetectedTool. category: toolchain group the tool belongs to. binaries: candidate executable names, tried in order until one is on PATH. version_args: arguments that make the binary print its version. pattern: regex whose first group captures the version from the command output.

detect()

Resolve the first available binary and parse its version.

mainboard.ToolCategory

Bases: StrEnum

Grouping for a discovered build tool in the host toolchain.

Providers

mainboard.AppleGPU

Bases: GPU

Apple Silicon integrated GPU backed by unified memory.

record cached property

Raw system_profiler display record.

label cached property

Apple GPU model name.

uuid cached property

Stable system UUID used as the integrated GPU identifier.

architecture cached property

Apple SoC family backing this GPU.

core_count cached property

Number of Apple GPU cores (the profiler reports it as a numeric string).

metal_support cached property

Metal support string reported by macOS.

memory property

Unified memory visible to CPU, GPU, and Neural Engine.

clock_readings property

Apple GPU clocks are not exposed without privileged sampling.

is_available() classmethod

Whether this host reports an Apple Silicon GPU.

gpu_records() cached classmethod

Apple GPU records from system_profiler.

all() classmethod

Return Apple Silicon GPUs reported by macOS.

mainboard.AppleNPU

Bases: NPU

Apple Neural Engine backed by unified memory.

label cached property

Apple Neural Engine model name.

architecture cached property

Apple SoC family backing the Neural Engine.

memory property

Unified memory visible to CPU, GPU, and Neural Engine.

clock_readings property

Apple Neural Engine clocks are not exposed through public APIs.

is_available() classmethod

Whether this host is an Apple Silicon machine.

all() classmethod

Return the local Apple Neural Engine when present.

mainboard.NvidiaGPU

Bases: GPU

NVIDIA CUDA device: static identity, build info, and live NVML sensors.

apis cached property

CUDA/NVML module handles.

cuda_device cached property

Stable cuda.core.Device instance for this visible index.

Only reached behind has_cuda_core, so the optional class is present here.

system_api cached property

The cuda.core.system module, present only behind has_cuda_core.

system_device cached property

Stable cuda.core.system.Device instance for NVML-backed data.

Only reached behind has_cuda_core, so the optional module is present here.

pci_bus_id cached property

PCI bus ID of the visible device, honoring CUDA_VISIBLE_DEVICES.

Read through cuda.bindings.runtime so it works even when the optional cuda.core layer failed to import.

handle cached property

NVML device handle resolved via PCI bus ID to respect CUDA_VISIBLE_DEVICES.

label cached property

Full GPU name string, e.g. NVIDIA GeForce RTX 4090.

uuid cached property

Unique NVIDIA GPU identifier.

cuda_architecture cached property

CUDA compute capability, e.g. ComputeCapability(8, 9).

runtime_properties cached property

Static device properties from cuda.bindings.runtime.

ABI-stable source for SM count and bandwidth when the optional cuda.core layer is unavailable.

architecture cached property

Human-readable NVIDIA architecture name, e.g. Ada.

arch_key cached property

The sm_NN compute-capability target, e.g. sm_90 — the per-arch dispatch key.

sm_count cached property

Number of streaming multiprocessors.

peak_bandwidth_gbs cached property

Theoretical peak memory bandwidth in GB/s.

driver_version cached property

Maximum CUDA version supported by the installed driver.

cuda_python cached property

Detected CUDA Python stack variant and CUPTI availability.

coherent cached property

Whether this GPU shares a cache-coherent memory pool with the host.

Probed, not guessed: a device that reports both cudaDevAttrPageableMemoryAccess and cudaDevAttrConcurrentManagedAccess sits on a coherent fabric where host RAM is a peer NUMA node of HBM (Grace Hopper, GB10), not a PCIe copy away. A discrete card (the 4090) reports neither, so unified stays False there. A binding that lacks the attribute query degrades to False rather than raising.

memory property

CUDA-visible GPU memory allocation state.

On GH200 and other coherent platforms this reflects HBM-resident allocations (the discrete-device counter) and carries unified=True, the probed signal that host RAM is a peer pool the residency policy can spend. Managed memory paged into Grace LPDDR is not counted here, matching nvidia-smi.

clocks property

Current SM and memory clock frequencies.

utilization property

GPU core and memory-controller utilization percentages; zeros if unsupported.

thermal property

Die temperature, thresholds, and throttle reasons; zeros where unsupported.

energy property

Current power draw and cumulative energy; zeros if unsupported.

pcie property

PCIe TX/RX throughput in KB/s; zeros on non-PCIe devices.

fan_speed_pct property

Fan speed as a percentage; 0 on fanless devices.

processes property

Running compute processes on this GPU.

is_available() classmethod

Whether CUDA reports at least one NVIDIA device.

all() classmethod

Return all CUDA-visible devices ordered by visible index.

nvml_memory()

Current memory state from NVML when cuda.core is unavailable.

runtime_memory()

Current memory state from CUDA Runtime when NVML memory is unsupported.

nvml_clocks()

Current clocks from NVML when cuda.core is unavailable.

Profiling

mainboard.Profiler(*, features=Feature.DEFAULT, activities=NativeActivity.DEFAULT, device_index=0, sample_interval_ms=50, max_spans=100000, auto=())

Collect selected evidence through one bounded profiling session.

span annotations stay dormant until this context is active. features controls what may be collected while the resulting Profile contains only evidence that was actually observed. Python sampling applies to run, attach, and dump.

under(collection) classmethod

Build a profiler from one collection policy.

The constructor takes the six choices flat because that is what a caller writing one line wants. Anything holding a policy already, a study most of all, should hand over the value rather than unpack it into six arguments and risk unpacking it differently next time.

stop_sampler()

Stop and release this session's optional device sampler.

enter(name)

Open one span and return the exact token later used to close it.

exit(token, wall_ns)

Close one span and fold its timing, device samples, and activity window.

target_snapshot(name)

Read one GPU snapshot only when it contains this process.

sample()

Poll target-process device telemetry while at least one span is open.

auto(modules)

Enable local sys.monitoring events only for code owned by modules.

module_codes(modules) staticmethod

Find owned module and nested code objects for local PEP 669 events.

owned_codes(module) staticmethod

Return function code owned by one module, including its class methods.

result()

Freeze the evidence collected so far into one Profile.

stats()

Return per-span aggregates for the current session.

bottlenecks(top=10)

Return the slowest span paths in the current session.

trace_report(top=10)

Return GPU activity attributed to span windows.

report()

Render the current result as plain text.

show(*, color=True)

Print the current result.

measure(reach, *, collection=None, sampler=None, strict=False) classmethod

Measure whatever reach names, under collection, driving sampler.

One method for all three ways of reaching a target, because which one applies is a property of the Reach rather than a choice of function. run and attach remain as the two shorthands a caller writes by hand. Reach.here() names the calling process, which has no target for this classmethod to launch, so it raises rather than launching an empty target: that measurement is what with Profiler(...) as profiler: is for.

run(target, *, module=None, args=(), features=Feature.DEFAULT, activities=NativeActivity.DEFAULT, sampler=None, timeout=None, strict=False) classmethod

Run one target once and collect every selected capability that works.

how to drive the external Python sampler, as the model that already describes

it. Its executable is also the interpreter the target runs under, so the two cannot drift apart the way two separate arguments could.

run_instrumented(target, *, tachyon, executable, features, activities, timeout) classmethod

Run the target once with local collectors and an optional Tachyon parent.

attach(pid, *, sampler=None, timeout=None) staticmethod

Attach Python sampling to one live process.

how to drive the external sampler. Tachyon already models every one of those

choices, so this takes the model rather than taking its fields loose and rebuilding it, which is what let two call sites disagree about the same policy.

dump(pid, *, all_threads=True, async_aware=False, executable=sys.executable, timeout=10.0) staticmethod

Return one sampled Python stack snapshot from a live process.

mainboard.Profile

Bases: FrozenModel

One immutable result containing only evidence that was observed.

Python samples, span timings, process GPU telemetry, native activities, and replayed counters are independently optional. A detected but unused GPU never creates output.

stats()

Per-name aggregates (calls/total/avg/peak), slowest total first.

bottlenecks(top=10)

The slowest region names by total wall time.

trace_report(top=10)

Deep GPU-time ranking (compute/copy split, hot regions and kernels).

efficiency(*, sm_count, peak_bandwidth_gbs=0.0, bytes_moved=0, blocks_per_sm=1, top=12)

Per-kernel launch shape, wave quantisation and achieved bandwidth.

A duration ranking says which kernel is slow and the timeline says whether the device was idle. Neither exposes a grid that leaves most block slots empty in its final wave, which reads as busy while draining a handful of blocks.

timeline(top_gaps=10)

Busy/idle accounting over observed activity, with the longest idle windows.

A kernel ranking says which kernel costs most; this says whether the device was working at all. A pipeline that reads kernel-bound is often idle between launches.

counter_bottlenecks(top=10)

Return the hottest demangled kernels from the counter pass.

diff(baseline)

Compare regions and demangled kernels against a baseline profile.

save(path)

Persist to JSON so a later run can :meth:load and :meth:diff it.

load(path) classmethod

Load a profile saved by :meth:save.

perfetto(path)

Write a Perfetto/Chrome timeline (open at ui.perfetto.dev).

show(*, color=True)

Print a rich table of the region stats (and the deep report if traced).

report()

A plain-text report containing only populated evidence sections.

mainboard.span(name)

span(name: Callable[P, Awaitable[R]]) -> Callable[P, Awaitable[R]]

span(name: Callable[P, R]) -> Callable[P, R]

span(name: str) -> _Span

Mark a named block or function for an active Profiler.

The annotation contains no collection policy. Without an active profiler it performs no clock, memory, marker, device, or context-variable work.

mainboard.profile(fn, *, iters=1, warmup=0, sync=None, kinds=Activity.DEFAULT, device_index=0)

Run fn under the profiler and return its bottleneck report.

fn: the zero-arg callable to profile (bind args with a lambda/partial). iters: timed runs of fn bracketed in one trace pass; warmup: untimed runs first. sync: a device barrier after each run (e.g. torch.cuda.synchronize) so async GPU work is captured rather than just the launch. kinds: the :class:Activity kinds to request; adapted down to what the device supports, with the dropped kinds recorded in :attr:ProfileReport.unavailable. device_index: which GPU from GPU.all() to profile and score against its peak.

mainboard.ProfileReport

Bases: FrozenModel

Structured bottleneck verdict for one profiled callable.

dominant_kernel/dominant_share_pct: the hottest kernel and its slice of kernel time. bound: the memory-vs-compute verdict (:class:Bound). total_kernel_ns/total_memcpy_ns: summed GPU time per class. achieved_bandwidth_gbps/peak_bandwidth_gbps: copy bandwidth measured against the device peak (the memory-bound signal). peak_memory_bytes/ avg_memory_bytes: the device-memory high-water mark and mean over the sampled run — the answer to "how much HBM did this kernel need". kernels: the per-kernel breakdown, hottest first. unavailable: activity-kind labels the device could not trace.

from_profile(profile, *, iterations, peak_bandwidth_gbps, supported=None, requested=None) classmethod

Distill a :class:Profile into a bottleneck verdict.

peak_bandwidth_gbps: device peak, to score copy bandwidth (0 disables the score). supported/requested: the :class:Activity kinds the device offered and the run asked for; their difference becomes unavailable so a partial trace is visible.

report()

A compact plain-text verdict and per-kernel table.

mainboard.KernelStat

Bases: FrozenModel

One kernel name's aggregate over the run: its share and representative shape.

The shape fields come from the last-seen launch of this name (kernels of one name share a launch config), so the report can show occupancy/registers/shared without a per-call row explosion. occupancy_pct is a launch-shape proxy: threads-per-block over the hardware max (1024), since the base CUPTI activity record carries the launch config but not achieved occupancy.

mainboard.Bound

Bases: Enum

Whether the dominant work is limited by memory traffic or compute throughput.

MEMORY when copies dominate the GPU time or the memory controller is the busier unit; COMPUTE when kernel math dominates; UNKNOWN when there was nothing to classify (no kernels, no copies, no utilization signal).

mainboard.gpu_busy(index=0, *, util_threshold=10, memory_threshold_pct=90.0)

Whether GPU index is under load right now (someone else is using it).

Busy means compute utilization above util_threshold percent or memory above memory_threshold_pct of capacity. Returns False when no GPU is present, so a CPU-only host always reads as idle.

mainboard.wait_for_idle(index=0, *, timeout=30.0, poll_interval=0.5, util_threshold=10, memory_threshold_pct=90.0, sleep=time.sleep)

Block until GPU index is idle, returning whether it became idle in timeout.

Polls :func:gpu_busy every poll_interval seconds. Returns True the moment the device is idle (immediately if it already is), or False once timeout seconds elapse while still busy — so a caller can decide to profile anyway or abort. sleep: the wait primitive, injected so tests need not spend real time.

Terminal view

mainboard.MachineView(machine=None)

Simple Rich schematic for the current machine.

print(*, color=True)

Render the machine schematic to the terminal.

renderable()

Return a compact schematic with connected hardware cells.

schematic_grid()

Render memory on the left and detected units on the right.

connected_units()

Render detected unit cells with arrows from memory.

detected_unit_cells()

Return only units that actually exist on this machine.

cpu_cell()

Render the CPU cell.

cpu_rows()

Return compact CPU identity and capacity rows.

gpu_cell(index)

Render the GPU cell.

gpu_rows(gpu)

Return provider-aware GPU rows.

memory_cell()

Render the system memory cell.

npu_cell(index)

Render the NPU cell.

npu_rows(npu)

Return provider-aware NPU rows.

memory_title()

Return the memory cell title.

gpu_title(index)

Return a compact GPU title.

npu_title(index)

Return a compact NPU title.

cell(title, border_style, rows)

Render one schematic cell.

arrow_to(label, connector)

Render one arrow from memory to a detected unit.

distinct_memory(unit)

Return memory only when it is distinct from the shared system pool.

memory_usage(memory)

Return one memory reading in human units.

fabric_label()

Return the dominant host-to-accelerator fabric label.

shared_memory_units()

Return unit names sharing or using the memory cell.

swap_label()

Return compact swap usage.

gpu_clock_label(gpu)

Return compact NVIDIA clock info.

metal_label(value)

Return a human label for macOS Metal support identifiers.

short_fabric_label()

Return a short fabric label for the connector.

connection_label(kind)

Return the memory connection label for one unit kind.

compact_summary()

Return one short summary line for the schematic.

bytes(value)

Format bytes as a compact binary unit string.

capacity_pair(used_bytes, total_bytes)

Format used and total bytes with a shared unit when possible.