Docs / Validation & claims

Validation & claims

cufemlab labels every analysis with exactly what its result means and how far you can trust it. This page defines the validation tiers, lists which of the 15 live analysis types fall in each, and states precisely what "internally validated, external passport not yet attached" is - and is not.

The newest type, cufem3d_gpu_field_analysis, is experimental: GPU-accelerated 3-D field analysis is available on compatible GPU workers, and runtime metadata reports the actual execution backend and fallback state. Every result carries an analytic cross-check against the magnetized-sphere reference, and an optional CPU/GPU numerical parity check runs when run_reference_check is enabled (off by default); the type has no external validation passport, and it is not an independently validated 3-D torque prediction.

No comparison to commercial tools. cufemlab makes no accuracy or speed claim versus any commercial package (ANSYS, JMAG, COMSOL, or any other). Where a number is checked at all, it is checked against an analytic identity, an energy balance, an open independent FE tool, or a held-out slice of our own data. Nothing here is a benchmark against a proprietary competitor.

The six tiers

A result carries one of six tier labels. The tier tells you what the number was checked against and what an engineer is entitled to conclude from it.

Tier 1

Externally cross-checked

The result ships an independent oracle card: the same case was solved by a separate, external reference and the two agree within a stated tolerance. This is the strongest tier we offer.

Reference is an outside tool or a closed-form analytic solution, not cufem's own method.

Tier 2

Internally validated

The result is verified against the method's own analytic solution or a conservation identity (method of manufactured solutions, convergence order, energy balance). It is not yet accompanied by an external passport.

Correct by the physics it claims to solve; external oracle card pending.

Tier 3

Experimental surrogate

A learned model that predicts a field. It is an in-distribution surrogate and self-validating demo, scored each run against the real solver on the same input. It is not a general-purpose neural operator.

Accurate inside the training distribution; degrades out of distribution.

Tier 4

Experimental

A new execution path: every result carries an analytic cross-check against the magnetized-sphere reference, and a CPU/GPU numerical parity check runs when the optional run_reference_check parameter is enabled (off by default), but carrying no external validation passport. Results report validation_card.status = not_validated. Only cufem3d_gpu_field_analysis. It is the GPU execution lane for the same 3-D edge-element field solve as the CPU utility, and it is not an independently validated 3-D torque prediction. It is fail-closed: with no compatible GPU the job fails rather than silently running on CPU.

Analytic magnetized-sphere reference every run; optional CPU/GPU parity. No external passport.

Tier 5

Engineering estimate

A reduced-order or coefficient-based calculation intended for design-space ranking and early sizing, not for sign-off. Useful and repeatable, but not a validated field solution.

Directional. Treat the absolute value as an estimate.

Tier 6

Utility

A convenience or reporting action - comparisons, signed report generation, quick demo checks. These produce artifacts and summaries, not a physics verdict to be validated.

No accuracy claim; it is a tool, not a solver result.

Where the 15 live types fall

Fifteen public analysis types: two externally cross-checked, five internally validated, one experimental (the GPU 3-D lane), one experimental surrogate, two engineering estimates and four utilities. Two plus five plus one plus one plus two plus four is fifteen. The scope column below marks each type's tier.

Tier 1 - Externally cross-checked
  • cogging_torque_movingband - flagship; cross-checked against FEMM and GetDP
  • cufem3d_field_validation - 3-D field validated against an analytic sphere
Tier 2 - Internally validated (external passport not yet attached)
  • thermal_conduction_steady - MMS L2 order ~1.93; energy-balance relative 3.7e-14 vs 1e-6 tol
  • thermal_conduction_transient - theta-method; Crank-Nicolson order 2 confirmed
  • eddy_thermal_coupled - skin-effect Joule loss to thermal chain; spatial order 1.998
  • core_loss_estimate - Bertotti 3-term separation; k_c closed form to 0.05%
  • coreloss_thermal_coupled - core-loss density to thermal response; energy chain
Tier 3 - Experimental surrogate
  • field_to_field_neural_operator - in-distribution FNO surrogate / self-validating demo
Tier 4 - Experimental
  • cufem3d_gpu_field_analysis - GPU 3-D field lane. Analytic magnetized-sphere reference every run, plus an optional CPU/GPU parity check (run_reference_check, off by default), and no external validation passport: results report validation_card.status = not_validated. It is not an independently validated 3-D torque prediction, and it is fail-closed.
Tier 5 - Engineering estimate
  • pmsm_operating_torque
  • iron_loss_estimate
Tier 6 - Utility
  • material_comparison
  • signed_report_generation
  • demo_motor_quick_check
  • cogging_sweep_2d (legacy)

Standalone eddy-current solvers exist in the engine and worker, but on the customer API eddy physics reaches users only through eddy_thermal_coupled. They are not separately exposed as customer analysis types.

What "internally validated, external passport not yet attached" means

All five new physics types return result.v1 with validation_card.status = "not_validated" and claim_allowed_public = false. That wording is precise and deliberate. Read it exactly:

What it does mean

  • The solver was checked against its own governing physics: a manufactured analytic solution, a measured convergence order, or a conservation/energy identity, and it passed within tolerance.
  • The method is self-consistent and reproducible - deterministic runs, HMAC-signed evidence.
  • The numbers on this page (orders ~1.93, 1.998, 2; energy balance 3.7e-14) come from those internal checks.

What it does NOT mean

  • It is not accompanied by an external oracle card - no independent outside tool or third-party reference is bundled with the result yet.
  • It is not "certified" or "third-party validated". Those words never apply to a not_validated card.
  • claim_allowed_public = false means the platform will not let the result be published as an externally validated claim.

Internal card vs external oracle card

The difference is the source of truth the result is checked against:

Internal validation card (Tier 2, status not_validated)
  • Truth source = the method's own analytic solution or conservation identity.
  • Answers: "Does the solver solve the equation it says it solves, at the expected order?"
  • Bundled today with all five new physics types.
External oracle card (Tier 1)
  • Truth source = an independent outside tool (FEMM, GetDP) or a closed-form analytic reference.
  • Answers: "Does an unrelated reference agree with this result on the same case?"
  • Bundled today only with cogging_torque_movingband and cufem3d_field_validation.

An external passport for the new physics types is a planned follow-on. Until it is attached, the honest statement is: "validated internally against its analytic / energy identity; external validation passport not yet attached."

The experimental surrogate, stated plainly

field_to_field_neural_operator is a Fourier Neural Operator (FNO2d, 4 spectral layers). It is an in-distribution surrogate and self-validating demo. Each run draws an in-distribution conductivity field from an integer conductivity_seed, predicts the temperature field, and scores itself against the real solver on that same field (self_validation_rel_l2).

  • Held-out rel-L2 is 0.68% on the training distribution. Out of distribution it degrades to 5.8%.
  • It beats a linear baseline (9.8%) and a POD-DeepONet (11.8%) on that distribution.
  • It is not a general-purpose neural operator. Arbitrary custom-field or CAD input is a planned follow-on, not shipped.

Data-rights note: TEAM-7 and H(curl)

Internal validation of our H(curl) edge-element and eddy-current solver line exists, including a TEAM-7 style benchmark run inside the engine. However, third-party TEAM-7 mesh and reference data is not published or bundled with cufemlab. We do not ship, redistribute, or quote outside reference datasets we do not have the rights to distribute. The internal validation is ours; the external benchmark data behind it stays with its owners.

Reading a result card in the API

Every job result is result.v1. The validation_card is where the tier and status live. Fetch it and read it before you rely on a number.

Read the validation cardPython
import os
from cufemlab_client import Client

client = Client(api_key=os.environ["CUFEMLAB_API_KEY"])  # default base https://app.cufemlab.secrotec.nl
project = client.create_project(name="thermal-check")

job = client.analyze(
    project_id=project.id,
    analysis_type="thermal_conduction_steady",
    input_params={},          # sensible defaults for every parameter
    gpu=True,
)
job.wait(poll_interval=10)
result = job.result()

print(result.verdict)                       # pass / fail
print(result.value)                         # result_summary.value
# validation_card.status == "not_validated" for the five new physics types:
# internally validated, external passport not yet attached.

In-app Jupyter injects a per-session scoped key, so Client.from_env() works with zero paste. Direct HTTP: POST /api/v1/jobs {analysis_type, input_params, project_id} with a Bearer access token or X-API-Key; poll GET /api/v1/jobs/{id}; read GET /api/v1/jobs/{id}/result. Errors: 422 invalid domain params, 402 INSUFFICIENT_CREDITS, 429 CONCURRENCY_LIMIT_EXCEEDED (max 2 concurrent jobs per user).

Summary