How to Audit an Open-Weight Model Before Importing It

A teammate pastes a model ID into a loader. It looks like downloading one data file. In practice, the URL can represent three separate imports: model data, executable software, and a legal contract.

The shortest path from that page to a Python process often looks like this:

# Unsafe example: do not run before reviewing the pinned repository.
from transformers import AutoModelForCausalLM
 
model = AutoModelForCausalLM.from_pretrained(
    "publisher/model",
    trust_remote_code=True,
)

That call can resolve a moving branch, download several classes of artifact, and import repository-defined Python before you have reviewed the code. It doesn't install every missing dependency for you, but imported code can still reach the filesystem, network, native extensions, and whatever credentials the process can access.

Approving the weights does not approve the code or the contract. Each import needs its own owner and gate.

Treat a model release as a versioned supply-chain input: claims, files, executable code, dependencies, provenance—evidence of where and how the artifact was produced—and legal terms. Pin it, inspect it as data, then decide what may execute.

The commands below target public repositories and huggingface_hub 1.26.0. They were checked against primary documentation on 2026-08-03. Record your actual tool versions because command-line flags and supported numeric storage types change.

The workbench below is a guided intake interview, not a security scanner. Pick the release type and intended use; it points to the evidence and approval gate that should come next. Use it before fetching a large weight set.

After the workbench returns a path, write down the first blocked gate. If the answer depends on a moving branch, unreviewed code, or unclear license text, the audit has already found useful work. It has not declared the release safe.

“Open weight” describes availability, not freedom

Open weights has no single standard license meaning. It usually means that a publisher lets you obtain model parameters under stated terms.

The Open Source AI Definition 1.0 (OSAID) sets a higher bar for Open Source AI: permission to use, study, modify, and share, plus the preferred form for modification, including data information, code, and parameters under qualifying terms.

Keep these categories separate:

Kimi K3 demonstrates why the label matters. Its pinned weight license grants broad rights but adds a Model-as-a-Service revenue threshold and a large-product attribution condition. The publisher's model card calls the release open-weight. That is more precise than calling the release open source.

This is an engineering classification, not legal advice. The gate is simple: the deployment plan must match the approved text, not a badge or social post.

Reproducibility starts with an immutable repository identity

main is a moving reference. Tags can also be retargeted on hosting systems that permit it. Record the full commit hash, often called a SHA, that a requested branch or tag resolved to.

First, record the client version:

python3 - <<'PY'
from importlib.metadata import version
 
print("huggingface_hub", version("huggingface_hub"))
PY

Then resolve a public repository without downloading its weights:

export HF_HUB_DISABLE_IMPLICIT_TOKEN=1
export REPO=HuggingFaceTB/SmolLM2-1.7B-Instruct
export REQUESTED_REV=main
 
export REV="$(
  python3 - <<'PY'
import os
from huggingface_hub import HfApi
 
info = HfApi().model_info(
    repo_id=os.environ["REPO"],
    revision=os.environ["REQUESTED_REV"],
)
if not info.sha:
    raise SystemExit("Hub response did not include a commit SHA")
print(info.sha)
PY
)"
 
printf '%s requested=%s resolved=%s\n' \
  "$REPO" "$REQUESTED_REV" "$REV"

HF_HUB_DISABLE_IMPLICIT_TOKEN=1 prevents a locally stored token from being sent on these public read requests. Gated or private releases need a separate, explicitly approved authentication path.

The Hub download documentation for 1.26.0 requires a full-length hash when revision is a commit. Store:

• repository ID and repository type;
• requested branch or tag;
• resolved full commit hash;
• resolution timestamp;
• publisher or organization identity;
• public, gated, or private access state;
• client version and network environment.

A commit identifies repository state. It does not prove that the publisher's account was uncompromised, that large-file objects are present, or that any file is safe.

Inventory first; download only what the current gate needs

The hf command-line interface (CLI) can calculate a download plan without fetching the files:

hf download "$REPO" \
  --revision "$REV" \
  --dry-run

Fetch the files needed for a static review:

hf download "$REPO" \
  --revision "$REV" \
  --include "*.json" \
  --include "*.md" \
  --include "*.py" \
  --include "LICENSE*" \
  --include ".gitattributes" \
  --local-dir model-audit

Downloading Python source as bytes is not the same as importing it.

For a checkpoint, the saved model parameters and related state, that fits your approved storage and network budget, download the pinned tensor files, files containing multidimensional numeric arrays, only after the dry run:

hf download "$REPO" \
  --revision "$REV" \
  --include "*.safetensors" \
  --include "model.safetensors.index.json" \
  --local-dir model-audit

For a multi-terabyte release, do not turn “I inspected the metadata” into “I verified every tensor.” Use the pinned Hub tree and large-file metadata for the inventory, inspect only the approved byte ranges or local files, and leave full-checkpoint reconciliation below the series' E3 reproducibility grade until somebody actually performs it.

The first inventory answers:

Do not infer “safe” from .safetensors, “small” from active parameters, or “permitted” from the ability to click Download.

A small inventory can already block execution

Suppose a pinned repository contains 12 safetensors shards, ordinary config files, modeling_custom.py, an auto_map entry that selects that custom class, and a custom weight license. You do not need to run the model to make the first decisions:

This is the first useful result: metadata review may continue while execution and deployment remain blocked.

A model card is a claim index

The model-card paper and Hugging Face model-card guidance ask publishers to report intended uses, limitations, training, and evaluation. That structure is useful. The publisher still authored it.

This series grades each claim separately: E3 is a reproducible observation from pinned artifacts; E2 is pinned primary-source support; E1 is a publisher or maintainer claim without enough protocol to reproduce it; E0 is missing, mutable-only, or conflicting evidence.

Turn each material statement into a ledger. The example uses YAML, a human-readable structured-data format:

claim:
  subject: checkpoint
  predicate: total_parameters
  value: 1.7B
  claimant: pinned_model_card
  evidence_grade: E1
  stronger_evidence_needed: exact_tensor_header_count

Then seek the evidence that matches the claim:

• tensor names, shapes, data types (dtypes), and offsets for parameter counts;
• pinned config and reviewed architecture code for dimensions;
• reports, data documentation, or logs for training provenance;
• prompt, runner, judge, and raw outputs for evaluations;
• exact legal text for rights and obligations.

The useful question is not “Is this a good model card?” It is “Which statements can I reproduce, which have pinned primary support, and which remain publisher claims?”

Config files constrain interpretation but do not prove the checkpoint

Treat JavaScript Object Notation (JSON) config files as data before invoking AutoConfig, AutoTokenizer, or a custom processor:

for path in \
  model-audit/config.json \
  model-audit/tokenizer_config.json
do
  if [ -f "$path" ]; then
    python3 -m json.tool "$path" >/dev/null
  fi
done

The following standard-library script prints fields that often change the capacity or execution plan:

import json
from pathlib import Path
 
root = Path("model-audit")
 
for name in (
    "config.json",
    "tokenizer_config.json",
    "preprocessor_config.json",
    "generation_config.json",
):
    path = root / name
    if not path.exists():
        continue
 
    data = json.loads(path.read_text(encoding="utf-8"))
    print(name)
 
    for key in (
        "architectures",
        "model_type",
        "dtype",
        "torch_dtype",
        "auto_map",
        "quantization_config",
        "chat_template",
        "model_max_length",
        "max_position_embeddings",
    ):
        if key in data:
            print(" ", key, "=", data[key])

Config dimensions can expose layer count, hidden widths, attention and key/value (KV) head counts, which determine how many key/value streams each layer stores, expert routing, vocabulary, context settings, weight tying, and custom classes. They cannot prove that the expected tensors exist, that every tensor uses the top-level dtype, or that the artifact set is complete.

Safetensors removes pickle objects, not the rest of the trust boundary

Python's pickle documentation is blunt: unpickling untrusted data can execute arbitrary code.

The safetensors format at commit 6eb4dc9 stores an eight-byte little-endian header length, a JSON header, and a fully indexed tensor byte buffer. It does not serialize Python objects.

That narrows one risk. It does not authenticate the publisher, prove that the weights are benign, validate model behavior, inspect custom Python, pin dependencies, or secure the serving container.

Download the complete audit_safetensors.py script. It uses only the Python standard library, fails closed on unknown dtypes, rejects duplicate JSON keys, checks tensor bounds and complete buffer coverage, and reconciles an optional shard index. The excerpt below shows the cross-shard gate; the download contains the byte-level parser.

In plain language, it answers three questions: does each file have a valid header, do the declared tensors cover exactly the available bytes, and does the cross-file index name the same tensors and shards?

It also rejects an index file that is only a Git Large File Storage (LFS) pointer instead of the intended content.

for path in files:
    result = inspect_file(path)                 # ①
    for name, shard in result["tensors"].items():
        if name in actual:
            duplicates.append((name, actual[name], shard))
        actual[name] = shard
 
if duplicates:
    raise ValueError(f"duplicate tensors across shards: {duplicates}")
 
discrepancies = {                              # ②
    "missing_shards": missing_shards,
    "unmapped_files": unmapped_files,
    "missing_tensors": missing_tensors,
    "unindexed_tensors": unindexed_tensors,
    "misplaced_tensors": misplaced,
}
if any(discrepancies.values()):                 # ③
    raise ValueError(
        "checkpoint reconciliation failed: "
        + json.dumps(discrepancies, sort_keys=True)
    )

① The full script validates each file's header, dtype, shape, offsets, and complete byte coverage before adding it to the cross-shard ledger. ② Reconciliation names every mismatch instead of collapsing them into one boolean. ③ Any mismatch exits non-zero, so a CI gate cannot print an incomplete checkpoint and continue successfully.

Run it only after you have obtained the intended local files:

python3 audit_safetensors.py model-audit

The dtype allowlist is intentionally versioned. If a later format adds a dtype, update the map from the pinned upstream source rather than guessing its width. This helper is an independently reviewable ledger tool, not a replacement for the pinned upstream parser in a serving path.

A shard index maps names; large-file storage maps bytes

If you skipped the byte-level parser, resume here. The operational question is simple: does the index point to every tensor file you actually obtained, and do those files contain the bytes the index claims?

model.safetensors.index.json maps tensor names to shard filenames and often declares a total payload size. The script checks:

• missing and extra shard files;
• missing, extra, duplicate, and misplaced tensor names;
• per-file buffer coverage;
• declared versus parsed payload bytes.

A Git checkout can still contain a small Git LFS pointer instead of the large object. The Git LFS pointer specification at commit b996449 defines that pointer's object ID and size. The repository commit identifies the pointer text; the LFS object ID identifies the large object.

Hash the local files without loading them:

python3 - <<'PY'
import hashlib
from pathlib import Path
 
root = Path("model-audit")
 
for path in sorted(candidate for candidate in root.rglob("*") if candidate.is_file()):
    digest = hashlib.sha256()
    with path.open("rb") as handle:
        for chunk in iter(lambda: handle.read(1024 * 1024), b""):
            digest.update(chunk)
    print(digest.hexdigest(), path)
PY

A local hash proves which bytes you inspected. It does not establish a trusted expected value or publisher identity by itself.

Remote code and dependencies remain executable

Transformers' security policy at commit b3a3603 recommends safetensors, source review for trust_remote_code=True, and revision pinning.

Start with a broad source inventory:

find model-audit -type f \( \
  -name "*.py" -o \
  -name "*.so" -o \
  -name "*.dylib" -o \
  -name "*.dll" -o \
  -name "Dockerfile*" -o \
  -name "requirements*.txt" -o \
  -name "pyproject.toml" \
\) -print

Then triage Python source. This search finds review targets; matches are not proof of malicious behavior:

find model-audit -type f -name "*.py" -exec \
  grep -nHE \
  'importlib|__import__|eval\(|exec\(|subprocess|os\.system|socket|requests|urllib|ctypes|cpp_extension|load_library|pip|download|open\(' \
  {} +

Abstract syntax tree (AST) parsing inventories imports without importing the reviewed modules:

import ast
from pathlib import Path
 
for path in sorted(Path("model-audit").rglob("*.py")):
    source = path.read_text(encoding="utf-8")
    tree = ast.parse(source, filename=str(path))
    imports = []
 
    for node in ast.walk(tree):
        if isinstance(node, ast.Import):
            imports.extend(alias.name for alias in node.names)
        elif isinstance(node, ast.ImportFrom):
            imports.append(node.module or "")
 
    print(path, sorted(set(imports)))

Static review defines the next trust boundary. It cannot prove the code safe, especially when imports reach native libraries or dynamic compilation.

The environment belongs in the audit too:

• Python and model-runtime versions;
• exact package lock with local hashes;
• wheel versus source-distribution policy: install a prebuilt Python package or build one locally;
• CUDA or ROCm, driver, compiler, and communication-library versions;
• container image digest and base-image provenance;
• native extension sources and build flags.

Pip's secure-install guidance explains that --require-hashes requires every dependency to be pinned and hashed. --only-binary=:all: avoids source-distribution builds. Those controls verify selected distribution files; they do not make installed code trustworthy when it runs.

Tokenizer and processor assets are part of model behavior

A checkpoint can load successfully and still produce bad or unsafe behavior because the application formats messages incorrectly.

Inspect:

• tokenizer and processor classes;
• vocabulary and merge files;
• beginning-of-sequence (BOS), end-of-sequence (EOS), padding (PAD), and unknown-token IDs;
• chat template and generation-prompt behavior;
• media, reasoning, and tool-control tokens;
• stop IDs and output limits;
• custom tokenizer or processor Python.

The Transformers chat-template documentation warns that applying a chat template and then adding special tokens again can duplicate BOS or EOS markers.

A declarative Jinja template is text interpreted by a known template renderer. Repository-defined tokenizer or processor Python is executable code. They require different gates.

Low-bit storage needs targets, scales, and runtime behavior

“Four-bit” is not a byte count.

Record:

• integer or floating-point encoding;
• packing layout and group size;
• scale and zero-point dtypes;
• target and excluded modules;
• activation and KV-cache precision;
• runtime kernel and version;
• whether loading expands, repacks, or duplicates tensors;
• temporary workspace during conversion or graph capture.

The forthcoming Kimi K3 checkpoint case study shows the failure mode: packed low-bit values, scales, higher-precision exceptions, and repository overhead do not equal parameters × 0.5 bytes.

Keep three quantities separate:

download bytes
resident engine representation
peak = weights + cache + activations + workspaces + communication + reserve

The final quantity usually targets high-bandwidth memory (HBM), the local memory on a graphics processing unit (GPU) available to the serving process.

Provenance, signatures, hashes, and scans answer different questions

Supply-chain evidence should not collapse into one green badge.

Provenance is verifiable information about where, when, how, and from which inputs an artifact was produced.

The Supply-chain Levels for Software Artifacts (SLSA) provenance 1.1 specification defines attestations that bind output subjects to a builder, build definition, external parameters, and resolved dependencies. Model publishers do not always provide equivalent training or packaging provenance. Missing provenance should remain visible in the ledger rather than being inferred from a repository history.

Legal review separates at least five scopes

Read exact documents, including effective dates and incorporated policies:

An API entitlement does not grant rights to downloaded weights. A weight license does not govern the publisher's hosted service. A permissive code license does not automatically cover parameters or training data.

Benchmark rows need a protocol, not a logo

Before repeating a benchmark comparison, record:

1. exact model revision or dated API version;
2. dataset revision and split;
3. prompt, chat template, and few-shot examples;
4. sampling, seed, and output limit;
5. reasoning mode, tools, retrieval, browser, or code execution;
6. evaluation-runner revision;
7. judge model, judge version, and rubric;
8. precision, quantization, and runtime;
9. hardware, topology, batch or concurrency, and timeout policy;
10. run count, variance, refusals, and failed-run handling;
11. raw outputs or enough retained evidence to reproduce the score.

Rows that use different tools, judges, context limits, or failure policies do not become comparable because they share a table.

If the protocol is incomplete, retain the publisher's score as E1. Do not turn it into a product capacity or quality fact.

The deliverable is a set of gates, not a trust score

The completed audit contains:

• pinned release and publisher identity;
• file, object-ID, and hash inventory;
• exact tensor and shard observations where available;
• tokenizer and prompt-format inventory;
• custom-code and native-extension trust boundary;
• dependency and container lock;
• quantization and runtime requirements;
• separate legal and policy documents;
• provenance and training-disclosure gaps;
• benchmark comparability ledger;
• disk, resident, and peak deployment estimates;
• unresolved conflicts and approved exceptions.

Do not average E3 tensor evidence with E0 training provenance. The mismatch is the result.

Failure patterns worth testing

• The model revision is pinned but tokenizer, processor, or evaluation code is not.
• A local checkout contains large-file pointers instead of objects.
• A top-level dtype is mistaken for every tensor's dtype.
• A header parser silently accepts a new sub-byte dtype with the wrong width.
• trust_remote_code=True appears before code review.
• A locked Python set still loads an unpinned container or native library.
• A signed commit is treated as proof that the content is safe.
• A hash is published without a trusted expected value.
• A license tag replaces exact legal text and incorporated policies.
• Active parameters size resident weights for a sparse model.
• On-disk bytes are presented as required HBM.
• One benchmark score hides different tools, judges, precision, or hardware.

The workshop passes when another engineer can reproduce each retained observation without importing repository code.

Do not ask whether you trust the model. Ask which bytes and claims you can prove, which code you have approved, and which unknowns block this use.

Workshop two applies the same discipline to side effects: a tool timeout is an unknown outcome until the application reconciles it. The course roadmap keeps that workflow boundary beside the artifact gates from this workshop.

Sources and references

Openness, disclosure, and legal scope

• Open Source AI Definition 1.0
• Model Cards for Model Reporting
• Hugging Face model-card documentation
• Pinned Kimi K3 license example

Repository and artifact inspection

• Hugging Face Hub downloads, version 1.26.0
• Hugging Face Hub environment variables, version 1.26.0
• Safetensors format snapshot
• Python pickle security warning
• Git LFS pointer specification snapshot
• Transformers security policy snapshot
• Transformers chat templates

Supply chain and dependencies

• SLSA provenance specification 1.1
• Pip secure installs
• Kimi K3 pinned artifact API