CatNIP 原生 BF16 权重分布与平滑度

Operational verdict

先看结论，但不把“平滑”混成一个数

主结论只针对二维 learned weight matrices；norm、bias、调制表仍完整收录，但不会与投影矩阵混算。

在 1,660 个二维 learned weight matrices 上，0.5% 的双轴最大 RMS CV ≤ 0.10，25.9% ≤ 0.20。
通道最大/中位 RMS 比的中位数为 2.635；26.9% 的矩阵不超过 2×。
采样分布的 excess kurtosis 中位数为 1.632，6×MAD outlier 比例中位数为 0.026%。
通道尺度最不均匀的是 transformer_blocks.47.attn2.to_out.0.weight，双轴最大/中位 RMS 比为 513.425。
尾部最重的是 transformer_blocks.42.attn2.to_out.0.weight，excess kurtosis=25120.498，6×MAD outlier=0.314%。

阈值敏感性

双轴最大 RMS CV

≤ 0.050.3%

≤ 0.10.5%

≤ 0.225.9%

≤ 0.356.6%

双轴最大/中位 RMS 比

≤ 1.250.4%

≤ 1.55.5%

≤ 226.9%

≤ 478.8%

All blocks × all modules

48 × 86 全模块总览

横轴是 block 内完整 tensor signature，纵轴是 block 0→47。灰色表示该指标不适用，例如一维 bias 没有双轴通道 CV。悬停看精确位置，点击进入单 tensor。

MetricFamily

low ← metric value → high；每种 metric 使用当前可见范围的独立色标

把鼠标移到图上。

点击像素后会跳到相应 block/tensor 的权重点阵、直方图和通道 RMS。

Tensor drill-down

一个具体 block 的一个具体权重

热力图是原始 BF16 权重的确定性均匀点阵；同一 signature 的 48 blocks 共用 99.5% 色标。直方图先按 tensor 自身均值/std 标准化，再与标准高斯对照。

Representative caseScopeTensorBlock trend metric

BF16 weight lattice · output row × input column

hover 查看原始坐标和值

Standardized value histogramteal=data · amber=N(0,1)

x=(w−mean)/std；首尾 bin 含 |z|≥8 的 overflow。

Output-channel RMS

覆盖全部输出通道；每个 RMS 使用均匀采样的输入通道。

Input-channel RMS

覆盖全部输入通道；每个 RMS 使用均匀采样的输出通道。

Same tensor signature across 48 blocks

Module families

按模块家族汇总

每个矩阵等权取中位数，避免最大 FFN 因参数量大而掩盖小模块。

Family	Matrices	Parameters	Output CV	Input CV	Max/median RMS	Excess kurtosis	6×MAD outlier
Video self-attention	240	3.228B	0.217	0.206	2.251	2.043	0.037%
Video text-attention	240	3.228B	0.343	0.258	4.376	4.117	0.191%
Audio self-attention	240	0.808B	0.175	0.182	2.239	1.674	0.023%
Audio text-attention	240	0.808B	0.186	0.340	2.959	1.572	0.029%
Audio → Video attention	240	1.214B	0.183	0.191	2.140	1.155	0.015%
Video → Audio attention	240	1.211B	0.184	0.228	2.403	1.369	0.017%
Video FFN	96	6.442B	0.209	0.175	2.853	1.886	0.030%
Audio FFN	96	1.611B	0.155	0.168	2.538	1.261	0.019%
Global video projections / conditioning	8	0.221B	0.203	0.170	8.428	14.421	0.832%
Global audio projections / conditioning	8	0.056B	0.401	0.233	5.648	4.292	0.349%
Global cross-modal conditioning	12	0.150B	0.091	0.174	1.784	0.964	0.009%

Exceptions, not averages

最值得点开的异常模块

分别按通道尺度比、重尾峰度和 robust outlier 排序；同一 tensor 可能因不同原因重复出现。

Reason	Tensor	Family	Shape	Score
channel ratio	transformer_blocks.47.attn2.to_out.0.weight	Video text-attention	4096×4096	513.42529
channel ratio	transformer_blocks.47.attn2.to_v.weight	Video text-attention	4096×4096	46.28002
channel ratio	transformer_blocks.37.attn2.to_out.0.weight	Video text-attention	4096×4096	39.97095
channel ratio	transformer_blocks.39.attn2.to_out.0.weight	Video text-attention	4096×4096	34.18536
channel ratio	transformer_blocks.36.attn2.to_out.0.weight	Video text-attention	4096×4096	33.55065
channel ratio	transformer_blocks.40.attn2.to_out.0.weight	Video text-attention	4096×4096	31.10374
channel ratio	transformer_blocks.38.attn2.to_out.0.weight	Video text-attention	4096×4096	30.64948
channel ratio	transformer_blocks.0.audio_attn2.to_out.0.weight	Audio text-attention	2048×2048	30.16899
channel ratio	transformer_blocks.42.attn2.to_out.0.weight	Video text-attention	4096×4096	27.98151
channel ratio	transformer_blocks.34.attn2.to_out.0.weight	Video text-attention	4096×4096	27.79010
excess kurtosis	transformer_blocks.42.attn2.to_out.0.weight	Video text-attention	4096×4096	2.51e+04
excess kurtosis	transformer_blocks.46.attn2.to_out.0.weight	Video text-attention	4096×4096	8638.51660
excess kurtosis	transformer_blocks.39.attn2.to_out.0.weight	Video text-attention	4096×4096	5720.93066
excess kurtosis	transformer_blocks.34.attn2.to_out.0.weight	Video text-attention	4096×4096	3025.31543
excess kurtosis	audio_adaln_single.emb.timestep_embedder.linear_2.weight	Global audio projections / conditioning	2048×2048	1686.86316
excess kurtosis	adaln_single.emb.timestep_embedder.linear_2.weight	Global video projections / conditioning	4096×4096	1376.09326
excess kurtosis	transformer_blocks.45.attn2.to_out.0.weight	Video text-attention	4096×4096	1201.89893
excess kurtosis	transformer_blocks.43.attn2.to_out.0.weight	Video text-attention	4096×4096	907.52014
excess kurtosis	transformer_blocks.36.attn2.to_out.0.weight	Video text-attention	4096×4096	890.83209
excess kurtosis	transformer_blocks.32.attn2.to_out.0.weight	Video text-attention	4096×4096	679.66870
6×MAD outlier	transformer_blocks.47.attn2.to_out.0.weight	Video text-attention	4096×4096	0.17815
6×MAD outlier	transformer_blocks.47.attn2.to_gate_logits.weight	Video text-attention	32×4096	0.11670
6×MAD outlier	transformer_blocks.4.ff.net.2.weight	Video FFN	4096×16384	0.11104
6×MAD outlier	transformer_blocks.5.ff.net.2.weight	Video FFN	4096×16384	0.10274
6×MAD outlier	transformer_blocks.3.ff.net.2.weight	Video FFN	4096×16384	0.09251
6×MAD outlier	transformer_blocks.6.ff.net.2.weight	Video FFN	4096×16384	0.08923
6×MAD outlier	transformer_blocks.7.ff.net.2.weight	Video FFN	4096×16384	0.08263
6×MAD outlier	transformer_blocks.47.attn2.to_v.weight	Video text-attention	4096×4096	0.08014
6×MAD outlier	transformer_blocks.9.ff.net.2.weight	Video FFN	4096×16384	0.06728
6×MAD outlier	patchify_proj.weight	Global video projections / conditioning	4096×128	0.06686

Method & limits

这里的“平滑”具体指什么

原生 dtype 与覆盖
safetensors header 精确验证 dtype/shape/numel；4,186 个 tensor 全覆盖，未加载模型、未反序列化到 GPU、未修改服务。

分布统计
一维 tensor 使用全部值；二维 tensor 使用每轴最多 256 个等距位置的 Cartesian grid。均值、分位数、MAD、偏度、峰度均为该确定性样本统计。

通道尺度
每个输出通道都计算 RMS，但输入轴最多采样 256 点；输入通道同理。因此通道覆盖完整，单通道 RMS 是近似值。

不要误读热力图
矩阵通道顺序没有天然空间邻接语义；“视觉平滑”只能辅助发现条纹/块结构。是否适合量化主要应看通道 CV、尺度比与尾部分布。

Operational thresholds
CV≤0.10/0.20、max/median≤2× 只是可复核的描述阈值，不是模型质量或量化可行性的充分条件。

与 activation 的区别
这份报告只分析静态 weight。即使 weight 平滑，也不能推出 Q/K/activation 平滑；后者仍需使用推理 trace 单独判断。

完整 manifest

{
  "schema_version": 1,
  "created_utc": "2026-08-04T08:46:43Z",
  "experiment": "Production CatNIP merged-checkpoint native BF16 weight smoothness analysis",
  "checkpoint_requested": "/workspace/wuhan_inference/models/rl16/merged_sst1500_rl16.safetensors",
  "checkpoint_resolved": "/workspace/wuhan_inference/models/merged_sst1500_rl16.safetensors",
  "checkpoint_size_bytes": 37978195728,
  "safetensors_header_bytes": 519432,
  "safetensors_header_sha256": "1997b4da18c9aa988d826ef37e05b2b84df8141971bc5ededfa8503dafa56b91",
  "tensor_count": 4186,
  "parameter_count": 18988838144,
  "dtype_counts": {
    "BF16": 4186
  },
  "rank_counts": {
    "1": 2236,
    "2": 1950
  },
  "block_count": 48,
  "block_ids": [
    0,
    1,
    2,
    3,
    4,
    5,
    6,
    7,
    8,
    9,
    10,
    11,
    12,
    13,
    14,
    15,
    16,
    17,
    18,
    19,
    20,
    21,
    22,
    23,
    24,
    25,
    26,
    27,
    28,
    29,
    30,
    31,
    32,
    33,
    34,
    35,
    36,
    37,
    38,
    39,
    40,
    41,
    42,
    43,
    44,
    45,
    46,
    47
  ],
  "block_tensor_signatures": 86,
  "block_signature_presence_counts": {
    "48": 86
  },
  "global_tensor_count": 58,
  "sampling": {
    "axis_samples": 256,
    "heat_shape_cap": [
      64,
      96
    ],
    "trace_points": 256,
    "rank1_distribution": "exact",
    "rank2_distribution": "deterministic Cartesian grid",
    "channel_method": "all channels, sampled opposite axis",
    "histogram_z_range": [
      -8.0,
      8.0
    ],
    "histogram_bins": 81
  },
  "runtime": {
    "python": "3.12.13",
    "torch": "2.12.0+cu130",
    "cpu_threads": 8,
    "cuda_used": false,
    "pid": 2021313
  },
  "wall_time_s": 122.38603186607361,
  "trace": "weight_trace.pt"
}

Exact representative validation

代表矩阵逐元素复核

下表对少量代表矩阵读取每一个原生 BF16 元素，并以 FP32 精确累加；用于检验 256×256 分布网格和对侧 256 点通道 RMS 是否改变定性结论。

Role	Tensor	Shape	Max RMS CV sampled → exact	Max/median RMS sampled → exact	Excess kurtosis sampled → exact
smoothest_block_matrix	transformer_blocks.39.audio_attn1.to_out.0.weight	2048×2048	0.112 → 0.104	1.436 → 1.484	0.494 → 0.529
typical_block_matrix	transformer_blocks.32.ff.net.2.weight	4096×16384	0.281 → 0.275	2.619 → 2.598	1.459 → 1.336
worst_channel_ratio	transformer_blocks.47.attn2.to_out.0.weight	4096×4096	2.251 → 1.969	513.425 → 165.373	325.949 → 2245.328
heaviest_sampled_tail	transformer_blocks.42.attn2.to_out.0.weight	4096×4096	0.695 → 0.455	27.982 → 10.383	2.51e+04 → 1750.065
block0_heaviest_tail	transformer_blocks.0.attn1.to_k.weight	4096×4096	1.038 → 0.839	17.349 → 9.715	611.313 → 201.113