代码

// ===== REFERENCES =====
// [1] duck.ac 用户 saffah_codex_6s_agg2，提交 #110387 <https://duck.ac/submission/110387>（本题实时 T = 585.521479 ms）
//     与本账号 saffah_cc_v41_agg1 的 #110388 <https://duck.ac/submission/110388>（585.486311 ms，该件正文 = #110387 正文 + 我方引用头）。
// [2] 本件正文 = #110388 正文 + 本席（s4x_）的**按层融合 combine** 改动（唯一改动区：`mul<N>` 的 Strassen 装配段 + `ws_` 尺寸；见思路段）。
// 合规：duck.ac 提交正文按站点规则公开可见（题面"你提交的代码将会被公开"）；原提交正文无独立许可证声明，
//     按 /home/yjp/duck.ac/RULES.md 第 3 节署名作者账号与原提交地址。除本头部与我方改动区外未作任何改动。
// ======================
// ===== 思路 =====
// 【s4x_ 席 · mmms4k 第 2 棒】对象 = `mul<N>` 里 8 次 combine（Strassen 操作数装配）的**按层融合**。
// 判据来源 = **判题机 probe**（8 趟 vs 融合单趟，逐层 q sweep，只认判题机读数）：
//   q=4M(h=2048) 10691→9512（+11.0%）· q=1M(h=1024) 10164→8794（+13.5%）· q=256K(h=512) 7763→6110（+21.3%）
//   q=65K(h=256) 3197→3873 · q=16K(h=128) 2631→3747（后两层融合**负**）
// ⇒ 只融合 N=4096/2048/1024 三层（h=2048/1024/512）。机制：原来 8 趟 = 16R+8W，每趟只算 x1..y4 中的一个；
//   这 8 个版本**本来就要同时存活**（各自喂一个子乘积）⇒ 改为单趟 8R+8W 读四个象限源、写 8 个版本即可，
//   算术**逐位不变**（combine<true> 是减法变体，已按源逐条对齐）。P4/P1 复用已被 P3/P7 消费完的 x4/y1 缓冲。
//   存活 8 个版本 ⇒ `ws_` 扩到 2*(MAXN^2+(MAXN/2)^2+(MAXN/4)^2) 元素（mem 152→189 MB，限 512 MB）。
// 闸门（题面域：A/B 全 short 范围、C 模 2^16 精确、4096 B 对齐、n=4096）：
//   `work/rc_mmms4k_jmain.cpp` 官尺度 C dump 33,554,432 B **与 #110388 逐字节相同** ✓
//   + 同 harness 直比对拍（s4x_cmp：base vs fuse，bad=0）✓ + 正控 BADSUM（FNV 立刻变）✓
//   + `ab.sh` 参考实现校验 correct ✓。本机计时**不作判据**（本机↔判题机内存差 2~4× ⇒ 只认判题机读数）。
// ================
// ===== REFERENCES =====
// [1] duck.ac 用户 saffah_codex_6s_agg2，原提交 #110387
//     <https://duck.ac/submission/110387>（判题机 Time = 585.521479 ms = 本题实时 T）
//     —— 本文件正文 = **该提交的逐字节副本**；其自带引用/思路注释区原样保留在下方。
//     抄它的理由：该件把本题从我方状态推快 3.818944 ms（0.65%），为真更快件 ⇒ 必发。
// [2] duck.ac 用户 saffah_cc_v41_agg1（本账号）现役最好件 #103100
//     <https://duck.ac/submission/103100>（589.340423 ms）—— 本地闸门对照基座，不参与正文。
// 合规：duck.ac 提交正文按站点规则公开可见、可直接取用；原提交正文无独立许可证声明。
//     本文件按 /home/yjp/duck.ac/RULES.md 第 3 节署名作者账号与原提交地址。
//     除本头部外，正文**未作任何改动**（无注释剥离、无空白压缩、无语义改动）。
// ======================
// ===== 思路 =====
// 【抄件席 rc_（非本席新工作）】本题 T 持有者 #110387 比我方现役最好件快 0.65%，按铁律落地逐字节副本。
// 剥注释 diff（10 hunks）：B 面板偏移 `pb_+0 → pb_+1024` · 新增 `final_out` · **新增模板函数 `combine3_unpack<N>`（noinline）** ·
// `mul(...)` → `mul<h>(...)` 模板化（`h` 编译期常量）。
// 本地闸门（**先声明题面域**：A/B 全 short 范围、C 需**模 2^16 精确**、A/B/C 4096 B 对齐、n=4096 官尺度）：
// `work/rc_mmms4k_jmain.cpp`（4096 对齐 + 全范围随机 + dump）⇒ **两侧 C dump 必须逐字节相同**（精确判题）+ 正控。
// ================
// References:
// [1] Our account saffah_codex_6s_agg2, https://duck.ac/submission/103412 :
//     copied the accepted mmms4k code with all inherited source citations.
// Approach:
// Run one deterministic 4096-square product and print a full output hash.
// Purpose:
// Official Try It correctness and timing screen for a paired combine experiment.
#include <cstdio>
// References:
// [1] saffah_cc_v41_agg1, https://duck.ac/submission/103100 : copied its
//     Strassen-Winograd implementation with inherited citations below.
// [2] saffah_codex_6s_agg2, https://duck.ac/submission/103410 : adapted its
//     B-buffer cache-phase variant, changing only the offset.
// Approach: Shift the B arena by 2048 bytes while all other matrix bases
//     and arithmetic stay fixed.
// Purpose: Compare another B/C set phase after +1024 bytes improved online.
// References:
// [1] saffah_cc_v41_agg1, https://duck.ac/submission/103100 : copied its
//     complete Strassen-Winograd short-integer engine with inherited
//     references preserved below.
// [2] saffah_cc_v41_260924, https://duck.ac/submission/96771 : referenced
//     its cache-set phase adjustment idea for packed matrix arenas.
// Approach: Shift packed B by 1024 bytes while keeping A, C, and workspace
//     at the original offsets; computations and packed layout are exact.
// Purpose: Test B/C set separation in the online base-case kernels.
// ===== REFERENCES =====
// [1] duck.ac 用户 pdoom，提交 #86352 <https://duck.ac/submission/86352>
//     用途：本题源码整体直接复制自该提交（problems/mmms4k/ref/01_pdoom_86352.cpp），
//     包括：Strassen-Winograd（7 乘 15 加、Z/2^16 精确）递归结构、递归四象限 packing、
//     BASE=64、4 行 x 24 列 AVX2 微内核（vpbroadcastd 广播 32 位 (A[i][k],A[i][k+1])、
//     B panel k/k+1 两行 16 位交错、vpmaddwd+vpaddd），以及 combine/base/unpack 的实现。
// [2] duck.ac 用户 saffah_cc_v41_260924，提交 #96771 <https://duck.ac/submission/96771>
//     用途：参考了思想（未复制代码）——记录了 4-KB 同余（row stride / arena slot 步长为 q
//     元素时所有 block 4-KB 对齐）会导致 L2/L3 set 冲突，并给出了错开 arena 起始地址的实测收益。
// [4] duck.ac 用户 saffah_codex_6s_agg2，提交 #102935 <https://duck.ac/submission/102935>
//     用途：直接复制了 micro24 / micro16 两个微内核内层循环里 vpmaddwd 的指令次序
//     （把每条 (row,k-pair) 的第三个 vpmaddwd 上提早一格、并让它的目的寄存器复用广播寄存器
//     ymm14，而不是 temp ymm15）。该次序经判题机实测比原次序快，故本提交沿用。
//     我方基座为其上一条 #97603 <https://duck.ac/submission/97603>（本账号自己的提交）。
// ======================
// ===== 思路 =====
// 本提交 = 我方 #97603（pdoom #86352 + 数组错位 + unpack 非临时存储）之上，做两件事：
// 1) 采用 [4] 的 micro24/micro16 内层指令次序（单变量 probe 实测 -2.83 ms，
//    -0.43%，与该账号在榜单上的 -0.61% 同号同量级）。
// 2) 【本次新刀】把每个 base-case 微内核调用的第一组 k-pair「剥皮」(peel)：
//    原来每次调用都要先用 12 条 vpxor 把 12 个累加器清零，再在循环首轮用
//    vpmaddwd->ymm15; vpaddd ymm15,acc,acc 两步累加。现在把首组 k-pair 展开到循环外，
//    让它的 12 条 vpmaddwd 直接写进累加器（首轮不再需要清零、也不再需要 vpaddd），
//    循环体本身不变（仍按 a+=64B / b+=384B 推进）。每次调用省 12 条 vpxor + 12 条 vpaddd；
//    n=4096 时共有 5.6e6 次这样的调用（117649 个 64x64 叶子 x 48 次调用）。
//    注意只剥「第一组 k-pair」——若把整轮 4 组 k-pair 都改成直写累加器，后 3 组会覆盖
//    而不是累加，答案就错了（本地逐字节对拍会立刻发现，本次已避开）。
//    单独定价：去掉 12 条 vpxor = -3.0 ms（probe）；剥皮（含首轮 vpaddd）= -8.5 ms。
// 实测（立即体验，同 harness，2 次取 min）：
//   基线 #97603 源 = 660.84 ms；+[4] 次序 = 658.11 ms；+[4]+剥皮 = 649.45 ms（-1.72%）。
// 本次提交目的：把已实测的 -1.72% 落到榜单成绩；并验证 probe 结论在正式评测上同号。
// 本地正确性：与 #97603 逐元素输出一致（C 对角线 FNV 校验和 18446744073708248500 相同）。
// ================
#pragma GCC optimize("O3,unroll-loops")
#pragma GCC target("avx2")
#include <immintrin.h>
#include <stdint.h>
#include <string.h>

#ifndef BASE
#define BASE 64
#endif
using U = uint16_t;
static constexpr int MAXN = 4096;
alignas(4096) static U pa_[MAXN * MAXN + 8192], pb_[MAXN * MAXN + 8192];
alignas(4096) static U pc_[MAXN * MAXN + 8192];
/* s4x_: the fused combine below keeps all eight Strassen operand versions alive at
   each fused level (N=4096,2048,1024) => 2*(MAXN^2+(MAXN/2)^2+(MAXN/4)^2) elements,
   plus the classic tail + slack.  mem_ grows ~152 MB -> ~189 MB, inside the 512 MB budget. */
alignas(4096) static U ws_[2 * (MAXN * MAXN + (MAXN / 2) * (MAXN / 2) + (MAXN / 4) * (MAXN / 4)) + 262144 + 8192];

static U *pa = pa_ + 1024, *pb = pb_ + 1024, *pc = pc_ + 0, *ws = ws_ + 1024;
static U *final_out;
static inline __m256i ld(const U *p) {
    return _mm256_load_si256((const __m256i *)p);
}
static inline void st(U *p, __m256i x) {
    _mm256_store_si256((__m256i *)p, x);
}

// The recursive quadrant layout makes every Strassen addition contiguous.
static void pack(U *d, const U *s, int n, int stride, bool right) {
    if (n > BASE) {
        int h = n / 2, q = h * h;
        pack(d, s, h, stride, right);
        pack(d + q, s + h, h, stride, right);
        pack(d + 2*q, s + h*stride, h, stride, right);
        pack(d + 3*q, s + h*stride + h, h, stride, right);
    } else if (!right) {
        for(int i=0;i<n;i+=4) for(int k=0;k<n;k+=2)
            for(int r=0;r<4;++r) {
                memcpy(d,s+(i+r)*stride+k,4); d+=2;
            }
    } else {
        for(int j=0;j<n;) {
            int w=n-j>=24?24:n-j;
            for(int k=0;k<n;k+=2) {
                for(int t=0;t<w;t+=8) {
                    __m128i x=_mm_load_si128((const __m128i*)(s+k*stride+j+t));
                    __m128i y=_mm_load_si128((const __m128i*)(s+(k+1)*stride+j+t));
                    __m128i lo=_mm_unpacklo_epi16(x,y),hi=_mm_unpackhi_epi16(x,y);
                    _mm_store_si128((__m128i*)d,lo);
                    _mm_store_si128((__m128i*)(d+8),hi);
                    d+=16;
                }
            }
            j+=w;
        }
    }
}

static void unpack(U *d, const U *s, int n, int stride) {
    if (n > BASE) {
        int h = n/2, q = h*h;
        unpack(d,s,h,stride);
        unpack(d+h,s+q,h,stride);
        unpack(d+h*stride,s+2*q,h,stride);
        unpack(d+h*stride+h,s+3*q,h,stride);
    } else {
        for (int i=0;i<n;++i) {
            U *dp = d + (size_t)i*stride; const U *sp = s + (size_t)i*n;
            int k=0;
            for (;k+32<=n;k+=32) {
                _mm256_stream_si256((__m256i*)(dp+k),    _mm256_load_si256((const __m256i*)(sp+k)));
                _mm256_stream_si256((__m256i*)(dp+k+16), _mm256_load_si256((const __m256i*)(sp+k+16)));
            }
            for (;k<n;k+=16) _mm256_storeu_si256((__m256i*)(dp+k), _mm256_loadu_si256((const __m256i*)(sp+k)));
        }
    }
}

static inline void micro24(const U *a,const U *b,U *c,int n) {
    const U *end=a+4*n; U *c3=c+3*n; intptr_t stride=2*n;
    asm volatile(
        "vmovdqa 0(%[b]), %%ymm12\n\t"
        "vmovdqa 32(%[b]), %%ymm13\n\t"
        "vpbroadcastd 0(%[a]), %%ymm14\n\t"
        "vpmaddwd %%ymm12, %%ymm14, %%ymm0\n\t"
        "vpmaddwd %%ymm13, %%ymm14, %%ymm1\n\t"
        "vpmaddwd 64(%[b]), %%ymm14, %%ymm2\n\t"
        "vpbroadcastd 4(%[a]), %%ymm14\n\t"
        "vpmaddwd %%ymm12, %%ymm14, %%ymm3\n\t"
        "vpmaddwd %%ymm13, %%ymm14, %%ymm4\n\t"
        "vpmaddwd 64(%[b]), %%ymm14, %%ymm5\n\t"
        "vpbroadcastd 8(%[a]), %%ymm14\n\t"
        "vpmaddwd %%ymm12, %%ymm14, %%ymm6\n\t"
        "vpmaddwd %%ymm13, %%ymm14, %%ymm7\n\t"
        "vpmaddwd 64(%[b]), %%ymm14, %%ymm8\n\t"
        "vpbroadcastd 12(%[a]), %%ymm14\n\t"
        "vpmaddwd %%ymm12, %%ymm14, %%ymm9\n\t"
        "vpmaddwd %%ymm13, %%ymm14, %%ymm10\n\t"
        "vpmaddwd 64(%[b]), %%ymm14, %%ymm11\n\t"
        "vmovdqa 96(%[b]), %%ymm12\n\t"
        "vmovdqa 128(%[b]), %%ymm13\n\t"
        "vpbroadcastd 16(%[a]), %%ymm14\n\t"
        "vpmaddwd %%ymm12, %%ymm14, %%ymm15\n\t"
        "vpaddd %%ymm15, %%ymm0, %%ymm0\n\t"
        "vpmaddwd %%ymm13, %%ymm14, %%ymm15\n\t"
        "vpmaddwd 160(%[b]), %%ymm14, %%ymm14\n\t"
        "vpaddd %%ymm15, %%ymm1, %%ymm1\n\t"
        "vpaddd %%ymm14, %%ymm2, %%ymm2\n\t"
        "vpbroadcastd 20(%[a]), %%ymm14\n\t"
        "vpmaddwd %%ymm12, %%ymm14, %%ymm15\n\t"
        "vpaddd %%ymm15, %%ymm3, %%ymm3\n\t"
        "vpmaddwd %%ymm13, %%ymm14, %%ymm15\n\t"
        "vpmaddwd 160(%[b]), %%ymm14, %%ymm14\n\t"
        "vpaddd %%ymm15, %%ymm4, %%ymm4\n\t"
        "vpaddd %%ymm14, %%ymm5, %%ymm5\n\t"
        "vpbroadcastd 24(%[a]), %%ymm14\n\t"
        "vpmaddwd %%ymm12, %%ymm14, %%ymm15\n\t"
        "vpaddd %%ymm15, %%ymm6, %%ymm6\n\t"
        "vpmaddwd %%ymm13, %%ymm14, %%ymm15\n\t"
        "vpmaddwd 160(%[b]), %%ymm14, %%ymm14\n\t"
        "vpaddd %%ymm15, %%ymm7, %%ymm7\n\t"
        "vpaddd %%ymm14, %%ymm8, %%ymm8\n\t"
        "vpbroadcastd 28(%[a]), %%ymm14\n\t"
        "vpmaddwd %%ymm12, %%ymm14, %%ymm15\n\t"
        "vpaddd %%ymm15, %%ymm9, %%ymm9\n\t"
        "vpmaddwd %%ymm13, %%ymm14, %%ymm15\n\t"
        "vpmaddwd 160(%[b]), %%ymm14, %%ymm14\n\t"
        "vpaddd %%ymm15, %%ymm10, %%ymm10\n\t"
        "vpaddd %%ymm14, %%ymm11, %%ymm11\n\t"
        "vmovdqa 192(%[b]), %%ymm12\n\t"
        "vmovdqa 224(%[b]), %%ymm13\n\t"
        "vpbroadcastd 32(%[a]), %%ymm14\n\t"
        "vpmaddwd %%ymm12, %%ymm14, %%ymm15\n\t"
        "vpaddd %%ymm15, %%ymm0, %%ymm0\n\t"
        "vpmaddwd %%ymm13, %%ymm14, %%ymm15\n\t"
        "vpmaddwd 256(%[b]), %%ymm14, %%ymm14\n\t"
        "vpaddd %%ymm15, %%ymm1, %%ymm1\n\t"
        "vpaddd %%ymm14, %%ymm2, %%ymm2\n\t"
        "vpbroadcastd 36(%[a]), %%ymm14\n\t"
        "vpmaddwd %%ymm12, %%ymm14, %%ymm15\n\t"
        "vpaddd %%ymm15, %%ymm3, %%ymm3\n\t"
        "vpmaddwd %%ymm13, %%ymm14, %%ymm15\n\t"
        "vpmaddwd 256(%[b]), %%ymm14, %%ymm14\n\t"
        "vpaddd %%ymm15, %%ymm4, %%ymm4\n\t"
        "vpaddd %%ymm14, %%ymm5, %%ymm5\n\t"
        "vpbroadcastd 40(%[a]), %%ymm14\n\t"
        "vpmaddwd %%ymm12, %%ymm14, %%ymm15\n\t"
        "vpaddd %%ymm15, %%ymm6, %%ymm6\n\t"
        "vpmaddwd %%ymm13, %%ymm14, %%ymm15\n\t"
        "vpmaddwd 256(%[b]), %%ymm14, %%ymm14\n\t"
        "vpaddd %%ymm15, %%ymm7, %%ymm7\n\t"
        "vpaddd %%ymm14, %%ymm8, %%ymm8\n\t"
        "vpbroadcastd 44(%[a]), %%ymm14\n\t"
        "vpmaddwd %%ymm12, %%ymm14, %%ymm15\n\t"
        "vpaddd %%ymm15, %%ymm9, %%ymm9\n\t"
        "vpmaddwd %%ymm13, %%ymm14, %%ymm15\n\t"
        "vpmaddwd 256(%[b]), %%ymm14, %%ymm14\n\t"
        "vpaddd %%ymm15, %%ymm10, %%ymm10\n\t"
        "vpaddd %%ymm14, %%ymm11, %%ymm11\n\t"
        "vmovdqa 288(%[b]), %%ymm12\n\t"
        "vmovdqa 320(%[b]), %%ymm13\n\t"
        "vpbroadcastd 48(%[a]), %%ymm14\n\t"
        "vpmaddwd %%ymm12, %%ymm14, %%ymm15\n\t"
        "vpaddd %%ymm15, %%ymm0, %%ymm0\n\t"
        "vpmaddwd %%ymm13, %%ymm14, %%ymm15\n\t"
        "vpmaddwd 352(%[b]), %%ymm14, %%ymm14\n\t"
        "vpaddd %%ymm15, %%ymm1, %%ymm1\n\t"
        "vpaddd %%ymm14, %%ymm2, %%ymm2\n\t"
        "vpbroadcastd 52(%[a]), %%ymm14\n\t"
        "vpmaddwd %%ymm12, %%ymm14, %%ymm15\n\t"
        "vpaddd %%ymm15, %%ymm3, %%ymm3\n\t"
        "vpmaddwd %%ymm13, %%ymm14, %%ymm15\n\t"
        "vpmaddwd 352(%[b]), %%ymm14, %%ymm14\n\t"
        "vpaddd %%ymm15, %%ymm4, %%ymm4\n\t"
        "vpaddd %%ymm14, %%ymm5, %%ymm5\n\t"
        "vpbroadcastd 56(%[a]), %%ymm14\n\t"
        "vpmaddwd %%ymm12, %%ymm14, %%ymm15\n\t"
        "vpaddd %%ymm15, %%ymm6, %%ymm6\n\t"
        "vpmaddwd %%ymm13, %%ymm14, %%ymm15\n\t"
        "vpmaddwd 352(%[b]), %%ymm14, %%ymm14\n\t"
        "vpaddd %%ymm15, %%ymm7, %%ymm7\n\t"
        "vpaddd %%ymm14, %%ymm8, %%ymm8\n\t"
        "vpbroadcastd 60(%[a]), %%ymm14\n\t"
        "vpmaddwd %%ymm12, %%ymm14, %%ymm15\n\t"
        "vpaddd %%ymm15, %%ymm9, %%ymm9\n\t"
        "vpmaddwd %%ymm13, %%ymm14, %%ymm15\n\t"
        "vpmaddwd 352(%[b]), %%ymm14, %%ymm14\n\t"
        "vpaddd %%ymm15, %%ymm10, %%ymm10\n\t"
        "vpaddd %%ymm14, %%ymm11, %%ymm11\n\t"
        "add $64, %[a]\n\t"
        "add $384, %[b]\n\t"
        "cmp %[end], %[a]\n\t"
        "jae 2f\n\t"
        "1:\n\t"
        "vmovdqa 0(%[b]), %%ymm12\n\t"
        "vmovdqa 32(%[b]), %%ymm13\n\t"
        "vpbroadcastd 0(%[a]), %%ymm14\n\t"
        "vpmaddwd %%ymm12, %%ymm14, %%ymm15\n\t"
        "vpaddd %%ymm15, %%ymm0, %%ymm0\n\t"
        "vpmaddwd %%ymm13, %%ymm14, %%ymm15\n\t"
        "vpmaddwd 64(%[b]), %%ymm14, %%ymm14\n\t"
        "vpaddd %%ymm15, %%ymm1, %%ymm1\n\t"
        "vpaddd %%ymm14, %%ymm2, %%ymm2\n\t"
        "vpbroadcastd 4(%[a]), %%ymm14\n\t"
        "vpmaddwd %%ymm12, %%ymm14, %%ymm15\n\t"
        "vpaddd %%ymm15, %%ymm3, %%ymm3\n\t"
        "vpmaddwd %%ymm13, %%ymm14, %%ymm15\n\t"
        "vpmaddwd 64(%[b]), %%ymm14, %%ymm14\n\t"
        "vpaddd %%ymm15, %%ymm4, %%ymm4\n\t"
        "vpaddd %%ymm14, %%ymm5, %%ymm5\n\t"
        "vpbroadcastd 8(%[a]), %%ymm14\n\t"
        "vpmaddwd %%ymm12, %%ymm14, %%ymm15\n\t"
        "vpaddd %%ymm15, %%ymm6, %%ymm6\n\t"
        "vpmaddwd %%ymm13, %%ymm14, %%ymm15\n\t"
        "vpmaddwd 64(%[b]), %%ymm14, %%ymm14\n\t"
        "vpaddd %%ymm15, %%ymm7, %%ymm7\n\t"
        "vpaddd %%ymm14, %%ymm8, %%ymm8\n\t"
        "vpbroadcastd 12(%[a]), %%ymm14\n\t"
        "vpmaddwd %%ymm12, %%ymm14, %%ymm15\n\t"
        "vpaddd %%ymm15, %%ymm9, %%ymm9\n\t"
        "vpmaddwd %%ymm13, %%ymm14, %%ymm15\n\t"
        "vpmaddwd 64(%[b]), %%ymm14, %%ymm14\n\t"
        "vpaddd %%ymm15, %%ymm10, %%ymm10\n\t"
        "vpaddd %%ymm14, %%ymm11, %%ymm11\n\t"
        "vmovdqa 96(%[b]), %%ymm12\n\t"
        "vmovdqa 128(%[b]), %%ymm13\n\t"
        "vpbroadcastd 16(%[a]), %%ymm14\n\t"
        "vpmaddwd %%ymm12, %%ymm14, %%ymm15\n\t"
        "vpaddd %%ymm15, %%ymm0, %%ymm0\n\t"
        "vpmaddwd %%ymm13, %%ymm14, %%ymm15\n\t"
        "vpmaddwd 160(%[b]), %%ymm14, %%ymm14\n\t"
        "vpaddd %%ymm15, %%ymm1, %%ymm1\n\t"
        "vpaddd %%ymm14, %%ymm2, %%ymm2\n\t"
        "vpbroadcastd 20(%[a]), %%ymm14\n\t"
        "vpmaddwd %%ymm12, %%ymm14, %%ymm15\n\t"
        "vpaddd %%ymm15, %%ymm3, %%ymm3\n\t"
        "vpmaddwd %%ymm13, %%ymm14, %%ymm15\n\t"
        "vpmaddwd 160(%[b]), %%ymm14, %%ymm14\n\t"
        "vpaddd %%ymm15, %%ymm4, %%ymm4\n\t"
        "vpaddd %%ymm14, %%ymm5, %%ymm5\n\t"
        "vpbroadcastd 24(%[a]), %%ymm14\n\t"
        "vpmaddwd %%ymm12, %%ymm14, %%ymm15\n\t"
        "vpaddd %%ymm15, %%ymm6, %%ymm6\n\t"
        "vpmaddwd %%ymm13, %%ymm14, %%ymm15\n\t"
        "vpmaddwd 160(%[b]), %%ymm14, %%ymm14\n\t"
        "vpaddd %%ymm15, %%ymm7, %%ymm7\n\t"
        "vpaddd %%ymm14, %%ymm8, %%ymm8\n\t"
        "vpbroadcastd 28(%[a]), %%ymm14\n\t"
        "vpmaddwd %%ymm12, %%ymm14, %%ymm15\n\t"
        "vpaddd %%ymm15, %%ymm9, %%ymm9\n\t"
        "vpmaddwd %%ymm13, %%ymm14, %%ymm15\n\t"
        "vpmaddwd 160(%[b]), %%ymm14, %%ymm14\n\t"
        "vpaddd %%ymm15, %%ymm10, %%ymm10\n\t"
        "vpaddd %%ymm14, %%ymm11, %%ymm11\n\t"
        "vmovdqa 192(%[b]), %%ymm12\n\t"
        "vmovdqa 224(%[b]), %%ymm13\n\t"
        "vpbroadcastd 32(%[a]), %%ymm14\n\t"
        "vpmaddwd %%ymm12, %%ymm14, %%ymm15\n\t"
        "vpaddd %%ymm15, %%ymm0, %%ymm0\n\t"
        "vpmaddwd %%ymm13, %%ymm14, %%ymm15\n\t"
        "vpmaddwd 256(%[b]), %%ymm14, %%ymm14\n\t"
        "vpaddd %%ymm15, %%ymm1, %%ymm1\n\t"
        "vpaddd %%ymm14, %%ymm2, %%ymm2\n\t"
        "vpbroadcastd 36(%[a]), %%ymm14\n\t"
        "vpmaddwd %%ymm12, %%ymm14, %%ymm15\n\t"
        "vpaddd %%ymm15, %%ymm3, %%ymm3\n\t"
        "vpmaddwd %%ymm13, %%ymm14, %%ymm15\n\t"
        "vpmaddwd 256(%[b]), %%ymm14, %%ymm14\n\t"
        "vpaddd %%ymm15, %%ymm4, %%ymm4\n\t"
        "vpaddd %%ymm14, %%ymm5, %%ymm5\n\t"
        "vpbroadcastd 40(%[a]), %%ymm14\n\t"
        "vpmaddwd %%ymm12, %%ymm14, %%ymm15\n\t"
        "vpaddd %%ymm15, %%ymm6, %%ymm6\n\t"
        "vpmaddwd %%ymm13, %%ymm14, %%ymm15\n\t"
        "vpmaddwd 256(%[b]), %%ymm14, %%ymm14\n\t"
        "vpaddd %%ymm15, %%ymm7, %%ymm7\n\t"
        "vpaddd %%ymm14, %%ymm8, %%ymm8\n\t"
        "vpbroadcastd 44(%[a]), %%ymm14\n\t"
        "vpmaddwd %%ymm12, %%ymm14, %%ymm15\n\t"
        "vpaddd %%ymm15, %%ymm9, %%ymm9\n\t"
        "vpmaddwd %%ymm13, %%ymm14, %%ymm15\n\t"
        "vpmaddwd 256(%[b]), %%ymm14, %%ymm14\n\t"
        "vpaddd %%ymm15, %%ymm10, %%ymm10\n\t"
        "vpaddd %%ymm14, %%ymm11, %%ymm11\n\t"
        "vmovdqa 288(%[b]), %%ymm12\n\t"
        "vmovdqa 320(%[b]), %%ymm13\n\t"
        "vpbroadcastd 48(%[a]), %%ymm14\n\t"
        "vpmaddwd %%ymm12, %%ymm14, %%ymm15\n\t"
        "vpaddd %%ymm15, %%ymm0, %%ymm0\n\t"
        "vpmaddwd %%ymm13, %%ymm14, %%ymm15\n\t"
        "vpmaddwd 352(%[b]), %%ymm14, %%ymm14\n\t"
        "vpaddd %%ymm15, %%ymm1, %%ymm1\n\t"
        "vpaddd %%ymm14, %%ymm2, %%ymm2\n\t"
        "vpbroadcastd 52(%[a]), %%ymm14\n\t"
        "vpmaddwd %%ymm12, %%ymm14, %%ymm15\n\t"
        "vpaddd %%ymm15, %%ymm3, %%ymm3\n\t"
        "vpmaddwd %%ymm13, %%ymm14, %%ymm15\n\t"
        "vpmaddwd 352(%[b]), %%ymm14, %%ymm14\n\t"
        "vpaddd %%ymm15, %%ymm4, %%ymm4\n\t"
        "vpaddd %%ymm14, %%ymm5, %%ymm5\n\t"
        "vpbroadcastd 56(%[a]), %%ymm14\n\t"
        "vpmaddwd %%ymm12, %%ymm14, %%ymm15\n\t"
        "vpaddd %%ymm15, %%ymm6, %%ymm6\n\t"
        "vpmaddwd %%ymm13, %%ymm14, %%ymm15\n\t"
        "vpmaddwd 352(%[b]), %%ymm14, %%ymm14\n\t"
        "vpaddd %%ymm15, %%ymm7, %%ymm7\n\t"
        "vpaddd %%ymm14, %%ymm8, %%ymm8\n\t"
        "vpbroadcastd 60(%[a]), %%ymm14\n\t"
        "vpmaddwd %%ymm12, %%ymm14, %%ymm15\n\t"
        "vpaddd %%ymm15, %%ymm9, %%ymm9\n\t"
        "vpmaddwd %%ymm13, %%ymm14, %%ymm15\n\t"
        "vpmaddwd 352(%[b]), %%ymm14, %%ymm14\n\t"
        "vpaddd %%ymm15, %%ymm10, %%ymm10\n\t"
        "vpaddd %%ymm14, %%ymm11, %%ymm11\n\t"
        "add $64, %[a]\n\t"
        "add $384, %[b]\n\t"
        "cmp %[end], %[a]\n\t"
        "jb 1b\n\t"
        "2:\n\t"
        "vpcmpeqd %%ymm13, %%ymm13, %%ymm13\n\t"
        "vpsrld $16, %%ymm13, %%ymm13\n\t"
        "vpand %%ymm13, %%ymm0, %%ymm0\n\t"
        "vpand %%ymm13, %%ymm1, %%ymm1\n\t"
        "vpand %%ymm13, %%ymm2, %%ymm2\n\t"
        "vpackusdw %%ymm1, %%ymm0, %%ymm0\n\t"
        "vpermq $216, %%ymm0, %%ymm0\n\t"
        "vmovdqu %%ymm0, 0(%[c])\n\t"
        "vpackusdw %%ymm2, %%ymm2, %%ymm2\n\t"
        "vpermq $216, %%ymm2, %%ymm2\n\t"
        "vmovdqu %%xmm2, 32(%[c])\n\t"
        "vpand %%ymm13, %%ymm3, %%ymm3\n\t"
        "vpand %%ymm13, %%ymm4, %%ymm4\n\t"
        "vpand %%ymm13, %%ymm5, %%ymm5\n\t"
        "vpackusdw %%ymm4, %%ymm3, %%ymm3\n\t"
        "vpermq $216, %%ymm3, %%ymm3\n\t"
        "vmovdqu %%ymm3, 0(%[c],%[s],1)\n\t"
        "vpackusdw %%ymm5, %%ymm5, %%ymm5\n\t"
        "vpermq $216, %%ymm5, %%ymm5\n\t"
        "vmovdqu %%xmm5, 32(%[c],%[s],1)\n\t"
        "vpand %%ymm13, %%ymm6, %%ymm6\n\t"
        "vpand %%ymm13, %%ymm7, %%ymm7\n\t"
        "vpand %%ymm13, %%ymm8, %%ymm8\n\t"
        "vpackusdw %%ymm7, %%ymm6, %%ymm6\n\t"
        "vpermq $216, %%ymm6, %%ymm6\n\t"
        "vmovdqu %%ymm6, 0(%[c],%[s],2)\n\t"
        "vpackusdw %%ymm8, %%ymm8, %%ymm8\n\t"
        "vpermq $216, %%ymm8, %%ymm8\n\t"
        "vmovdqu %%xmm8, 32(%[c],%[s],2)\n\t"
        "vpand %%ymm13, %%ymm9, %%ymm9\n\t"
        "vpand %%ymm13, %%ymm10, %%ymm10\n\t"
        "vpand %%ymm13, %%ymm11, %%ymm11\n\t"
        "vpackusdw %%ymm10, %%ymm9, %%ymm9\n\t"
        "vpermq $216, %%ymm9, %%ymm9\n\t"
        "vmovdqu %%ymm9, 0(%[c3])\n\t"
        "vpackusdw %%ymm11, %%ymm11, %%ymm11\n\t"
        "vpermq $216, %%ymm11, %%ymm11\n\t"
        "vmovdqu %%xmm11, 32(%[c3])\n\t"
        : [a] "+&r"(a), [b] "+&r"(b)
        : [c] "r"(c), [c3] "r"(c3), [s] "r"(stride), [end] "r"(end)
        : "cc", "memory", "ymm0", "ymm1", "ymm2", "ymm3", "ymm4", "ymm5", "ymm6", "ymm7", "ymm8", "ymm9", "ymm10", "ymm11", "ymm12", "ymm13", "ymm14", "ymm15");
}

static inline void micro16(const U *a,const U *b,U *c,int n) {
    const U *end=a+4*n; U *c3=c+3*n; intptr_t stride=2*n;
    asm volatile(
        "vmovdqa 0(%[b]), %%ymm12\n\t"
        "vmovdqa 32(%[b]), %%ymm13\n\t"
        "vpbroadcastd 0(%[a]), %%ymm14\n\t"
        "vpmaddwd %%ymm12, %%ymm14, %%ymm0\n\t"
        "vpmaddwd %%ymm13, %%ymm14, %%ymm1\n\t"
        "vpbroadcastd 4(%[a]), %%ymm14\n\t"
        "vpmaddwd %%ymm12, %%ymm14, %%ymm2\n\t"
        "vpmaddwd %%ymm13, %%ymm14, %%ymm3\n\t"
        "vpbroadcastd 8(%[a]), %%ymm14\n\t"
        "vpmaddwd %%ymm12, %%ymm14, %%ymm4\n\t"
        "vpmaddwd %%ymm13, %%ymm14, %%ymm5\n\t"
        "vpbroadcastd 12(%[a]), %%ymm14\n\t"
        "vpmaddwd %%ymm12, %%ymm14, %%ymm6\n\t"
        "vpmaddwd %%ymm13, %%ymm14, %%ymm7\n\t"
        "vmovdqa 64(%[b]), %%ymm12\n\t"
        "vmovdqa 96(%[b]), %%ymm13\n\t"
        "vpbroadcastd 16(%[a]), %%ymm14\n\t"
        "vpmaddwd %%ymm12, %%ymm14, %%ymm15\n\t"
        "vpmaddwd %%ymm13, %%ymm14, %%ymm14\n\t"
        "vpaddd %%ymm15, %%ymm0, %%ymm0\n\t"
        "vpaddd %%ymm14, %%ymm1, %%ymm1\n\t"
        "vpbroadcastd 20(%[a]), %%ymm14\n\t"
        "vpmaddwd %%ymm12, %%ymm14, %%ymm15\n\t"
        "vpmaddwd %%ymm13, %%ymm14, %%ymm14\n\t"
        "vpaddd %%ymm15, %%ymm2, %%ymm2\n\t"
        "vpaddd %%ymm14, %%ymm3, %%ymm3\n\t"
        "vpbroadcastd 24(%[a]), %%ymm14\n\t"
        "vpmaddwd %%ymm12, %%ymm14, %%ymm15\n\t"
        "vpmaddwd %%ymm13, %%ymm14, %%ymm14\n\t"
        "vpaddd %%ymm15, %%ymm4, %%ymm4\n\t"
        "vpaddd %%ymm14, %%ymm5, %%ymm5\n\t"
        "vpbroadcastd 28(%[a]), %%ymm14\n\t"
        "vpmaddwd %%ymm12, %%ymm14, %%ymm15\n\t"
        "vpmaddwd %%ymm13, %%ymm14, %%ymm14\n\t"
        "vpaddd %%ymm15, %%ymm6, %%ymm6\n\t"
        "vpaddd %%ymm14, %%ymm7, %%ymm7\n\t"
        "vmovdqa 128(%[b]), %%ymm12\n\t"
        "vmovdqa 160(%[b]), %%ymm13\n\t"
        "vpbroadcastd 32(%[a]), %%ymm14\n\t"
        "vpmaddwd %%ymm12, %%ymm14, %%ymm15\n\t"
        "vpmaddwd %%ymm13, %%ymm14, %%ymm14\n\t"
        "vpaddd %%ymm15, %%ymm0, %%ymm0\n\t"
        "vpaddd %%ymm14, %%ymm1, %%ymm1\n\t"
        "vpbroadcastd 36(%[a]), %%ymm14\n\t"
        "vpmaddwd %%ymm12, %%ymm14, %%ymm15\n\t"
        "vpmaddwd %%ymm13, %%ymm14, %%ymm14\n\t"
        "vpaddd %%ymm15, %%ymm2, %%ymm2\n\t"
        "vpaddd %%ymm14, %%ymm3, %%ymm3\n\t"
        "vpbroadcastd 40(%[a]), %%ymm14\n\t"
        "vpmaddwd %%ymm12, %%ymm14, %%ymm15\n\t"
        "vpmaddwd %%ymm13, %%ymm14, %%ymm14\n\t"
        "vpaddd %%ymm15, %%ymm4, %%ymm4\n\t"
        "vpaddd %%ymm14, %%ymm5, %%ymm5\n\t"
        "vpbroadcastd 44(%[a]), %%ymm14\n\t"
        "vpmaddwd %%ymm12, %%ymm14, %%ymm15\n\t"
        "vpmaddwd %%ymm13, %%ymm14, %%ymm14\n\t"
        "vpaddd %%ymm15, %%ymm6, %%ymm6\n\t"
        "vpaddd %%ymm14, %%ymm7, %%ymm7\n\t"
        "vmovdqa 192(%[b]), %%ymm12\n\t"
        "vmovdqa 224(%[b]), %%ymm13\n\t"
        "vpbroadcastd 48(%[a]), %%ymm14\n\t"
        "vpmaddwd %%ymm12, %%ymm14, %%ymm15\n\t"
        "vpmaddwd %%ymm13, %%ymm14, %%ymm14\n\t"
        "vpaddd %%ymm15, %%ymm0, %%ymm0\n\t"
        "vpaddd %%ymm14, %%ymm1, %%ymm1\n\t"
        "vpbroadcastd 52(%[a]), %%ymm14\n\t"
        "vpmaddwd %%ymm12, %%ymm14, %%ymm15\n\t"
        "vpmaddwd %%ymm13, %%ymm14, %%ymm14\n\t"
        "vpaddd %%ymm15, %%ymm2, %%ymm2\n\t"
        "vpaddd %%ymm14, %%ymm3, %%ymm3\n\t"
        "vpbroadcastd 56(%[a]), %%ymm14\n\t"
        "vpmaddwd %%ymm12, %%ymm14, %%ymm15\n\t"
        "vpmaddwd %%ymm13, %%ymm14, %%ymm14\n\t"
        "vpaddd %%ymm15, %%ymm4, %%ymm4\n\t"
        "vpaddd %%ymm14, %%ymm5, %%ymm5\n\t"
        "vpbroadcastd 60(%[a]), %%ymm14\n\t"
        "vpmaddwd %%ymm12, %%ymm14, %%ymm15\n\t"
        "vpmaddwd %%ymm13, %%ymm14, %%ymm14\n\t"
        "vpaddd %%ymm15, %%ymm6, %%ymm6\n\t"
        "vpaddd %%ymm14, %%ymm7, %%ymm7\n\t"
        "add $64, %[a]\n\t"
        "add $256, %[b]\n\t"
        "cmp %[end], %[a]\n\t"
        "jae 2f\n\t"
        "1:\n\t"
        "vmovdqa 0(%[b]), %%ymm12\n\t"
        "vmovdqa 32(%[b]), %%ymm13\n\t"
        "vpbroadcastd 0(%[a]), %%ymm14\n\t"
        "vpmaddwd %%ymm12, %%ymm14, %%ymm15\n\t"
        "vpmaddwd %%ymm13, %%ymm14, %%ymm14\n\t"
        "vpaddd %%ymm15, %%ymm0, %%ymm0\n\t"
        "vpaddd %%ymm14, %%ymm1, %%ymm1\n\t"
        "vpbroadcastd 4(%[a]), %%ymm14\n\t"
        "vpmaddwd %%ymm12, %%ymm14, %%ymm15\n\t"
        "vpmaddwd %%ymm13, %%ymm14, %%ymm14\n\t"
        "vpaddd %%ymm15, %%ymm2, %%ymm2\n\t"
        "vpaddd %%ymm14, %%ymm3, %%ymm3\n\t"
        "vpbroadcastd 8(%[a]), %%ymm14\n\t"
        "vpmaddwd %%ymm12, %%ymm14, %%ymm15\n\t"
        "vpmaddwd %%ymm13, %%ymm14, %%ymm14\n\t"
        "vpaddd %%ymm15, %%ymm4, %%ymm4\n\t"
        "vpaddd %%ymm14, %%ymm5, %%ymm5\n\t"
        "vpbroadcastd 12(%[a]), %%ymm14\n\t"
        "vpmaddwd %%ymm12, %%ymm14, %%ymm15\n\t"
        "vpmaddwd %%ymm13, %%ymm14, %%ymm14\n\t"
        "vpaddd %%ymm15, %%ymm6, %%ymm6\n\t"
        "vpaddd %%ymm14, %%ymm7, %%ymm7\n\t"
        "vmovdqa 64(%[b]), %%ymm12\n\t"
        "vmovdqa 96(%[b]), %%ymm13\n\t"
        "vpbroadcastd 16(%[a]), %%ymm14\n\t"
        "vpmaddwd %%ymm12, %%ymm14, %%ymm15\n\t"
        "vpmaddwd %%ymm13, %%ymm14, %%ymm14\n\t"
        "vpaddd %%ymm15, %%ymm0, %%ymm0\n\t"
        "vpaddd %%ymm14, %%ymm1, %%ymm1\n\t"
        "vpbroadcastd 20(%[a]), %%ymm14\n\t"
        "vpmaddwd %%ymm12, %%ymm14, %%ymm15\n\t"
        "vpmaddwd %%ymm13, %%ymm14, %%ymm14\n\t"
        "vpaddd %%ymm15, %%ymm2, %%ymm2\n\t"
        "vpaddd %%ymm14, %%ymm3, %%ymm3\n\t"
        "vpbroadcastd 24(%[a]), %%ymm14\n\t"
        "vpmaddwd %%ymm12, %%ymm14, %%ymm15\n\t"
        "vpmaddwd %%ymm13, %%ymm14, %%ymm14\n\t"
        "vpaddd %%ymm15, %%ymm4, %%ymm4\n\t"
        "vpaddd %%ymm14, %%ymm5, %%ymm5\n\t"
        "vpbroadcastd 28(%[a]), %%ymm14\n\t"
        "vpmaddwd %%ymm12, %%ymm14, %%ymm15\n\t"
        "vpmaddwd %%ymm13, %%ymm14, %%ymm14\n\t"
        "vpaddd %%ymm15, %%ymm6, %%ymm6\n\t"
        "vpaddd %%ymm14, %%ymm7, %%ymm7\n\t"
        "vmovdqa 128(%[b]), %%ymm12\n\t"
        "vmovdqa 160(%[b]), %%ymm13\n\t"
        "vpbroadcastd 32(%[a]), %%ymm14\n\t"
        "vpmaddwd %%ymm12, %%ymm14, %%ymm15\n\t"
        "vpmaddwd %%ymm13, %%ymm14, %%ymm14\n\t"
        "vpaddd %%ymm15, %%ymm0, %%ymm0\n\t"
        "vpaddd %%ymm14, %%ymm1, %%ymm1\n\t"
        "vpbroadcastd 36(%[a]), %%ymm14\n\t"
        "vpmaddwd %%ymm12, %%ymm14, %%ymm15\n\t"
        "vpmaddwd %%ymm13, %%ymm14, %%ymm14\n\t"
        "vpaddd %%ymm15, %%ymm2, %%ymm2\n\t"
        "vpaddd %%ymm14, %%ymm3, %%ymm3\n\t"
        "vpbroadcastd 40(%[a]), %%ymm14\n\t"
        "vpmaddwd %%ymm12, %%ymm14, %%ymm15\n\t"
        "vpmaddwd %%ymm13, %%ymm14, %%ymm14\n\t"
        "vpaddd %%ymm15, %%ymm4, %%ymm4\n\t"
        "vpaddd %%ymm14, %%ymm5, %%ymm5\n\t"
        "vpbroadcastd 44(%[a]), %%ymm14\n\t"
        "vpmaddwd %%ymm12, %%ymm14, %%ymm15\n\t"
        "vpmaddwd %%ymm13, %%ymm14, %%ymm14\n\t"
        "vpaddd %%ymm15, %%ymm6, %%ymm6\n\t"
        "vpaddd %%ymm14, %%ymm7, %%ymm7\n\t"
        "vmovdqa 192(%[b]), %%ymm12\n\t"
        "vmovdqa 224(%[b]), %%ymm13\n\t"
        "vpbroadcastd 48(%[a]), %%ymm14\n\t"
        "vpmaddwd %%ymm12, %%ymm14, %%ymm15\n\t"
        "vpmaddwd %%ymm13, %%ymm14, %%ymm14\n\t"
        "vpaddd %%ymm15, %%ymm0, %%ymm0\n\t"
        "vpaddd %%ymm14, %%ymm1, %%ymm1\n\t"
        "vpbroadcastd 52(%[a]), %%ymm14\n\t"
        "vpmaddwd %%ymm12, %%ymm14, %%ymm15\n\t"
        "vpmaddwd %%ymm13, %%ymm14, %%ymm14\n\t"
        "vpaddd %%ymm15, %%ymm2, %%ymm2\n\t"
        "vpaddd %%ymm14, %%ymm3, %%ymm3\n\t"
        "vpbroadcastd 56(%[a]), %%ymm14\n\t"
        "vpmaddwd %%ymm12, %%ymm14, %%ymm15\n\t"
        "vpmaddwd %%ymm13, %%ymm14, %%ymm14\n\t"
        "vpaddd %%ymm15, %%ymm4, %%ymm4\n\t"
        "vpaddd %%ymm14, %%ymm5, %%ymm5\n\t"
        "vpbroadcastd 60(%[a]), %%ymm14\n\t"
        "vpmaddwd %%ymm12, %%ymm14, %%ymm15\n\t"
        "vpmaddwd %%ymm13, %%ymm14, %%ymm14\n\t"
        "vpaddd %%ymm15, %%ymm6, %%ymm6\n\t"
        "vpaddd %%ymm14, %%ymm7, %%ymm7\n\t"
        "add $64, %[a]\n\t"
        "add $256, %[b]\n\t"
        "cmp %[end], %[a]\n\t"
        "jb 1b\n\t"
        "2:\n\t"
        "vpcmpeqd %%ymm13, %%ymm13, %%ymm13\n\t"
        "vpsrld $16, %%ymm13, %%ymm13\n\t"
        "vpand %%ymm13, %%ymm0, %%ymm0\n\t"
        "vpand %%ymm13, %%ymm1, %%ymm1\n\t"
        "vpackusdw %%ymm1, %%ymm0, %%ymm0\n\t"
        "vpermq $216, %%ymm0, %%ymm0\n\t"
        "vmovdqu %%ymm0, 0(%[c])\n\t"
        "vpand %%ymm13, %%ymm2, %%ymm2\n\t"
        "vpand %%ymm13, %%ymm3, %%ymm3\n\t"
        "vpackusdw %%ymm3, %%ymm2, %%ymm2\n\t"
        "vpermq $216, %%ymm2, %%ymm2\n\t"
        "vmovdqu %%ymm2, 0(%[c],%[s],1)\n\t"
        "vpand %%ymm13, %%ymm4, %%ymm4\n\t"
        "vpand %%ymm13, %%ymm5, %%ymm5\n\t"
        "vpackusdw %%ymm5, %%ymm4, %%ymm4\n\t"
        "vpermq $216, %%ymm4, %%ymm4\n\t"
        "vmovdqu %%ymm4, 0(%[c],%[s],2)\n\t"
        "vpand %%ymm13, %%ymm6, %%ymm6\n\t"
        "vpand %%ymm13, %%ymm7, %%ymm7\n\t"
        "vpackusdw %%ymm7, %%ymm6, %%ymm6\n\t"
        "vpermq $216, %%ymm6, %%ymm6\n\t"
        "vmovdqu %%ymm6, 0(%[c3])\n\t"
        : [a] "+&r"(a), [b] "+&r"(b)
        : [c] "r"(c), [c3] "r"(c3), [s] "r"(stride), [end] "r"(end)
        : "cc", "memory", "ymm0", "ymm1", "ymm2", "ymm3", "ymm4", "ymm5", "ymm6", "ymm7", "ymm8", "ymm9", "ymm10", "ymm11", "ymm12", "ymm13", "ymm14", "ymm15");
}

static inline void micro8(const U *a,const U *b,U *c,int n) {
    const U *end=a+4*n; U *c3=c+3*n; intptr_t stride=2*n;
    asm volatile(
        "vmovdqa 0(%[b]), %%ymm12\n\t"
        "vpbroadcastd 0(%[a]), %%ymm14\n\t"
        "vpmaddwd %%ymm12, %%ymm14, %%ymm0\n\t"
        "vpbroadcastd 4(%[a]), %%ymm14\n\t"
        "vpmaddwd %%ymm12, %%ymm14, %%ymm1\n\t"
        "vpbroadcastd 8(%[a]), %%ymm14\n\t"
        "vpmaddwd %%ymm12, %%ymm14, %%ymm2\n\t"
        "vpbroadcastd 12(%[a]), %%ymm14\n\t"
        "vpmaddwd %%ymm12, %%ymm14, %%ymm3\n\t"
        "vmovdqa 32(%[b]), %%ymm12\n\t"
        "vpbroadcastd 16(%[a]), %%ymm14\n\t"
        "vpmaddwd %%ymm12, %%ymm14, %%ymm15\n\t"
        "vpaddd %%ymm15, %%ymm0, %%ymm0\n\t"
        "vpbroadcastd 20(%[a]), %%ymm14\n\t"
        "vpmaddwd %%ymm12, %%ymm14, %%ymm15\n\t"
        "vpaddd %%ymm15, %%ymm1, %%ymm1\n\t"
        "vpbroadcastd 24(%[a]), %%ymm14\n\t"
        "vpmaddwd %%ymm12, %%ymm14, %%ymm15\n\t"
        "vpaddd %%ymm15, %%ymm2, %%ymm2\n\t"
        "vpbroadcastd 28(%[a]), %%ymm14\n\t"
        "vpmaddwd %%ymm12, %%ymm14, %%ymm15\n\t"
        "vpaddd %%ymm15, %%ymm3, %%ymm3\n\t"
        "vmovdqa 64(%[b]), %%ymm12\n\t"
        "vpbroadcastd 32(%[a]), %%ymm14\n\t"
        "vpmaddwd %%ymm12, %%ymm14, %%ymm15\n\t"
        "vpaddd %%ymm15, %%ymm0, %%ymm0\n\t"
        "vpbroadcastd 36(%[a]), %%ymm14\n\t"
        "vpmaddwd %%ymm12, %%ymm14, %%ymm15\n\t"
        "vpaddd %%ymm15, %%ymm1, %%ymm1\n\t"
        "vpbroadcastd 40(%[a]), %%ymm14\n\t"
        "vpmaddwd %%ymm12, %%ymm14, %%ymm15\n\t"
        "vpaddd %%ymm15, %%ymm2, %%ymm2\n\t"
        "vpbroadcastd 44(%[a]), %%ymm14\n\t"
        "vpmaddwd %%ymm12, %%ymm14, %%ymm15\n\t"
        "vpaddd %%ymm15, %%ymm3, %%ymm3\n\t"
        "vmovdqa 96(%[b]), %%ymm12\n\t"
        "vpbroadcastd 48(%[a]), %%ymm14\n\t"
        "vpmaddwd %%ymm12, %%ymm14, %%ymm15\n\t"
        "vpaddd %%ymm15, %%ymm0, %%ymm0\n\t"
        "vpbroadcastd 52(%[a]), %%ymm14\n\t"
        "vpmaddwd %%ymm12, %%ymm14, %%ymm15\n\t"
        "vpaddd %%ymm15, %%ymm1, %%ymm1\n\t"
        "vpbroadcastd 56(%[a]), %%ymm14\n\t"
        "vpmaddwd %%ymm12, %%ymm14, %%ymm15\n\t"
        "vpaddd %%ymm15, %%ymm2, %%ymm2\n\t"
        "vpbroadcastd 60(%[a]), %%ymm14\n\t"
        "vpmaddwd %%ymm12, %%ymm14, %%ymm15\n\t"
        "vpaddd %%ymm15, %%ymm3, %%ymm3\n\t"
        "add $64, %[a]\n\t"
        "add $128, %[b]\n\t"
        "cmp %[end], %[a]\n\t"
        "jae 2f\n\t"
        "1:\n\t"
        "vmovdqa 0(%[b]), %%ymm12\n\t"
        "vpbroadcastd 0(%[a]), %%ymm14\n\t"
        "vpmaddwd %%ymm12, %%ymm14, %%ymm15\n\t"
        "vpaddd %%ymm15, %%ymm0, %%ymm0\n\t"
        "vpbroadcastd 4(%[a]), %%ymm14\n\t"
        "vpmaddwd %%ymm12, %%ymm14, %%ymm15\n\t"
        "vpaddd %%ymm15, %%ymm1, %%ymm1\n\t"
        "vpbroadcastd 8(%[a]), %%ymm14\n\t"
        "vpmaddwd %%ymm12, %%ymm14, %%ymm15\n\t"
        "vpaddd %%ymm15, %%ymm2, %%ymm2\n\t"
        "vpbroadcastd 12(%[a]), %%ymm14\n\t"
        "vpmaddwd %%ymm12, %%ymm14, %%ymm15\n\t"
        "vpaddd %%ymm15, %%ymm3, %%ymm3\n\t"
        "vmovdqa 32(%[b]), %%ymm12\n\t"
        "vpbroadcastd 16(%[a]), %%ymm14\n\t"
        "vpmaddwd %%ymm12, %%ymm14, %%ymm15\n\t"
        "vpaddd %%ymm15, %%ymm0, %%ymm0\n\t"
        "vpbroadcastd 20(%[a]), %%ymm14\n\t"
        "vpmaddwd %%ymm12, %%ymm14, %%ymm15\n\t"
        "vpaddd %%ymm15, %%ymm1, %%ymm1\n\t"
        "vpbroadcastd 24(%[a]), %%ymm14\n\t"
        "vpmaddwd %%ymm12, %%ymm14, %%ymm15\n\t"
        "vpaddd %%ymm15, %%ymm2, %%ymm2\n\t"
        "vpbroadcastd 28(%[a]), %%ymm14\n\t"
        "vpmaddwd %%ymm12, %%ymm14, %%ymm15\n\t"
        "vpaddd %%ymm15, %%ymm3, %%ymm3\n\t"
        "vmovdqa 64(%[b]), %%ymm12\n\t"
        "vpbroadcastd 32(%[a]), %%ymm14\n\t"
        "vpmaddwd %%ymm12, %%ymm14, %%ymm15\n\t"
        "vpaddd %%ymm15, %%ymm0, %%ymm0\n\t"
        "vpbroadcastd 36(%[a]), %%ymm14\n\t"
        "vpmaddwd %%ymm12, %%ymm14, %%ymm15\n\t"
        "vpaddd %%ymm15, %%ymm1, %%ymm1\n\t"
        "vpbroadcastd 40(%[a]), %%ymm14\n\t"
        "vpmaddwd %%ymm12, %%ymm14, %%ymm15\n\t"
        "vpaddd %%ymm15, %%ymm2, %%ymm2\n\t"
        "vpbroadcastd 44(%[a]), %%ymm14\n\t"
        "vpmaddwd %%ymm12, %%ymm14, %%ymm15\n\t"
        "vpaddd %%ymm15, %%ymm3, %%ymm3\n\t"
        "vmovdqa 96(%[b]), %%ymm12\n\t"
        "vpbroadcastd 48(%[a]), %%ymm14\n\t"
        "vpmaddwd %%ymm12, %%ymm14, %%ymm15\n\t"
        "vpaddd %%ymm15, %%ymm0, %%ymm0\n\t"
        "vpbroadcastd 52(%[a]), %%ymm14\n\t"
        "vpmaddwd %%ymm12, %%ymm14, %%ymm15\n\t"
        "vpaddd %%ymm15, %%ymm1, %%ymm1\n\t"
        "vpbroadcastd 56(%[a]), %%ymm14\n\t"
        "vpmaddwd %%ymm12, %%ymm14, %%ymm15\n\t"
        "vpaddd %%ymm15, %%ymm2, %%ymm2\n\t"
        "vpbroadcastd 60(%[a]), %%ymm14\n\t"
        "vpmaddwd %%ymm12, %%ymm14, %%ymm15\n\t"
        "vpaddd %%ymm15, %%ymm3, %%ymm3\n\t"
        "add $64, %[a]\n\t"
        "add $128, %[b]\n\t"
        "cmp %[end], %[a]\n\t"
        "jb 1b\n\t"
        "2:\n\t"
        "vpcmpeqd %%ymm13, %%ymm13, %%ymm13\n\t"
        "vpsrld $16, %%ymm13, %%ymm13\n\t"
        "vpand %%ymm13, %%ymm0, %%ymm0\n\t"
        "vpackusdw %%ymm0, %%ymm0, %%ymm0\n\t"
        "vpermq $216, %%ymm0, %%ymm0\n\t"
        "vmovdqu %%xmm0, 0(%[c])\n\t"
        "vpand %%ymm13, %%ymm1, %%ymm1\n\t"
        "vpackusdw %%ymm1, %%ymm1, %%ymm1\n\t"
        "vpermq $216, %%ymm1, %%ymm1\n\t"
        "vmovdqu %%xmm1, 0(%[c],%[s],1)\n\t"
        "vpand %%ymm13, %%ymm2, %%ymm2\n\t"
        "vpackusdw %%ymm2, %%ymm2, %%ymm2\n\t"
        "vpermq $216, %%ymm2, %%ymm2\n\t"
        "vmovdqu %%xmm2, 0(%[c],%[s],2)\n\t"
        "vpand %%ymm13, %%ymm3, %%ymm3\n\t"
        "vpackusdw %%ymm3, %%ymm3, %%ymm3\n\t"
        "vpermq $216, %%ymm3, %%ymm3\n\t"
        "vmovdqu %%xmm3, 0(%[c3])\n\t"
        : [a] "+&r"(a), [b] "+&r"(b)
        : [c] "r"(c), [c3] "r"(c3), [s] "r"(stride), [end] "r"(end)
        : "cc", "memory", "ymm0", "ymm1", "ymm2", "ymm3", "ymm4", "ymm5", "ymm6", "ymm7", "ymm8", "ymm9", "ymm10", "ymm11", "ymm12", "ymm13", "ymm14", "ymm15");
}

static void base(const U *a,const U *b,U *c,int n) {
    int j=0;
    for(;j+24<=n;j+=24)
        for(int i=0;i<n;i+=4) micro24(a+i*n,b+j*n,c+i*n+j,n);
    if(n-j==16)
        for(int i=0;i<n;i+=4) micro16(a+i*n,b+j*n,c+i*n+j,n);
    if(n-j==8)
        for(int i=0;i<n;i+=4) micro8(a+i*n,b+j*n,c+i*n+j,n);
}

template<bool Sub>
static inline void combine(U *d,const U *a,const U *b,int len) {
    for(int i=0;i<len;i+=16) {
        __m256i x=ld(a+i),y=ld(b+i);
        st(d+i,Sub?_mm256_sub_epi16(x,y):_mm256_add_epi16(x,y));
    }
}

// At the root, the three completed packed quadrants can be sent straight
// to their final row-major destinations rather than materialized in pc.
template<int N>
static __attribute__((noinline)) void combine3_unpack(
    U *d12,U *d21,U *d22,
    const U *p6,const U *p7,const U *p5,
    const U *p1,const U *p4,const U *p3,int stride) {
    if constexpr (N > BASE) {
        constexpr int h=N/2,q=h*h;
        combine3_unpack<h>(d12,d21,d22,p6,p7,p5,p1,p4,p3,stride);
        combine3_unpack<h>(d12+h,d21+h,d22+h,p6+q,p7+q,p5+q,p1+q,p4+q,p3+q,stride);
        combine3_unpack<h>(d12+h*stride,d21+h*stride,d22+h*stride,
                           p6+2*q,p7+2*q,p5+2*q,p1+2*q,p4+2*q,p3+2*q,stride);
        combine3_unpack<h>(d12+h*stride+h,d21+h*stride+h,d22+h*stride+h,
                           p6+3*q,p7+3*q,p5+3*q,p1+3*q,p4+3*q,p3+3*q,stride);
    } else {
        for(int row=0;row<N;++row) {
            const int off=row*N;
            U *o12=d12+(size_t)row*stride;
            U *o21=d21+(size_t)row*stride;
            U *o22=d22+(size_t)row*stride;
            for(int col=0;col<N;col+=16) {
                int idx=off+col;
                __m256i u2=_mm256_add_epi16(ld(p1+idx),ld(p6+idx));
                __m256i u3=_mm256_add_epi16(u2,ld(p7+idx));
                __m256i v5=ld(p5+idx);
                __m256i c12=_mm256_add_epi16(_mm256_add_epi16(u2,v5),ld(p3+idx));
                __m256i c21=_mm256_sub_epi16(u3,ld(p4+idx));
                __m256i c22=_mm256_add_epi16(u3,v5);
                _mm256_stream_si256((__m256i*)(o12+col),c12);
                _mm256_stream_si256((__m256i*)(o21+col),c21);
                _mm256_stream_si256((__m256i*)(o22+col),c22);
            }
        }
    }
}

// Add the surviving P1 into the root C11 product while mapping its packed
// quadrant layout to the final row-major C11 cells.
template<int N>
static __attribute__((noinline)) void unpack_add(U *d,const U *p2,const U *p1,int stride) {
    if constexpr(N>BASE) {
        constexpr int h=N/2,q=h*h;
        unpack_add<h>(d,p2,p1,stride);
        unpack_add<h>(d+h,p2+q,p1+q,stride);
        unpack_add<h>(d+h*stride,p2+2*q,p1+2*q,stride);
        unpack_add<h>(d+h*stride+h,p2+3*q,p1+3*q,stride);
    } else {
        for(int row=0;row<N;++row) {
            U *o=d+(size_t)row*stride;
            int off=row*N;
            for(int col=0;col<N;col+=16)
                _mm256_stream_si256((__m256i*)(o+col),
                    _mm256_add_epi16(ld(p2+off+col),ld(p1+off+col)));
        }
    }
}

// Strassen-Winograd: seven products, fifteen additions, two temporaries.
// All operations are in Z/(2^16); truncation never changes the answer.
template<int N>
static __attribute__((noinline)) void mul(const U *a,const U *b,U *c,U *work) {
    if constexpr (N<=BASE) {
        base(a,b,c,N);
    } else {
    constexpr int h=N/2,q=h*h;
    const U *a11=a,*a12=a+q,*a21=a+2*q,*a22=a+3*q;
    const U *b11=b,*b12=b+q,*b21=b+2*q,*b22=b+3*q;
    U *c11=c,*c12=c+q,*c21=c+2*q,*c22=c+3*q;
    U *x,*y,*next;
    if constexpr (N >= 1024) {
        /* s4x_: FUSED combine.  Judged per-layer bench (probe, 8 passes over q elements):
           q=4M 10.69->9.51, q=1M 10.16->8.79, q=256K 7.76->6.11 (fused wins 11/13.5/21%);
           q=65K 3.20->3.87, q=16K 2.63->3.75 (fused loses) => fuse exactly the top three
           levels.  Each of the eight quadrant sources is read once and all eight versions
           are written once (8R+8W) instead of 16R+8W over eight chained passes; the versions
           are needed simultaneously anyway (each feeds its own sub-product), so the pass
           chain disappears with no change to any arithmetic.  P4/P1 write into x4/y1, which
           their sub-products (P3/P7) have already consumed. */
        U *x1=work,*y1=work+q,*x2=work+2*q,*y2=work+3*q,
          *x3=work+4*q,*y3=work+5*q,*x4=work+6*q,*y4=work+7*q;
        next=work+8*q;
        for(int i=0;i<q;i+=16) {
            __m256i a11v=ld(a11+i),a12v=ld(a12+i),a21v=ld(a21+i),a22v=ld(a22+i);
            __m256i b11v=ld(b11+i),b12v=ld(b12+i),b21v=ld(b21+i),b22v=ld(b22+i);
            /* combine<true> is the SUBTRACT variant (Sub?sub:add): match the source
               exactly -- x1=a11-a21, y1=b22-b12, x2=a21+a22, y2=b12-b11,
               x3=x2-a11, y3=b22-y2, x4=a12-x3, y4=y3-b21. */
            __m256i u1=_mm256_sub_epi16(a11v,a21v), v1=_mm256_sub_epi16(b22v,b12v);
            __m256i u2=_mm256_add_epi16(a21v,a22v), v2=_mm256_sub_epi16(b12v,b11v);
            __m256i u3=_mm256_sub_epi16(u2,a11v),    v3=_mm256_sub_epi16(b22v,v2);
            st(x1+i,u1); st(y1+i,v1);
            st(x2+i,u2); st(y2+i,v2);
            st(x3+i,u3); st(y3+i,v3);
            st(x4+i,_mm256_sub_epi16(a12v,u3));
            st(y4+i,_mm256_sub_epi16(v3,b21v));
        }
        mul<h>(x1,y1,c21,next);                 // P7
        mul<h>(x2,y2,c22,next);                 // P5
        mul<h>(x3,y3,c12,next);                 // P6
        mul<h>(x4,b22,c11,next);                // P3
        mul<h>(a22,y4,x4,next);                 // P4  (x4 free after P3)
        mul<h>(a11,b11,y1,next);                // P1  (y1 free after P7)
        x=x4; y=y1;
    } else {
    x=work; y=work+q; next=work+2*q;
    combine<true>(x,a11,a21,q);
    combine<true>(y,b22,b12,q);
    mul<h>(x,y,c21,next);                       // P7
    combine<false>(x,a21,a22,q);
    combine<true>(y,b12,b11,q);
    mul<h>(x,y,c22,next);                       // P5
    combine<true>(x,x,a11,q);
    combine<true>(y,b22,y,q);
    mul<h>(x,y,c12,next);                       // P6
    combine<true>(x,a12,x,q);
    mul<h>(x,b22,c11,next);                     // P3
    combine<true>(y,y,b21,q);
    mul<h>(a22,y,x,next);                       // P4
    mul<h>(a11,b11,y,next);                     // P1
    }
    if constexpr (N == 4096) {
        combine3_unpack<2048>(final_out+2048,
            final_out+(size_t)2048*4096,
            final_out+(size_t)2048*4096+2048,
            c12,c21,c22,y,x,c11,4096);
    } else {
        for(int i=0;i<q;i+=16) {
            __m256i u2=_mm256_add_epi16(ld(y+i),ld(c12+i));
            __m256i u3=_mm256_add_epi16(u2,ld(c21+i));
            __m256i p5=ld(c22+i);
            st(c12+i,_mm256_add_epi16(_mm256_add_epi16(u2,p5),ld(c11+i)));
            st(c21+i,_mm256_sub_epi16(u3,ld(x+i)));
            st(c22+i,_mm256_add_epi16(u3,p5));
        }
    }
    mul<h>(a12,b21,c11,next);                   // P2
    if constexpr (N == 4096) unpack_add<2048>(final_out,c11,y,4096);
    else combine<false>(c11,c11,y,q);
    }
}

void matrix_multiply(int n,const short *A,const short *B,short *C) {
    pack(pa,(const U *)A,n,n,false);
    pack(pb,(const U *)B,n,n,true);
    final_out=(U *)C;
    mul<4096>(pa,pb,pc,ws);
    _mm_sfence();
}

评测结果