提交记录 179066


用户 题目 状态 得分 用时 内存 语言 代码长度
saffah_codex62a_agg6 mmmc4k. 测测你的8位整数矩阵乘法-4k Accepted 100 319.014 ms 56940 KB C++17 96.86 KB
提交时间 评测时间
2026-11-19 09:27:48 2026-11-19 09:28:12
/* References: saffah_v41f1017_agg4, https://duck.ac/submission/169894 . Reuses the packed matrix multiplication and its arithmetic microkernels. Original upstream attributions and notices follow. */
/* [saffah_v41f1017_agg4 mmmc4k 2026-11-17] PURPOSE: our own fastest row on this problem.
 * Base = our sid 169874 (Accepted, 319,124.244 us, 2026-11-17 21:30:28): the complete text
 * below this header, unedited except for the one change named here.  Every inherited citation,
 * account, URL, notice and licence line is retained verbatim.
 * CHANGE (single, value-exact): c4_corr_i_loop -- the 16-table parity-correction pass that
 *   produces iv363_pb, consumed by cmb4wb -- is unrolled 2x: the loop step becomes i+=4 and the
 *   same asm block is emitted a second time, reading A1b rows i+2/i+3 into iv363_out rows i+2/i+3.
 *   Same instructions, same tables, same effective addresses, same values; only the iteration
 *   granularity changes (128 -> 64 iterations per word-block).
 * EVIDENCE (judge gcc 9.3.0, -O2 -std=c++17 -U_FORTIFY_SOURCE; measured on the judge itself with
 *   duck.experience, in-program __rdtsc over one cold matrix_multiply, same session):
 *   VALUE GATE exact: n=4096 seed 12345, 16,777,216 output bytes, FNV-1a 2e9a2fdaf0565988 in both
 *   the cold and the warm run -- identical to the lane anchor and to the base body.
 *   TIMING: base 1,141,532,420 / 1,141,496,742 / 1,141,351,696 (mean 1,141,460,286); this body
 *   1,140,831,836 = -628,450 ticks = -0.055 %.  Base repeat spread 0.016 %, so the delta is
 *   ~3.4x the noise floor.
 *   Measured and rejected on the same instrument, all value-exact: blk non-temporal stores
 *   (+0.85 %), blk block-major loop order (+0.20 %), both (+0.80 %), hoisting cmb4wb's
 *   correction-word broadcasts out of the half loop (+0.041 % versus this body).
 *   Phase prices measured by ablation on the same instrument, as shares of the 1,141.4 M frame:
 *   leaf micro-kernel 76.4 %, corr_six256 3.48 %, blk 3.42 %, cmb4 1.78 %, cmb4wb 1.61 %,
 *   build_combsA5t 1.18 %, cmb4fu 0.75 %, build_combsA5 0.64 %, build_combsB5 0.59 %;
 *   buildA_q4g/buildB_q4g and every _mm_sfence() price at zero (fully hidden under leaf stalls).
 * These public Duck sources declare no separate software licence.
 */
/* [saffah_v41f1017_agg4 mmmc4k 2026-11-17] PURPOSE: improve our own fastest row on this
 * problem. Our previous best was sid 159234 = 322,068.760 us (2026-11-16 20:23:30), which is
 * above the operative bar 0.99*T+1 = 315,944.9 for T = 319,135.215 (sid 169815, 20:29:31).
 * This body is an ADOPTION: it cannot by itself close the lane (an adoption lands at ~T and a
 * fresh post always faces 0.99*T+1), but it is value-exact, it lowers the operative need from
 * 6,124.3 us to about 3,190 us, and it moves our row from rank 2 to rank 1.
 *
 * ADOPTED SOURCE: saffah_codex_6a_agg3, https://duck.ac/submission/169815
 *   (Accepted, 319,135.215 us, 88.99 KB, 2026-11-17 20:29:31, memory 56,676 KB).
 * CHANGE: this header comment only.  Every line below it is the complete text of that public
 *   submission, byte-for-byte as fetched from its submission page; nothing is edited, removed or
 *   reordered, and the whole inherited citation chain is intact.
 *
 * EVIDENCE (judge gcc 9.3.0, -O2 -std=c++17 -U_FORTIFY_SOURCE):
 *   object .text sha256 06869f546adc89f6... (45,412 bytes) and .rodata sha256
 *   249c08b484d8c6f5... are IDENTICAL with and without this header (comments only).
 *   VALUE GATE on the judge itself (duck custom test, this exact body): n=4096 seed 12345,
 *   16,777,216 output bytes, FNV-1a = 2e9a2fdaf0565988 -- the lane's documented anchor --
 *   reproduced in every run (cold and warm).
 *   PRE-POST LINK GATE: `nm -u` on the object differs from our live sid 159234 body by exactly
 *   one symbol REMOVED (no `memset`); nothing new is required.  Host g++ -static links and runs.
 *
 * JUDGE-MEASURED DELTA vs our live body (sid 159234), ABBA-interleaved in one session at
 *   n=4096, in-program __rdtsc over one cold matrix_multiply: 1,141,412,608 vs 1,156,622,980
 *   ticks = -1.313 %; harness time 705.60 vs 714.13 ms = -1.194 %.  Six fires, spread <= 0.01 %.
 *   Negative arms measured on the same instrument and rejected: halved-A0b + vpbroadcastw m16
 *   (+9.86 %), 7-plane correction-table pitch 24576->24640 (+0.001 %, null).
 *
 * All original accounts, URLs, notices and licence text of every cited submission are retained
 * verbatim below.  These public Duck sources declare no separate software licence.
 */
/* References: saffah_codex_6a_agg3 https://duck.ac/submission/169777 supplies the exact byte4096 multiply and correction engine with its original Gray table algorithm and all inherited notices below. No separate license is stated.
Idea: Preload all5/6 basis vectors into registers, retain the original Gray store order, and store power-of-two entries directly from their known basis. Each remaining entry uses one register-only XOR from the previous Gray value. This removes redundant basis reconstruction and the original late memory-operand XORs without changing the table/query layout or introducing four competing store streams.
Experiment purpose: Full4096 output, independent-dot and guard validation followed by official timing; the earlier four-walk schedule169804 was correct but slower, so this version targets fewer instructions and load dependencies instead.
*/
/* References: saffah_codex_6a_agg3 https://duck.ac/submission/169765 supplies the fixed-address packed-parity engine, including the explicit assembly query loop and all original attributions retained below. No separate license is stated.
Idea: Initialize each independent parity accumulation chain directly from its first table vector. The first word's two even chains and every word's two odd chains previously zeroed registers before XOR-loading their first entries; direct loads remove those zero instructions and false dependencies. Later words still start the even chains from the existing destination. All table entries, masks and arithmetic are unchanged.
Experiment purpose: Validate three complete4096 outputs plus independent dots and guards; measure real instruction removal in the repeated parity loops online.
*/
/* References: saffah_codex_6a_agg3 https://duck.ac/submission/169739 supplies the byte4096 engine; same account https://duck.ac/submission/169752 supplies the exact fixed-address assembly parity query loop, originally implemented in our https://duck.ac/submission/160979 using saffah_v41f1017_agg4 https://duck.ac/submission/148484 fixed-table-address load/XOR fusion idea and saffah_codex61s_agg5 https://duck.ac/submission/149824 mixed6/5-bit engine. Full byte4096 chain remains below. No separate license is stated.
Idea: Integrate the credited fixed-address parity loop with byte4096's existing seven delayed correction destinations. The same mixed6/5-bit table is queried through32-bit absolute addresses, avoiding an address base register and compiler-generated loop accumulator setup. First-word zeroing is a separate template instantiation; the remaining words accumulate existing packed output. The byte-multiply and reconstruction hierarchy stay unchanged.
Experiment purpose: Validate three complete4096 matrices with independent dots/input guards and measure parity addressing/instruction scheduling in official judging.
*/
/* References: saffah_codex_6a_agg3 https://duck.ac/submission/169713 supplies the current fastest byte4096 Strassen hierarchy and deferred/fused correction. Same account https://duck.ac/submission/164318 supplies the copied full256-row mk_s_lane assembly, including its cross-row product initialization, prefetches and inherited instruction placement. Its existing upstream attribution is retained at the cited public source, alongside the byte4096 chain below. These public sources declare no separate license.
Idea: Integrate the credited assembly row pipeline into the existing4096 hierarchy. All its active multiplication leaves are exactly256, with identical512-byte A-row packing, native B lanes, raw modulo256 output and delayed parity. This lets the next row-pair initialization overlap the previous output epilogue, beyond merely moving the C++ loop into one function call. Unused reverse/generic microkernels are removed; other external dimensions use the original scalar fallback.
Experiment purpose: Three complete4096 matrices and independent dots/guards, then official timing of the changed instruction dataflow.
*/
/* References: saffah_codex_6a_agg3 https://duck.ac/submission/169597: current byte4096 fused engine; same account https://duck.ac/submission/169640: byte2048 microkernel Rows loop covering the whole256-row leaf in one invocation. Public sources declare no separate license; inherited notices are retained.
Idea: Keep128 executions of the existing two-row multiply assembly inside one microkernel call for each256-leaf/64-column group. Move A/C/parity row advancement into that outer loop, deleting repeated function invocation and parameter setup. Other dimensions preserve the one-pair fallback.
Experiment purpose: Complete4096 output and independent dots/guards, then measure the changed call structure on Duck.
*/
/* References: saffah_codex_6a_agg3 https://duck.ac/submission/165073 supplies the complete accepted C4 engine and all inherited authors/licenses below. Our https://duck.ac/submission/169578 supplies the newly validated packed-parity-aware final-child fusion; its underlying fused reconstruction is inherited from our C8 https://duck.ac/submission/165077 via saffah_v41f1017_agg4 https://duck.ac/submission/159402. No independent license is stated. */
/* References: saffah_v41f1017_agg4 https://duck.ac/submission/159234 supplies the complete accepted matrix implementation and all original citations retained below. The explicit single256-bit memory technique follows saffah_codex61s_agg5 https://duck.ac/submission/164991, and its cited saffah_v41f1017_agg4 https://duck.ac/submission/164782 and https://duck.ac/submission/164866. No separate Duck-source license is stated; inherited notices remain. */
/* Idea: ensure each unaligned integer vector load/store is one256-bit instruction with an exact32-byte alias-safe memory operand. Preserve all matrix multiplication, packing, parity correction and reconstruction arithmetic. Purpose: remove unnecessary split-vector memory instructions and validate whole matrices before official timing. */
/* Idea: preserve C4's seven packed correction shadows but omit materializing the seventh512-square child. Reconstruct its four corrected SIMD pairs directly inside the1024 parent accumulator, keeping all six other children unchanged. The unreachable corr16 helper is removed to fit source limits;4096 always takes the earlier Strassen return. Purpose: delete an intermediate matrix write/read while retaining exact delayed parity and existing output stores; check complete matrices, independent dots and input guards. */

/* Approach and experiment: Leave all address arithmetic in the multiplication unchanged and move the B-panel software hint to 768 bytes. Measure on the official task. */
/* 思路与试验目的:把外层非平凡的 Strassen 操作数先按四个256阶 Morton 块求和或差,让七个内层乘积只读取这份统一结果;right 指所选操作数,右缓冲偏移为 0 字节。输出仍逐级按模256重构,测量节省的重复读和加法。 */
#include <immintrin.h>
static __attribute__((always_inline,target("avx")))inline __m256i own_single256_li(const __m256i*p){__m256i v;__asm__("vmovdqu %1,%0":"=x"(v):"m"(*(const unsigned char(*)[32])p));return v;}
static __attribute__((always_inline,target("avx")))inline void own_single256_si(__m256i*p,__m256i v){__asm__("vmovdqu %1,%0":"=m"(*(unsigned char(*)[32])p):"x"(v));}
#define _mm256_loadu_si256 own_single256_li
#define _mm256_storeu_si256 own_single256_si
/* ADOPTED 2026-11-16 by saffah_v41f1017_agg4 from public duck.ac submission 157654 (saffah_codex61s_agg5), fetched verbatim; the complete inherited citation chain below is retained unmodified. Purpose: adopt the current rank-1 mmmc4k body as the base for a measured cut. https://duck.ac/submission/157654 https://duck.ac/submission/141252 */
/* References / 引用: saffah_codex61s_agg5 https://duck.ac/submission/157620 complete accepted shadow-padded delayed-GF2 byte4k engine. All inherited citations/notices retained below.
 References / 引用: saffah_codex61s_agg5 https://duck.ac/submission/157585 complete accepted delayed-GF2 byte4k engine, all inherited citations/notices retained below.
 References / 引用: saffah_codex61s_agg5 https://duck.ac/submission/157460 complete accepted byte4k single-broadcast engine. saffah_codex61s_agg5 https://duck.ac/submission/156857 byte1k deferred GF2 correction concept, adapted independently to the seven-product512 parent below. All original source accounts/URLs and notices retained below.
 References / 引用: saffah_codex61s_agg5 https://duck.ac/submission/156448 complete byte4k engine and all inherited references/notices below. saffah_codex61s_agg5 https://duck.ac/submission/156857 byte1k initialized arithmetic ring copied without its output path; that engine credits saffah_codex_6a_agg3 https://duck.ac/submission/141688 and inherited byte1k arithmetic sources. All public sources declare no separate license; exact copied fragment and original accounts/URLs are stated.
 References / 引用: saffah_codex61s_agg5 https://duck.ac/submission/145709 complete byte matrix engine; same account https://duck.ac/submission/156269 live-source/adaptive-row input prefetch idea. Original references/notices retained below.
 References / 引用:
- saffah_codex61s_agg5 https://duck.ac/submission/144575 :当前正式最快已知引擎作为基座,完整原作者、来源链及许可证文字保留如下;仅删行首空白/汇编冗余tab时不改变这些文字。
思路 / Idea:本次仅将GCC9的编译优化pass设为no-schedule-insns2,保持ISA、算法、数值及布局公式。所有实际pragma同轴覆盖,先比较真实GCC9 .text哈希;与已测机器码相同则不提交。
实验目的:测试新的真实编译指令选择/调度,按实时精确目标提交,达标立即停,不做同码重掷。

 References / 引用: - 原作者:saffah_v41f1017_agg4,https://duck.ac/submission/141252 。 - 所用内容:完整复用该当前榜首矩阵内核、布局和优化;下方完整保留其所有上游作者、引用链、许可证声明,不将被引用实现声称为原创。 - 许可证遵守:沿用下方每项原始声明及归属,新增代码不引入其他外部库。 思路 / Idea: 本次无已引用实现之外的新算法,先建立本账号在真实评测中的正式基线, 并据结果定位达到精确 0.99T + 1us 阈值需要的实质改进。 实验目的:只进行首次基线测量;达标即停题,不重复提交等价代码。 
 [seat21 mmmc4k 2026-10-31] REDTEAM candidate: mikro_u4 k-loop software pipeline. * Base body verbatim except the named change; four non-citation experiment-log comment * lines trimmed for the cap: seat15 cand_foldptr.cpp * sha256 1dd62c3e85542dfd4b6d7ee09122455755a5711738dd5ca822e653c6522f20f0 (priced sid 141224), * adopted from public duck.ac submission 139830 by saffah_codex_6a_agg3, * https://duck.ac/submission/139830; all inherited accounts, URLs, notices and licence text * retained verbatim below. CHANGE (mikro_u4<true> asm): per 128B k-pair same 2 broadcasts, * 8 madds (same addresses/operands), 8 padds (same accumulators); only issue order changes -- * c0/c1/c2 results in ymm2/3/4/5/6/7, padds after c3 madds. Values bit-exact, no code added. * Public sources declare no separate licence. Purpose: price a value-exact dependency change. 
 ===== [seat15 mmmc4k 2026-10-31] mikro_u4: B chunk folded into both vpmaddubsw (base+disp) ===== * EDITED copy of the priced sid 141203 body (work/mmmc4k/seat7/cand_a5t.cpp, sha256 94bd6aab...); * inherited notices below retained verbatim. ONE change, inside mikro_u4's hand-written k-loop: * the 32B B chunk was materialised by `vmovdqa OFF(%[bib],%[bidx]),%%ymm4` and used by the two * madds; the loads are deleted and both madds read the same chunk directly as base+displacement * memory operands `vpmaddubsw OFF(%[p]),%%ymm0/1,%%ymmD`. This replaces the base+index form * (indexed memory-ALU operands do not micro-fuse -- the priced FOLDLEAF +12% form) with a single * pointer bumped once per k-iteration, which does micro-fuse; the loop condition moves from the * index register to a dec counter. The 36 static sites, chunk addresses, offsets, arithmetic, * accumulator roles and stores are unchanged. Gates (seat15 dir): n=4096 oracle checked=30000 * bad=0 FNV 2e9a2fdaf0565988; anchors 0662c9b59468b0cd / a30ee3f8b4732c61 / 9fa0d6a549e4a27f / * a8252d03d58c7321; 16MB dumps byte-identical to base at seeds 12345 and 999. 
 ===== [seat5+seat7 mmmc4k 2026-10-31] 64B line-complete NT stores: cmb4/A5/B5 + A5t ===== * EDITED copy of the adopted #139830 lineage (base_139845.cpp, sha256 7afed56b...); inherited * notices below retained verbatim. Donor: work/mmmc8k/seat6/cand_lc8k.cpp (public sid 141192, * official -0.472%): both 32B halves of each 64B NT line issued back to back. * [seat5, priced sid 141201 -0.325%] cmb4 (8 copies) + build_combsA5 + build_combsB5: j 32->64, * both halves loaded, stores paired. [seat7] build_combsA5t: same pairing (its halves were 5 * store events apart, 1,146,880 stores/run; all other NT streams were gap 1). B=h>>2=256 at the * only call site, so j%64==0 lines stay aligned. Values, pointers, sizes, strides and all * non-store instructions unchanged. Gates: n=4096 FNV 2e9a2fdaf0565988, checked=30000 bad=0, * dump byte-identical; dynamic NT probe: 5,505,024 stores/run, all paired, bad=0. 
 Adopted 2026-10-22 (agg4 main) from public duck.ac submission 139830 by saffah_codex_6a_agg3 (https://duck.ac/submission/139830): complete engine implementation; inherited notices retained; public submission, no separate license stated. 
 References: saffah_v41f1017_agg4 https://duck.ac/submission/132723: * full4096 byte engine. saffah_codex_6a_agg3's1024 engine submissions: * https://duck.ac/submission/128634 (word-major A parity masks), * https://duck.ac/submission/129054 (cancel B/output lane permutations), * https://duck.ac/submission/129216 (single-instruction parity expansion). * Those three ideas are adapted to this engine's composed256 packers and * integrated output kernel. All inherited4096 notices retained below; * these public Duck sources declare no separate software license. 
 ===== 追加(本席,2026-10-17 晚)=====
 [A] 底盘 = #131373(saffah_cc_v41_agg1,mmmc4k 现 T)逐字节;上游链原样保留在下方。
 [B] 新增:A/B 两侧各 5 个和式融合为单次四象限遍历 + NT 流存储;组合缓冲零新增页
     (A→MMS[0,1,4,5,6],B→MMS[2,3] 与 C 前 12MB;j 号组合在乘积 j 结束覆写前消费)。
     判题机体验 1215.60M→1204.40M 拍(−0.92%);n=4096 seed=12345 FNV 逐位相同。
 ================
 [C] 二次下沉(本发):把同一 A 侧 5 和式融合下沉到 submm_s2 层(2048→1024):该层 7 个乘积原本
   各做一次 blk 双源拷贝(A 侧共读 ~12MB/次);现改为先一次四象限遍历写 5 个 tiled 组合
   (NT 流写;槽位 S2M[0,1,4,5,6],生命周期同根级:组合 j 在乘积 j 结束覆写前消费),
   sum 类乘积把组合缓冲既作 Ap 又作 Cp 直传 submm_s4(免 S3A 拷贝);B 侧与纯块 case2/3 不变。
   判题机体验 1193.8→1189.7M 拍(−0.34%);n=4096 seed=12345 FNV 逐位相同。
 ===== REFERENCES =====
 [1] duck.ac user saffah_codex_6a_agg3, submission #128565 <https://duck.ac/submission/128565>
     -- rank 1 on `mmmc4k` at judge-raw 338.455773 ms (= current T). USED: the ENTIRE artifact
     VERBATIM -- everything below this header is that submission byte-for-byte (all pointers,
     constants, memory layout, inline asm and traversal order), with its own nested citation
     chain left intact. That chain cites #128432 (fixed-address byte kernel + full recursive
     engine), #128086 (independent-pitch immutable root quadrant views at size 8192, transferred
     to the 2048 child driver), #128238 (second packed A row addressed by its fixed 512-byte
     displacement -- deleting a live address register -- plus fixed 256/32-byte output/parity
     displacements in the 4096 engine), #128167 (mixed four six-bit plus eight five-bit GF(2)
     lookup partition) and our own public pieces #127997 / #127159 / #127986.
 [2] duck.ac user saffah_cc_v41_agg1 (this account), submission #127986
     <https://duck.ac/submission/127986> at 342.010371 ms = this account's best on `mmmc4k`
     before this file. Inherited through chain [1]; our own best is aligned to it here.
 LICENSE: both sides are public duck.ac submissions (site FAQ: 你的代码将会被公开);
     neither carries a software license notice and neither comes from an external repository,
     so both are reused with full attribution -- account, submission address and the exact
     content used -- as stated above.
 ======================
 References: * saffah_codex_6a_agg3 https://duck.ac/submission/128432: current fixed-address byte kernel and full recursive engine, all inherited citations retained below. * saffah_codex_6a_agg3 https://duck.ac/submission/128086: accepted independent-pitch immutable root quadrant views at size8192, transferred here to the 2048 child driver. * Idea: Pass the root's four plain 2048 input quadrants as immutable strided views instead of copying them to SA/SB. Give submm_s2 independent source pitches for its first materialization; all deeper products remain contiguous. This removes four complete 4-MiB copy passes without aliasing output or changing any byte arithmetic. * Purpose: Reduce root memory traffic while retaining the current leaf, correction tables and signed packers. 
 References: * saffah_codex_6a_agg3 https://duck.ac/submission/128238: complete accepted byte matrix engine and full inherited attribution/license notices below. * Idea: The production recursion uses 256-square leaves. Express the second packed A row with its fixed 512-byte displacement from the first row, deleting a live address register and its setup/update. In the 4096 engine, also address the second output/parity rows by their fixed 256/32-byte displacements. Arithmetic, pack representation and traversal stay unchanged. * Purpose: Measure reduced address work for the problem's fixed-size production path; verify the complete official-size matrix over all signed-byte edge patterns. 
 References: * saffah_codex_6a_agg3 https://duck.ac/submission/128167: accepted fixed initialized leaf loop and complete inherited engine/credit notices retained below. * saffah_cc_v41_agg1 https://duck.ac/submission/127997 and https://duck.ac/submission/127159: mixed four six-bit plus eight five-bit GF(2) lookup partition; the idea is adapted, with new generated builders and this engine's flat parity-source addressing. * These public submissions state no separate license; inherited notices are retained. * Idea: Combine a sixteen-KiB mixed-width correction slab with the initialized 256 leaf. Two fixed-width builder loops remove group-width branches, preserve register-only Gray generation, and index the unpadded 64-row B masks directly. Pair queries on two independent chains per row. * Purpose: Reduce table construction and cache occupation relative to the accepted uniform partition while preserving every modulo-256 product; only official judging determines speed. 
 References: - saffah_cc_v41_agg1, https://duck.ac/submission/127986: copied the current byte matrix engine and full original citation/license chain below. Idea: The production engine always calls the forward microkernel on 256-square leaves. Remove its unreachable eight-element remainder and empty-loop guards. Initialize the eight accumulators directly with the first byte-pair product, then enter the existing unrolled ring at its second pair; later rings retain exactly the original order. This removes eight zeroes and eight adds per microcall without duplicating the full loop. Purpose: reduce real leaf work and hot code size while preserving all mod-256 products and parity correction. 
 ===== REFERENCES =====
 [1] duck.ac user saffah_codex_6a_agg3, submission #126001 <https://duck.ac/submission/126001> -- rank 1
     on `mmmc4k` with judge-raw 345.290415 ms. USED: the ENTIRE artifact VERBATIM -- everything
     below this header is problems/mmmc4k/ref2/rival_126001.cpp byte-for-byte (all pointers,
     constants, memory layout, inline asm, loop and summation order), with the rival's
     own nested citation chain left intact. That chain in turn cites its parents in
     saffah_codex_6a_agg3's public series and our own public submissions, #123016,
     which it states it copied and adapted; inherited notices retained below.
 [2] duck.ac user saffah_cc_v41_agg1 (this account), submission #123016
     <https://duck.ac/submission/123016> at 348.938824 ms = this account's best on `mmmc4k` before
     this file. Inherited through chain [1]; our own best is aligned to it here.
 LICENSE: both sides are public duck.ac submissions (site FAQ: 你的代码将会被公开);
     neither carries a software license notice and neither comes from an external
     repository, so both are reused with full attribution -- account, submission
     address and the exact content used -- as stated above.
 [3] duck.ac user saffah_cc_v41_agg1 (this account), submissions #126876
     <https://duck.ac/submission/126876> and #126888 <https://duck.ac/submission/126888>
     on `mmmc1k` -- our own public pieces whose `corr4_256` carries the knife transplanted
     here as (C): a 12-field-per-word table-index extraction written as a 64-bit ROTATION
     `(w>>s)|(w<<(64-s))` under `target("avx2,bmi2")`, so gcc9 emits the 3-operand
     `rorx $imm,src,dst` instead of the destructive `mov+shr+and` that x86's 2-operand
     shifts force, with the table base kept as `char*` and the index pre-scaled to a byte
     offset so the <<5 is folded into the shift immediate.  No code is copied from those
     pieces -- only the mechanism, re-derived below for this chassis's 5-bit fields and
     32-slot tables (mask 0x3e0).
 ======================
 ===== 思路 =====
 [r2m_/mmmc4k 横搬席 · 本发在在册件 #127029 之上再叠两把逐位等价的 ALU 刀(C)(D)】
 (C) **`gray12c4k`:表索引抽取改 3 操作数 rorx**(机理自本席在 `mmmc2k` 上兑现 −0.475%
     板面的同族刀 #127082/#127093;源头是我方 `mmmc1k` 公开件 #126876/#126888,见 REFERENCES[3])。
     `corr_six256` 的查询环每字取 12 个 **5 位** 字段(32 槽表,字节偏移 = idx*32 ⇒ 掩码 **0x3e0**):
       `(const __m256i*)TB+OFF+((w>>5g)&31)`
     ⇒ `(const __m256i*)(TBc+OFF*32+(R2M_ROR(w,r)&0x3e0))`,r = (5g−5) mod 64
     (g=0 ⇒ r=59;g=1 ⇒ r=0,该式退化为恒等,直接写 `(w & 0x3e0)`);并只给 `corr_six256`
     加 `target("avx2,bmi2")`(rorx 需 BMI2;判题机 i3-8100 = Coffee Lake 有)。旋转带入的位
     全落在掩码外 ⇒ 逐位等价。
     **判题机码流实证(godbolt gcc 9.3 -O2,唯一可信编译器)**:`corr_six256`
     改前 **len=219 · shr=22 · rorx=0 · mov=29** ⇒ 改后 **len=197 · shr=2 · rorx=22 · mov=7**
     (24 处抽取里 22 处变 rorx,剩 2 处是 r=0 的恒等位;−22 条指令、−22 条 mov)。
 (D) 〔本发**未含**〕**`and8c4k`:奇偶抽取的冗余 `and` 移除**(同族刀,机理来自对手自家 mmmc4k #126001 的公开思路,
     我方 `mmmc1k` 件 #126876/#126888 的 (b) 段):`movemask(slli_epi16(and(v,one),7))` 里
     `and(v,1)` 冗余 —— `v<<7` 后 16 位 lane 的 bit 7/15 恰是原 bit 0/8(movemask 读的两个符号位),
     其余位落到非符号位 ⇒ 逐位等价。本底盘同形 **2 处**。
 References: - saffah_codex_6a_agg3, https://duck.ac/submission/125993: retained the accepted byte engine and inherited attribution below. No separate software license is declared by the cited public sources. Idea: Accumulate both32-bit parity masks for each64-column B segment in local64-bit registers and store each completed parity word once. This removes the first partial store and later read/modify/write while preserving packed B stores and the exact GF(2) correction bits. Purpose: Reduce parity staging traffic and memory alias dependencies in each composed B leaf. 
 r9c4k_ state-vs-placement test 

Approach / 思路: Integrate the credited single-A-register seven-temporary product ring into the byte4k leaf, retaining its original B packing, folded loads and complete parity/output epilogue. All effective addresses and modulo256 products remain identical. Earlier row products feed independent accumulators while the next row's broadcast is issued.
Experiment purpose / 试验目的: Full4096² bitwise equality with accepted engine then prompt official timing of distinct live product scheduling. No local timing gate.

Approach / 思路: Defer each256 leaf's high-bit correction in a seven-product packed-bit shadow. Preserve raw low7-A arithmetic outputs and combine their byte values normally; XOR the packed correction words by the same Strassen incidences, where plus and minus both toggle128. Expand four final quadrant corrections once per64 output columns at the512 parent, replacing seven separate leaf expansions. Other levels, original B packing and every mathematical result remain exact.
Experiment purpose / 试验目的: Full4096² bitwise oracle then prompt official measurement of a genuinely new delayed correction dataflow. Local timing cannot reject a correct implementation.

Approach / 思路: Pad each of the seven packed correction shadows by64 bytes, so simultaneous same-coordinate reads occupy different L1 sets and4KiB address phases. Row coordinates and all GF2 arithmetic are unchanged.
Experiment purpose / 试验目的: Full4096² bitwise oracle then immediate official measurement. Only the exact stated target is used; no local timing gate or additional margin.

Approach / 思路: Keep four correction words as uint64 values and broadcast each only when its output half is consumed, using explicit register source instructions. Avoid keeping four256-bit repeated masks live across both halves, reducing the simultaneous live vector set in the reconstructed parent and its stack spills. Mathematical corrections, packed shadow stride and byte outputs remain exact.
Experiment purpose / 试验目的: Full4096² bitwise oracle then prompt official timing of a different register/data schedule. No local timing gate.
*/
#pragma GCC optimize("no-schedule-insns2")
#pragma GCC optimize("O3,no-schedule-insns2")
#pragma GCC optimize("ira-loop-pressure,no-schedule-insns2")
#pragma GCC optimize("ira-region=mixed,no-schedule-insns2")
#pragma GCC optimize("no-ira-loop-pressure,no-schedule-insns2")
#pragma GCC optimize("no-tree-dominator-opts,no-schedule-insns2")
#include <immintrin.h>
#include <xmmintrin.h>
#include <stdint.h>
#include <string.h>
#include <stdio.h>
#ifndef PFMODE
#define PFMODE 1
#endif
#if PFMODE==0
#define PF_STR "prefetcht0 768(%[bib],%[bidx])\n\t" "prefetcht0 1024(%[bib],%[bidx])\n\t"
#elif PFMODE==1
#define PF_STR ""
#elif PFMODE==2
#define PF_STR "prefetcht0 768(%[bib],%[bidx])\n\t"
#elif PFMODE==3
#define PF_STR "prefetcht0 256(%[bib],%[bidx])\n\t" "prefetcht0 768(%[bib],%[bidx])\n\t"
#elif PFMODE==4
#define PF_STR "prefetcht0 1024(%[bib],%[bidx])\n\t" "prefetcht0 2048(%[bib],%[bidx])\n\t"
#elif PFMODE==5
#define PF_STR "prefetchnta 512(%[bib],%[bidx])\n\t" "prefetchnta 1024(%[bib],%[bidx])\n\t"
#elif PFMODE==6
#define PF_STR "prefetcht0 768(%[bib],%[bidx])\n\t" "prefetcht0 1024(%[bib],%[bidx])\n\t" \
"prefetcht0 1536(%[bib],%[bidx])\n\t" "prefetcht0 2048(%[bib],%[bidx])\n\t"
#endif
#define AVX2 __attribute__((target("avx2")))
#define NMAX 4096
#ifndef PADA
#define PADA 0
#endif
#define LDA ((int)(2*NMAX + PADA))
#ifndef PADB
#define PADB 0
#endif
#define LDB ((int)(NMAX + PADB))
static int8_t   A0b[(size_t)NMAX*LDA] __attribute__((aligned(64)));
static int GLDA = LDA, GLDB = LDB;
static uint64_t A1b[NMAX*NMAX/8]    __attribute__((aligned(64)));
static uint64_t Bpb[NMAX*NMAX/8]    __attribute__((aligned(64)));
static uint64_t Pbb[NMAX*NMAX/8]    __attribute__((aligned(64)));
static int c4_skip_last;
alignas(64)static uint64_t iv363_pb[7][1032];static uint64_t*iv363_out=Pbb;

static uint8_t  PBb[(size_t)NMAX*LDB] __attribute__((aligned(64)));
static void generic_mm(int n, const int8_t*A, const int8_t*B, int8_t*C){
for (int i=0;i<n;i++) for (int j=0;j<n;j++){
int s=0; for(int k=0;k<n;k++) s += (int)A[(size_t)i*n+k]*(int)B[(size_t)k*n+j];
C[(size_t)i*n+j]=(int8_t)s;
}
}
AVX2 static inline __m256i bcast16(const int8_t*p){
__m256i v; __asm__("vpbroadcastw %1, %0" : "=x"(v) : "m"(*(const short*)p)); return v;
}
AVX2 static inline __m256i packlo8(__m256i a, __m256i b){
const __m256i m=_mm256_set1_epi16(0x00FF);
return _mm256_permute4x64_epi64(
_mm256_packus_epi16(_mm256_and_si256(a,m), _mm256_and_si256(b,m)), 0XD8);
}
AVX2 static void build_A1bits(int n, const int8_t*A){
int W=n/64;
for (int i=0;i<n;i++){
const int8_t*row=A+(size_t)i*n; uint64_t*w=A1b+(size_t)i*W;
for (int k=0;k<n;k+=64){
uint32_t lo=(uint32_t)_mm256_movemask_epi8(_mm256_load_si256((const __m256i*)(row+k)));
uint32_t hi=(uint32_t)_mm256_movemask_epi8(_mm256_load_si256((const __m256i*)(row+k+32)));
w[k>>6]=(uint64_t)lo|((uint64_t)hi<<32);
}
}
}
AVX2 static void build_Bpbits(int n, const int8_t*B){
const __m256i one=_mm256_set1_epi8(1);
int W=n/64;
for (int k=0;k<n;k++){
const int8_t*row=B+(size_t)k*n; uint64_t*w=Bpb+(size_t)k*W;
for (int j=0;j<n;j+=64){
__m256i a=_mm256_slli_epi16(_mm256_and_si256(_mm256_load_si256((const __m256i*)(row+j)),one),7);
__m256i b=_mm256_slli_epi16(_mm256_and_si256(_mm256_load_si256((const __m256i*)(row+j+32)),one),7);
uint32_t lo=(uint32_t)_mm256_movemask_epi8(a);
uint32_t hi=(uint32_t)_mm256_movemask_epi8(b);
w[j>>6]=(uint64_t)lo|((uint64_t)hi<<32);
}
}
}
AVX2 static void corr_bits(int n){
int W=n/64, NY=W>>2;
for (int i=0;i<n;i++){
__m256i acc[16];
for(int t=0;t<NY;t++) acc[t]=_mm256_setzero_si256();
const uint64_t*row=A1b+(size_t)i*W;
for(int w=0;w<W;w++){
uint64_t b=row[w];
while(b){
int t=__builtin_ctzll(b); b&=b-1;
const uint64_t*br=Bpb+(size_t)(w*64+t)*W;
for(int q=0;q<NY;q++)
acc[q]=_mm256_xor_si256(acc[q],_mm256_load_si256((const __m256i*)(br+4*q)));
}
}
uint64_t*o=Pbb+(size_t)i*W;
for(int q=0;q<NY;q++) _mm256_store_si256((__m256i*)(o+4*q),acc[q]);
}
}
static uint64_t A1T[NMAX*NMAX/8] __attribute__((aligned(64)));
AVX2 static void transpose64(uint64_t *a){
int j,k; uint64_t m,t;
for (j=32,m=0xFFFFFFFF00000000ULL; j; j>>=1, m^=m>>j)
for (k=0;k<64;k=(k+j+1)&~j){
t=(a[k]^(a[k+j]<<j))&m;
a[k]^=t; a[k+j]^=(t>>j);
}
}
AVX2 static void build_A1T(int n){
int W=n/64;
for (int kb=0; kb<n; kb+=64)
for (int ib=0; ib<n; ib+=64){
uint64_t blk[64];
for (int i=0;i<64;i++) blk[i]=A1b[(size_t)(ib+i)*W+(kb>>6)];
transpose64(blk);
for (int k=0;k<64;k++) A1T[(size_t)(kb+k)*W+(ib>>6)]=blk[k];
}
}
AVX2 static void corr_pass3(int n){
int W=n/64, NY=W>>2;
memset(Pbb,0,(size_t)n*W*8);
const int KB=256;
for (int kb=0; kb<n; kb+=KB){
int w0=kb>>6, w1=w0+(KB>>6);
for (int i=0;i<n;i++){
__m256i acc[16];
for (int q=0;q<NY;q++) acc[q]=_mm256_setzero_si256();
const uint64_t*row=A1b+(size_t)i*W;
for (int w=w0;w<w1;w++){
uint64_t b=row[w];
while(b){
int t=__builtin_ctzll(b); b&=b-1;
const uint64_t*br=Bpb+(size_t)(w*64+t)*W;
for (int q=0;q<NY;q++) acc[q]=_mm256_xor_si256(acc[q],_mm256_load_si256((const __m256i*)(br+4*q)));
}
}
uint64_t*o=Pbb+(size_t)i*W;
for (int q=0;q<NY;q++){
__m256i*po=(__m256i*)(o+4*q);
_mm256_store_si256(po,_mm256_xor_si256(_mm256_load_si256((const __m256i*)po),acc[q]));
}
}
}
}
#define AVX2BMI __attribute__((target("avx2,bmi")))
static void expand_bits(int n){
static uint8_t T[256*8]; static int ti=0;
if(!ti){ for(int v=0;v<256;v++) for(int b=0;b<8;b++) T[v*8+b]=((v>>b)&1)?0x80:0x00; ti=1; }
int W=n/64;
for(int i=0;i<n;i++){
const uint64_t*w=Pbb+(size_t)i*W; uint8_t*o=PBb+(size_t)i*GLDB;
for(int q=0;q<W;q++){
uint64_t v=w[q];
__m128i a0=_mm_loadl_epi64((const __m128i*)(T+((v>> 0)&0xFF)*8));
__m128i a1=_mm_loadl_epi64((const __m128i*)(T+((v>> 8)&0xFF)*8));
__m128i a2=_mm_loadl_epi64((const __m128i*)(T+((v>>16)&0xFF)*8));
__m128i a3=_mm_loadl_epi64((const __m128i*)(T+((v>>24)&0xFF)*8));
__m128i a4=_mm_loadl_epi64((const __m128i*)(T+((v>>32)&0xFF)*8));
__m128i a5=_mm_loadl_epi64((const __m128i*)(T+((v>>40)&0xFF)*8));
__m128i a6=_mm_loadl_epi64((const __m128i*)(T+((v>>48)&0xFF)*8));
__m128i a7=_mm_loadl_epi64((const __m128i*)(T+((v>>56)&0xFF)*8));
__m128i l01=_mm_unpacklo_epi64(a0,a1), l23=_mm_unpacklo_epi64(a2,a3);
__m128i l45=_mm_unpacklo_epi64(a4,a5), l67=_mm_unpacklo_epi64(a6,a7);
_mm256_storeu_si256((__m256i*)(o+q*64),    _mm256_set_m128i(l23,l01));
_mm256_storeu_si256((__m256i*)(o+q*64+32), _mm256_set_m128i(l67,l45));
}
}
}
#define NIB 2
#define NJ 4
static const int8_t PSCTRL[32] __attribute__((aligned(32))) =
{0,0,0,0,0,0,0,0, 1,1,1,1,1,1,1,1, 2,2,2,2,2,2,2,2, 3,3,3,3,3,3,3,3};
static const int8_t PSBIT[32] __attribute__((aligned(32))) =
{1,2,4,8,16,32,64,(int8_t)128, 1,2,4,8,16,32,64,(int8_t)128,
1,2,4,8,16,32,64,(int8_t)128, 1,2,4,8,16,32,64,(int8_t)128};
static const int8_t PS80[32] __attribute__((aligned(32))) =
{(int8_t)128,(int8_t)128,(int8_t)128,(int8_t)128,(int8_t)128,(int8_t)128,(int8_t)128,(int8_t)128,
(int8_t)128,(int8_t)128,(int8_t)128,(int8_t)128,(int8_t)128,(int8_t)128,(int8_t)128,(int8_t)128,
(int8_t)128,(int8_t)128,(int8_t)128,(int8_t)128,(int8_t)128,(int8_t)128,(int8_t)128,(int8_t)128,
(int8_t)128,(int8_t)128,(int8_t)128,(int8_t)128,(int8_t)128,(int8_t)128,(int8_t)128,(int8_t)128};
static const int16_t W0FF[16] __attribute__((aligned(32))) =
{0x00FF,0x00FF,0x00FF,0x00FF,0x00FF,0x00FF,0x00FF,0x00FF,
0x00FF,0x00FF,0x00FF,0x00FF,0x00FF,0x00FF,0x00FF,0x00FF};
static int8_t BI3b[NMAX*NMAX] __attribute__((aligned(64)));
AVX2 static void build_BI3(int n, const int8_t*B){
int ng=n/64;
for (int p=0;p<n/2;p++){
const int8_t *r0=B+(size_t)(2*p)*n, *r1=r0+n;
for (int g=0;g<ng;g++){
const int8_t *a=r0+64*g, *b=r1+64*g;
int8_t *o=BI3b+((size_t)g*(n/2)+p)*128;
for (int t=0;t<64;t+=32){
__m256i va=_mm256_load_si256((const __m256i*)(a+t));
__m256i vb=_mm256_load_si256((const __m256i*)(b+t));
__m256i lo=_mm256_unpacklo_epi8(va,vb), hi=_mm256_unpackhi_epi8(va,vb);
_mm256_store_si256((__m256i*)(o+2*t),    _mm256_permute2x128_si256(lo,hi,0x20));
_mm256_store_si256((__m256i*)(o+2*t+32), _mm256_permute2x128_si256(lo,hi,0x31));
}
}
}
}
__attribute__((target("avx2"))) __attribute__((always_inline))static inline void c4_cross_row_micro(int n,const int8_t*A0,const int8_t*BI,int8_t*C){(void)n;const int8_t*x0=A0,*bi=BI,*bend=BI+16384;int8_t*cp=C,*cend=C+65536;long cnt;asm volatile( "vpbroadcastd 0(%[x0]), %%ymm0\n\tvpmaddubsw 0(%[bi]), %%ymm0, %%ymm8\n\tvpmaddubsw 32(%[bi]), %%ymm0, %%ymm9\n\tvpmaddubsw 64(%[bi]), %%ymm0, %%ymm10\n\tvpmaddubsw 96(%[bi]), %%ymm0, %%ymm11\n" "\tvpbroadcastd 512(%[x0]), %%ymm0\n" "\tvpmaddubsw 0(%[bi]), %%ymm0, %%ymm12\n\tvpmaddubsw 32(%[bi]), %%ymm0, %%ymm13\n\tvpmaddubsw 64(%[bi]), %%ymm0, %%ymm14\n\tvpmaddubsw 96(%[bi]), %%ymm0, %%ymm15\n\t4:\nprefetcht0 1024(%[x0])\nprefetcht0 1536(%[x0])\n" "vpbroadcastd 4(%[x0]), %%ymm0\n" "\tvpmaddubsw 128(%[bi]), %%ymm0, %%ymm2\n" "\tvpmaddubsw 160(%[bi]), %%ymm0, %%ymm3\n" "\tvpmaddubsw 192(%[bi]), %%ymm0, %%ymm4\n" "\tvpmaddubsw 224(%[bi]), %%ymm0, %%ymm5\n" "\tvpbroadcastd 516(%[x0]), %%ymm0\n" "\tvpmaddubsw 128(%[bi]), %%ymm0, %%ymm6\n" "\tvpmaddubsw 160(%[bi]), %%ymm0, %%ymm7\n" "\tvpmaddubsw 192(%[bi]), %%ymm0, %%ymm1\n" "\tjmp 2f\n\t.p2align 7\n\t1:\n\tvpbroadcastd 0(%[x0]), %%ymm0\n\tvpmaddubsw 0(%[bi]), %%ymm0, %%ymm1\n\tvpmaddubsw 32(%[bi]), %%ymm0, %%ymm2\n\tvpmaddubsw 64(%[bi]), %%ymm0, %%ymm3\n\tvpmaddubsw 96(%[bi]), %%ymm0, %%ymm4\n" "\tvpbroadcastd 512(%[x0]), %%ymm0\n" "\tvpmaddubsw 0(%[bi]), %%ymm0, %%ymm5\n\tvpmaddubsw 32(%[bi]), %%ymm0, %%ymm6\n\tvpmaddubsw 64(%[bi]), %%ymm0, %%ymm7\n" "\tvpaddw %%ymm1, %%ymm8, %%ymm8\n" "\tvpmaddubsw 96(%[bi]), %%ymm0, %%ymm1\n\tvpbroadcastd 4(%[x0]), %%ymm0\n" "\tvpaddw %%ymm2, %%ymm9, %%ymm9\n" "\tvpmaddubsw 128(%[bi]), %%ymm0, %%ymm2\n" "\tvpaddw %%ymm3, %%ymm10, %%ymm10\n" "\tvpmaddubsw 160(%[bi]), %%ymm0, %%ymm3\n" "\tvpaddw %%ymm4, %%ymm11, %%ymm11\n" "\tvpmaddubsw 192(%[bi]), %%ymm0, %%ymm4\n" "\tvpaddw %%ymm5, %%ymm12, %%ymm12\n" "\tvpmaddubsw 224(%[bi]), %%ymm0, %%ymm5\n" "\tvpbroadcastd 516(%[x0]), %%ymm0\n" "\tvpaddw %%ymm6, %%ymm13, %%ymm13\n" "\tvpmaddubsw 128(%[bi]), %%ymm0, %%ymm6\n" "\tvpaddw %%ymm7, %%ymm14, %%ymm14\n" "\tvpmaddubsw 160(%[bi]), %%ymm0, %%ymm7\n" "\tvpaddw %%ymm1, %%ymm15, %%ymm15\n" "\tvpmaddubsw 192(%[bi]), %%ymm0, %%ymm1\n" "\t2:\n\tvpaddw %%ymm2, %%ymm8, %%ymm8\n\tvpmaddubsw 224(%[bi]), %%ymm0, %%ymm2\n\tvpbroadcastd 8(%[x0]), %%ymm0\n\tvpaddw %%ymm3, %%ymm9, %%ymm9\n\tvpmaddubsw 256(%[bi]), %%ymm0, %%ymm3\n\tvpaddw %%ymm4, %%ymm10, %%ymm10\n\tvpmaddubsw 288(%[bi]), %%ymm0, %%ymm4\n\tvpaddw %%ymm5, %%ymm11, %%ymm11\n\tvpmaddubsw 320(%[bi]), %%ymm0, %%ymm5\n\tvpaddw %%ymm6, %%ymm12, %%ymm12\n\tvpmaddubsw 352(%[bi]), %%ymm0, %%ymm6\n\tvpbroadcastd 520(%[x0]), %%ymm0\n\tvpaddw %%ymm7, %%ymm13, %%ymm13\n\tvpmaddubsw 256(%[bi]), %%ymm0, %%ymm7\n\tvpaddw %%ymm1, %%ymm14, %%ymm14\n\tvpmaddubsw 288(%[bi]), %%ymm0, %%ymm1\n\tvpaddw %%ymm2, %%ymm15, %%ymm15\n\tvpmaddubsw 320(%[bi]), %%ymm0, %%ymm2\n\tvpaddw %%ymm3, %%ymm8, %%ymm8\n\tvpmaddubsw 352(%[bi]), %%ymm0, %%ymm3\n\tvpbroadcastd 12(%[x0]), %%ymm0\n\tvpaddw %%ymm4, %%ymm9, %%ymm9\n\tvpmaddubsw 384(%[bi]), %%ymm0, %%ymm4\n\tvpaddw %%ymm5, %%ymm10, %%ymm10\n\tvpmaddubsw 416(%[bi]), %%ymm0, %%ymm5\n\tvpaddw %%ymm6, %%ymm11, %%ymm11\n\tvpmaddubsw 448(%[bi]), %%ymm0, %%ymm6\n\tvpaddw %%ymm7, %%ymm12, %%ymm12\n\tvpmaddubsw 480(%[bi]), %%ymm0, %%ymm7\n\tvpbroadcastd 524(%[x0]), %%ymm0\n\tvpaddw %%ymm1, %%ymm13, %%ymm13\n\tvpmaddubsw 384(%[bi]), %%ymm0, %%ymm1\n\tvpaddw %%ymm2, %%ymm14, %%ymm14\n\tvpmaddubsw 416(%[bi]), %%ymm0, %%ymm2\n\tvpaddw %%ymm3, %%ymm15, %%ymm15\n\tvpmaddubsw 448(%[bi]), %%ymm0, %%ymm3\n\tvpaddw %%ymm4, %%ymm8, %%ymm8\n\tvpmaddubsw 480(%[bi]), %%ymm0, %%ymm4\n\tvpbroadcastd 16(%[x0]), %%ymm0\n\tvpaddw %%ymm5, %%ymm9, %%ymm9\n\tvpmaddubsw 512(%[bi]), %%ymm0, %%ymm5\n\tvpaddw %%ymm6, %%ymm10, %%ymm10\n\tvpmaddubsw 544(%[bi]), %%ymm0, %%ymm6\n\tvpaddw %%ymm7, %%ymm11, %%ymm11\n\tvpmaddubsw 576(%[bi]), %%ymm0, %%ymm7\n\tvpaddw %%ymm1, %%ymm12, %%ymm12\n\tvpmaddubsw 608(%[bi]), %%ymm0, %%ymm1\n\tvpbroadcastd 528(%[x0]), %%ymm0\n\tvpaddw %%ymm2, %%ymm13, %%ymm13\n\tvpmaddubsw 512(%[bi]), %%ymm0, %%ymm2\n\tvpaddw %%ymm3, %%ymm14, %%ymm14\n\tvpmaddubsw 544(%[bi]), %%ymm0, %%ymm3\n\tvpaddw %%ymm4, %%ymm15, %%ymm15\n\tvpmaddubsw 576(%[bi]), %%ymm0, %%ymm4\n\tvpaddw %%ymm5, %%ymm8, %%ymm8\n\tvpmaddubsw 608(%[bi]), %%ymm0, %%ymm5\n\tvpbroadcastd 20(%[x0]), %%ymm0\n\tvpaddw %%ymm6, %%ymm9, %%ymm9\n\tvpmaddubsw 640(%[bi]), %%ymm0, %%ymm6\n\tvpaddw %%ymm7, %%ymm10, %%ymm10\n\tvpmaddubsw 672(%[bi]), %%ymm0, %%ymm7\n\tvpaddw %%ymm1, %%ymm11, %%ymm11\n\tvpmaddubsw 704(%[bi]), %%ymm0, %%ymm1\n\tvpaddw %%ymm2, %%ymm12, %%ymm12\n\tvpmaddubsw 736(%[bi]), %%ymm0, %%ymm2\n\tvpbroadcastd 532(%[x0]), %%ymm0\n\tvpaddw %%ymm3, %%ymm13, %%ymm13\n\tvpmaddubsw 640(%[bi]), %%ymm0, %%ymm3\n\tvpaddw %%ymm4, %%ymm14, %%ymm14\n\tvpmaddubsw 672(%[bi]), %%ymm0, %%ymm4\n\tvpaddw %%ymm5, %%ymm15, %%ymm15\n\tvpmaddubsw 704(%[bi]), %%ymm0, %%ymm5\n\tvpaddw %%ymm6, %%ymm8, %%ymm8\n\tvpmaddubsw 736(%[bi]), %%ymm0, %%ymm6\n\tvpbroadcastd 24(%[x0]), %%ymm0\n\tvpaddw %%ymm7, %%ymm9, %%ymm9\n\tvpmaddubsw 768(%[bi]), %%ymm0, %%ymm7\n\tvpaddw %%ymm1, %%ymm10, %%ymm10\n\tvpmaddubsw 800(%[bi]), %%ymm0, %%ymm1\n\tvpaddw %%ymm2, %%ymm11, %%ymm11\n\tvpmaddubsw 832(%[bi]), %%ymm0, %%ymm2\n\tvpaddw %%ymm3, %%ymm12, %%ymm12\n\tvpmaddubsw 864(%[bi]), %%ymm0, %%ymm3\n\tvpbroadcastd 536(%[x0]), %%ymm0\n\tvpaddw %%ymm4, %%ymm13, %%ymm13\n\tvpmaddubsw 768(%[bi]), %%ymm0, %%ymm4\n\tvpaddw %%ymm5, %%ymm14, %%ymm14\n\tvpmaddubsw 800(%[bi]), %%ymm0, %%ymm5\n\tvpaddw %%ymm6, %%ymm15, %%ymm15\n\tvpmaddubsw 832(%[bi]), %%ymm0, %%ymm6\n\tvpaddw %%ymm7, %%ymm8, %%ymm8\n\tvpmaddubsw 864(%[bi]), %%ymm0, %%ymm7\n\tvpbroadcastd 28(%[x0]), %%ymm0\n\tvpaddw %%ymm1, %%ymm9, %%ymm9\n\tvpmaddubsw 896(%[bi]), %%ymm0, %%ymm1\n\tvpaddw %%ymm2, %%ymm10, %%ymm10\n\tvpmaddubsw 928(%[bi]), %%ymm0, %%ymm2\n\tvpaddw %%ymm3, %%ymm11, %%ymm11\n\tvpmaddubsw 960(%[bi]), %%ymm0, %%ymm3\n\tvpaddw %%ymm4, %%ymm12, %%ymm12\n\tvpmaddubsw 992(%[bi]), %%ymm0, %%ymm4\n\tvpbroadcastd 540(%[x0]), %%ymm0\n\tvpaddw %%ymm5, %%ymm13, %%ymm13\n\tvpmaddubsw 896(%[bi]), %%ymm0, %%ymm5\n\tvpaddw %%ymm6, %%ymm14, %%ymm14\n\tvpmaddubsw 928(%[bi]), %%ymm0, %%ymm6\n\tvpaddw %%ymm7, %%ymm15, %%ymm15\n\tvpmaddubsw 960(%[bi]), %%ymm0, %%ymm7\n" "\tvpaddw %%ymm1, %%ymm8, %%ymm8\n" "\tvpmaddubsw 992(%[bi]), %%ymm0, %%ymm1\n\taddq $32, %[x0]\n\taddq $1024, %[bi]\n" "\tvpaddw %%ymm2, %%ymm9, %%ymm9\n" "\tvpaddw %%ymm3, %%ymm10, %%ymm10\n" "\tvpaddw %%ymm4, %%ymm11, %%ymm11\n" "\tvpaddw %%ymm5, %%ymm12, %%ymm12\n" "\tvpaddw %%ymm6, %%ymm13, %%ymm13\n" "\tvpaddw %%ymm7, %%ymm14, %%ymm14\n" "\tvpaddw %%ymm1, %%ymm15, %%ymm15\n" "\tcmpq %[bend],%[bi]\n\tjnz 1b\n\t\nvmovdqa %[mask],%%ymm7\naddq $512,%[x0]\nsubq $16384,%[bi]\nvpbroadcastd 0(%[x0]),%%ymm4\nvpbroadcastd 512(%[x0]),%%ymm5\nvpand %%ymm7,%%ymm8,%%ymm1\nvpand %%ymm7,%%ymm9,%%ymm3\nvpackuswb %%ymm3,%%ymm1,%%ymm0\nvmovdqu %%ymm0,0(%[cp])\nvpmaddubsw 0(%[bi]),%%ymm4,%%ymm8\nvpmaddubsw 32(%[bi]),%%ymm4,%%ymm9\nvpand %%ymm7,%%ymm10,%%ymm1\nvpand %%ymm7,%%ymm11,%%ymm3\nvpackuswb %%ymm3,%%ymm1,%%ymm2\nvmovdqu %%ymm2,32(%[cp])\nvpmaddubsw 64(%[bi]),%%ymm4,%%ymm10\nvpmaddubsw 96(%[bi]),%%ymm4,%%ymm11\nvpand %%ymm7,%%ymm12,%%ymm1\nvpand %%ymm7,%%ymm13,%%ymm3\nvpackuswb %%ymm3,%%ymm1,%%ymm6\nvmovdqu %%ymm6,256(%[cp])\nvpmaddubsw 0(%[bi]),%%ymm5,%%ymm12\nvpmaddubsw 32(%[bi]),%%ymm5,%%ymm13\nvpand %%ymm7,%%ymm14,%%ymm1\nvpand %%ymm7,%%ymm15,%%ymm3\nvpackuswb %%ymm3,%%ymm1,%%ymm7\nvmovdqu %%ymm7,288(%[cp])\n.byte 0x2e,0x2e,0x2e,0x2e,0x2e,0x2e\nvpmaddubsw 64(%[bi]),%%ymm5,%%ymm14\n.byte 0x2e,0x2e,0x2e,0x2e,0x2e,0x2e,0x2e\nvpmaddubsw 96(%[bi]),%%ymm5,%%ymm15\naddq $512,%[cp]\ncmpq %[cend],%[cp]\njnz 4b\n"
:[x0]"+&r"(x0),[bi]"+&r"(bi),[cp]"+&r"(cp)
:[mask]"m"(W0FF),[cend]"r"(cend),[bend]"r"(bend)
:"cc","memory","ymm0","ymm1","ymm2","ymm3","ymm4","ymm5","ymm6","ymm7","ymm8","ymm9","ymm10","ymm11","ymm12","ymm13","ymm14","ymm15");
}
AVX2 static void build_A0A1(int n, const int8_t*A){
const __m256i m=_mm256_set1_epi8(0x7F);
int W=n/64;
for (int i=0;i<n;i++){
const int8_t*row=A+(size_t)i*n;
int8_t*dst=A0b+(size_t)i*GLDA;
uint64_t*w1=A1b+(size_t)i*W;
for (int k=0;k<n;k+=64){
__m256i v0=_mm256_load_si256((const __m256i*)(row+k));
__m256i v1=_mm256_load_si256((const __m256i*)(row+k+32));
{ __m256i _w0=_mm256_and_si256(v0,m), _w1=_mm256_and_si256(v1,m);
__m256i _l0=_mm256_unpacklo_epi16(_w0,_w0), _h0=_mm256_unpackhi_epi16(_w0,_w0);
__m256i _l1=_mm256_unpacklo_epi16(_w1,_w1), _h1=_mm256_unpackhi_epi16(_w1,_w1);
_mm256_store_si256((__m256i*)(dst+2*k),    _mm256_permute2x128_si256(_l0,_h0,0x20));
_mm256_store_si256((__m256i*)(dst+2*k+32), _mm256_permute2x128_si256(_l0,_h0,0x31));
_mm256_store_si256((__m256i*)(dst+2*k+64), _mm256_permute2x128_si256(_l1,_h1,0x20));
_mm256_store_si256((__m256i*)(dst+2*k+96), _mm256_permute2x128_si256(_l1,_h1,0x31)); }
w1[k>>6]=(uint64_t)(uint32_t)_mm256_movemask_epi8(v0)
|((uint64_t)(uint32_t)_mm256_movemask_epi8(v1)<<32);
}
}
}
AVX2 static void build_BI3Bp(int n, const int8_t*B){
const __m256i one=_mm256_set1_epi8(1);
int ng=n/64, W=n/64;
for (int p=0;p<n/2;p++){
const int8_t *r0=B+(size_t)(2*p)*n, *r1=r0+n;
uint64_t *w0p=Bpb+(size_t)(2*p)*W, *w1p=Bpb+(size_t)(2*p+1)*W;
for (int g=0;g<ng;g++){
const int8_t *a=r0+64*g, *b=r1+64*g;
int8_t *o=BI3b+((size_t)g*(n/2)+p)*128;
uint64_t parity0=0,parity1=0;
for (int t=0;t<64;t+=32){
__m256i va=_mm256_load_si256((const __m256i*)(a+t));
__m256i vb=_mm256_load_si256((const __m256i*)(b+t));
__m256i lo=_mm256_unpacklo_epi8(va,vb), hi=_mm256_unpackhi_epi8(va,vb);
_mm256_store_si256((__m256i*)(o+2*t),    _mm256_permute2x128_si256(lo,hi,0x20));
_mm256_store_si256((__m256i*)(o+2*t+32), _mm256_permute2x128_si256(lo,hi,0x31));
uint32_t m0=(uint32_t)_mm256_movemask_epi8(_mm256_slli_epi16(va,7));
uint32_t m1=(uint32_t)_mm256_movemask_epi8(_mm256_slli_epi16(vb,7));
parity0 |= (uint64_t)m0<<t; parity1 |= (uint64_t)m1<<t;
}
w0p[g]=parity0; w1p[g]=parity1;
}
}
}
#define SM 2048
static int8_t SA[SM*SM]     __attribute__((aligned(64)));
static int8_t SB[SM*SM]     __attribute__((aligned(64)));
static int8_t MMS[7][SM*SM+24576] __attribute__((aligned(64)));
#define AQT(J,S) if (S){ __m256i _u0=_mm256_load_si256((const __m256i*)(p##J+k));           \
__m256i _u1=_mm256_load_si256((const __m256i*)(p##J+k+32));       \
if ((S)>0){ v0=_mm256_add_epi8(v0,_u0); v1=_mm256_add_epi8(v1,_u1); } \
else      { v0=_mm256_sub_epi8(v0,_u0); v1=_mm256_sub_epi8(v1,_u1); } }
#define BQT(J,S) if (S){ __m256i _x=_mm256_load_si256((const __m256i*)(q##J##a+64*g+t));    \
__m256i _y=_mm256_load_si256((const __m256i*)(q##J##b+64*g+t));    \
if ((S)>0){ va=_mm256_add_epi8(va,_x); vb=_mm256_add_epi8(vb,_y); } \
else      { va=_mm256_sub_epi8(va,_x); vb=_mm256_sub_epi8(vb,_y); } }
template<int S1,int S2,int S3>
AVX2 static void buildA_q4t(int h,int n,const int8_t*P,
int r0,int c0,int r1,int c1,int r2,int c2,int r3,int c3){
const __m256i mx=_mm256_set1_epi8(0x7F);
int W=h/64;
for (int i=0;i<h;i++){
const int8_t*p0=P+(size_t)(r0+i)*n+c0, *p1=P+(size_t)(r1+i)*n+c1,
*p2=P+(size_t)(r2+i)*n+c2, *p3=P+(size_t)(r3+i)*n+c3;
int8_t*dst=A0b+(size_t)i*GLDA;
uint64_t*w1=A1b+(size_t)i*W;
for (int k=0;k<h;k+=64){
_mm_prefetch((const char*)(p0+2*n+k), _MM_HINT_T0);
if constexpr(S1)_mm_prefetch((const char*)(p1+2*n+k), _MM_HINT_T0);
if constexpr(S2)_mm_prefetch((const char*)(p2+2*n+k), _MM_HINT_T0);
if constexpr(S3)_mm_prefetch((const char*)(p3+2*n+k), _MM_HINT_T0);
__m256i v0=_mm256_load_si256((const __m256i*)(p0+k));
__m256i v1=_mm256_load_si256((const __m256i*)(p0+k+32));
AQT(1,S1); AQT(2,S2); AQT(3,S3);
__asm__ volatile("prefetchw %0" :: "m"(*(char*)(dst+2*k+256)));
__asm__ volatile("prefetchw %0" :: "m"(*(char*)(dst+2*k+320)));
{ __m256i _w0=_mm256_and_si256(v0,mx), _w1=_mm256_and_si256(v1,mx);
__m256i _l0=_mm256_unpacklo_epi16(_w0,_w0), _h0=_mm256_unpackhi_epi16(_w0,_w0);
__m256i _l1=_mm256_unpacklo_epi16(_w1,_w1), _h1=_mm256_unpackhi_epi16(_w1,_w1);
_mm256_store_si256((__m256i*)(dst+2*k),    _mm256_permute2x128_si256(_l0,_h0,0x20));
_mm256_store_si256((__m256i*)(dst+2*k+32), _mm256_permute2x128_si256(_l0,_h0,0x31));
_mm256_store_si256((__m256i*)(dst+2*k+64), _mm256_permute2x128_si256(_l1,_h1,0x20));
_mm256_store_si256((__m256i*)(dst+2*k+96), _mm256_permute2x128_si256(_l1,_h1,0x31)); }
(h==256?A1b[(size_t)(k>>6)*h+i]:w1[k>>6])=(uint64_t)(uint32_t)_mm256_movemask_epi8(v0)
|((uint64_t)(uint32_t)_mm256_movemask_epi8(v1)<<32);
}
}
}
template<int S1,int S2,int S3>
AVX2 static void buildB_q4t(int h,int n,const int8_t*P,
int r0,int c0,int r1,int c1,int r2,int c2,int r3,int c3){
const __m256i one=_mm256_set1_epi8(1);
int ng=h/64, W=h/64;
for (int p=0;p<h/2;p++){
const int8_t *a0=P+(size_t)(r0+2*p)*n+c0,   *a1=P+(size_t)(r0+2*p+1)*n+c0;
const int8_t *q1a=P+(size_t)(r1+2*p)*n+c1,  *q1b=P+(size_t)(r1+2*p+1)*n+c1;
const int8_t *q2a=P+(size_t)(r2+2*p)*n+c2,  *q2b=P+(size_t)(r2+2*p+1)*n+c2;
const int8_t *q3a=P+(size_t)(r3+2*p)*n+c3,  *q3b=P+(size_t)(r3+2*p+1)*n+c3;
uint64_t *w0p=Bpb+(size_t)(2*p)*W, *w1p=Bpb+(size_t)(2*p+1)*W;
for (int g=0;g<ng;g++){
_mm_prefetch((const char*)(a0+64*g+2*n), _MM_HINT_T0);
_mm_prefetch((const char*)(a1+64*g+2*n), _MM_HINT_T0);
if constexpr(S1)_mm_prefetch((const char*)(q1a+64*g+2*n), _MM_HINT_T0);
if constexpr(S1)_mm_prefetch((const char*)(q1b+64*g+2*n), _MM_HINT_T0);
if constexpr(S2)_mm_prefetch((const char*)(q2a+64*g+2*n), _MM_HINT_T0);
if constexpr(S2)_mm_prefetch((const char*)(q2b+64*g+2*n), _MM_HINT_T0);
if constexpr(S3)_mm_prefetch((const char*)(q3a+64*g+2*n), _MM_HINT_T0);
if constexpr(S3)_mm_prefetch((const char*)(q3b+64*g+2*n), _MM_HINT_T0);
uint64_t parity0=0,parity1=0;
for (int t=0;t<64;t+=32){
__asm__ volatile("prefetchw %0" :: "m"(*(char*)(BI3b+((size_t)g*(h/2)+p)*128+2*t+256)));
__asm__ volatile("prefetchw %0" :: "m"(*(char*)(BI3b+((size_t)g*(h/2)+p)*128+2*t+320)));
__m256i va=_mm256_load_si256((const __m256i*)(a0+64*g+t));
__m256i vb=_mm256_load_si256((const __m256i*)(a1+64*g+t));
BQT(1,S1); BQT(2,S2); BQT(3,S3);
__m256i lo=_mm256_unpacklo_epi8(va,vb), hi=_mm256_unpackhi_epi8(va,vb);
int8_t *o=BI3b+((size_t)g*(h/2)+p)*128;
_mm256_store_si256((__m256i*)(o+2*t),    (h==256?lo:_mm256_permute2x128_si256(lo,hi,0x20)));
_mm256_store_si256((__m256i*)(o+2*t+32), (h==256?hi:_mm256_permute2x128_si256(lo,hi,0x31)));
uint32_t mm0=(uint32_t)_mm256_movemask_epi8(_mm256_slli_epi16(va,7));
uint32_t mm1=(uint32_t)_mm256_movemask_epi8(_mm256_slli_epi16(vb,7));
parity0 |= (uint64_t)mm0<<t; parity1 |= (uint64_t)mm1<<t;
}
w0p[g]=parity0; w1p[g]=parity1;
}
}
}
#define AQ_TERM(J) if (s##J){ __m256i _u0=_mm256_load_si256((const __m256i*)(p##J+k));          \
__m256i _u1=_mm256_load_si256((const __m256i*)(p##J+k+32));      \
if (s##J>0){ v0=_mm256_add_epi8(v0,_u0); v1=_mm256_add_epi8(v1,_u1); } \
else       { v0=_mm256_sub_epi8(v0,_u0); v1=_mm256_sub_epi8(v1,_u1); } }
#define BQ_TERM(J) if (s##J){ __m256i _x=_mm256_load_si256((const __m256i*)(q##J##a+64*g+t));   \
__m256i _y=_mm256_load_si256((const __m256i*)(q##J##b+64*g+t));   \
if (s##J>0){ va=_mm256_add_epi8(va,_x); vb=_mm256_add_epi8(vb,_y); } \
else       { va=_mm256_sub_epi8(va,_x); vb=_mm256_sub_epi8(vb,_y); } }
AVX2 static void buildA_q(int h,int n,const int8_t*X,int r0,int c0,
const int8_t*Y,int r1,int c1,int op){
const __m256i mx=_mm256_set1_epi8(0x7F);
int W=h/64;
for (int i=0;i<h;i++){
const int8_t*xa=X+(size_t)(r0+i)*n+c0, *ya=Y+(size_t)(r1+i)*n+c1;
int8_t*dst=A0b+(size_t)i*GLDA;
uint64_t*w1=A1b+(size_t)i*W;
for (int k=0;k<h;k+=64){
__m256i v0=_mm256_load_si256((const __m256i*)(xa+k));
__m256i v1=_mm256_load_si256((const __m256i*)(xa+k+32));
if (op){ __m256i u0=_mm256_load_si256((const __m256i*)(ya+k));
__m256i u1=_mm256_load_si256((const __m256i*)(ya+k+32));
if(op==1){ v0=_mm256_add_epi8(v0,u0); v1=_mm256_add_epi8(v1,u1); }
else     { v0=_mm256_sub_epi8(v0,u0); v1=_mm256_sub_epi8(v1,u1); } }
{ __m256i _w0=_mm256_and_si256(v0,mx), _w1=_mm256_and_si256(v1,mx);
__m256i _l0=_mm256_unpacklo_epi16(_w0,_w0), _h0=_mm256_unpackhi_epi16(_w0,_w0);
__m256i _l1=_mm256_unpacklo_epi16(_w1,_w1), _h1=_mm256_unpackhi_epi16(_w1,_w1);
_mm256_store_si256((__m256i*)(dst+2*k),    _mm256_permute2x128_si256(_l0,_h0,0x20));
_mm256_store_si256((__m256i*)(dst+2*k+32), _mm256_permute2x128_si256(_l0,_h0,0x31));
_mm256_store_si256((__m256i*)(dst+2*k+64), _mm256_permute2x128_si256(_l1,_h1,0x20));
_mm256_store_si256((__m256i*)(dst+2*k+96), _mm256_permute2x128_si256(_l1,_h1,0x31)); }
(h==256?A1b[(size_t)(k>>6)*h+i]:w1[k>>6])=(uint64_t)(uint32_t)_mm256_movemask_epi8(v0)
|((uint64_t)(uint32_t)_mm256_movemask_epi8(v1)<<32);
}
}
}
AVX2 static void buildB_q(int h,int n,const int8_t*X,int r0,int c0,
const int8_t*Y,int r1,int c1,int op){
const __m256i one=_mm256_set1_epi8(1);
int ng=h/64, W=h/64;
for (int p=0;p<h/2;p++){
const int8_t *ra=X+(size_t)(r0+2*p)*n+c0,   *rb=X+(size_t)(r0+2*p+1)*n+c0;
const int8_t *sa=Y+(size_t)(r1+2*p)*n+c1,   *sb=Y+(size_t)(r1+2*p+1)*n+c1;
uint64_t *w0p=Bpb+(size_t)(2*p)*W, *w1p=Bpb+(size_t)(2*p+1)*W;
for (int g=0;g<ng;g++){
uint64_t parity0=0,parity1=0;
for (int t=0;t<64;t+=32){
__m256i va=_mm256_load_si256((const __m256i*)(ra+64*g+t));
__m256i vb=_mm256_load_si256((const __m256i*)(rb+64*g+t));
if (op){ __m256i pa=_mm256_load_si256((const __m256i*)(sa+64*g+t));
__m256i pb=_mm256_load_si256((const __m256i*)(sb+64*g+t));
if(op==1){ va=_mm256_add_epi8(va,pa); vb=_mm256_add_epi8(vb,pb); }
else     { va=_mm256_sub_epi8(va,pa); vb=_mm256_sub_epi8(vb,pb); } }
__m256i lo=_mm256_unpacklo_epi8(va,vb), hi=_mm256_unpackhi_epi8(va,vb);
int8_t *o=BI3b+((size_t)g*(h/2)+p)*128;
_mm256_store_si256((__m256i*)(o+2*t),    (h==256?lo:_mm256_permute2x128_si256(lo,hi,0x20)));
_mm256_store_si256((__m256i*)(o+2*t+32), (h==256?hi:_mm256_permute2x128_si256(lo,hi,0x31)));
uint32_t mm0=(uint32_t)_mm256_movemask_epi8(_mm256_slli_epi16(va,7));
uint32_t mm1=(uint32_t)_mm256_movemask_epi8(_mm256_slli_epi16(vb,7));
parity0 |= (uint64_t)mm0<<t; parity1 |= (uint64_t)mm1<<t;
}
w0p[g]=parity0; w1p[g]=parity1;
}
}
}
static int8_t S2M[7][1024*1024+24576]  __attribute__((aligned(64)));
static int8_t S3A[1024*1024]        __attribute__((aligned(64)));
static int8_t S3B[1024*1024]        __attribute__((aligned(64)));
static int8_t S3M[7][512*512+24576]    __attribute__((aligned(64)));
static int8_t S4A[512*512]          __attribute__((aligned(64)));
static int8_t S4B[512*512]          __attribute__((aligned(64)));
static int8_t S4M[7][256*256+24576]    __attribute__((aligned(64)));
AVX2 static void buildA_q4g(int h,int n,const int8_t*P,
int r0,int c0,int r1,int c1,int s1,
int r2,int c2,int s2, int r3,int c3,int s3){
const __m256i mx=_mm256_set1_epi8(0x7F);
int W=h/64;
for (int i=0;i<h;i++){
const int8_t*p0=P+(size_t)(r0+i)*n+c0, *p1=P+(size_t)(r1+i)*n+c1,
*p2=P+(size_t)(r2+i)*n+c2, *p3=P+(size_t)(r3+i)*n+c3;
int8_t*dst=A0b+(size_t)i*GLDA;
uint64_t*w1=A1b+(size_t)i*W;
for (int k=0;k<h;k+=64){
__m256i v0=_mm256_load_si256((const __m256i*)(p0+k));
__m256i v1=_mm256_load_si256((const __m256i*)(p0+k+32));
AQ_TERM(1); AQ_TERM(2); AQ_TERM(3);
{ __m256i _w0=_mm256_and_si256(v0,mx), _w1=_mm256_and_si256(v1,mx);
__m256i _l0=_mm256_unpacklo_epi16(_w0,_w0), _h0=_mm256_unpackhi_epi16(_w0,_w0);
__m256i _l1=_mm256_unpacklo_epi16(_w1,_w1), _h1=_mm256_unpackhi_epi16(_w1,_w1);
_mm256_store_si256((__m256i*)(dst+2*k),    _mm256_permute2x128_si256(_l0,_h0,0x20));
_mm256_store_si256((__m256i*)(dst+2*k+32), _mm256_permute2x128_si256(_l0,_h0,0x31));
_mm256_store_si256((__m256i*)(dst+2*k+64), _mm256_permute2x128_si256(_l1,_h1,0x20));
_mm256_store_si256((__m256i*)(dst+2*k+96), _mm256_permute2x128_si256(_l1,_h1,0x31)); }
(h==256?A1b[(size_t)(k>>6)*h+i]:w1[k>>6])=(uint64_t)(uint32_t)_mm256_movemask_epi8(v0)
|((uint64_t)(uint32_t)_mm256_movemask_epi8(v1)<<32);
}
}
}
AVX2 static void buildA_q4(int h,int n,const int8_t*P,
int r0,int c0,int r1,int c1,int s1,
int r2,int c2,int s2, int r3,int c3,int s3){
#define PACKA(S1,S2,S3) return buildA_q4t<S1,S2,S3>(h,n,P,r0,c0,r1,c1,r2,c2,r3,c3)
if (s1==-1&&s2==-1&&s3== 1){ PACKA(-1,-1, 1); }
if (s1==-1&&s2== 0&&s3== 0){ PACKA(-1, 0, 0); }
if (s1==-1&&s2== 1&&s3==-1){ PACKA(-1, 1,-1); }
if (s1== 0&&s2==-1&&s3== 0){ PACKA( 0,-1, 0); }
if (s1== 0&&s2== 0&&s3== 0){ PACKA( 0, 0, 0); }
if (s1== 0&&s2== 1&&s3== 0){ PACKA( 0, 1, 0); }
if (s1== 1&&s2==-1&&s3==-1){ PACKA( 1,-1,-1); }
if (s1== 1&&s2== 0&&s3== 0){ PACKA( 1, 0, 0); }
if (s1== 1&&s2== 1&&s3== 1){ PACKA( 1, 1, 1); }
buildA_q4g(h,n,P,r0,c0,r1,c1,s1,r2,c2,s2,r3,c3,s3);
}
AVX2 static void buildB_q4g(int h,int n,const int8_t*P,
int r0,int c0,int r1,int c1,int s1,
int r2,int c2,int s2, int r3,int c3,int s3){
const __m256i one=_mm256_set1_epi8(1);
int ng=h/64, W=h/64;
for (int p=0;p<h/2;p++){
const int8_t *a0=P+(size_t)(r0+2*p)*n+c0,   *a1=P+(size_t)(r0+2*p+1)*n+c0;
const int8_t *q1a=P+(size_t)(r1+2*p)*n+c1,  *q1b=P+(size_t)(r1+2*p+1)*n+c1;
const int8_t *q2a=P+(size_t)(r2+2*p)*n+c2,  *q2b=P+(size_t)(r2+2*p+1)*n+c2;
const int8_t *q3a=P+(size_t)(r3+2*p)*n+c3,  *q3b=P+(size_t)(r3+2*p+1)*n+c3;
uint64_t *w0p=Bpb+(size_t)(2*p)*W, *w1p=Bpb+(size_t)(2*p+1)*W;
for (int g=0;g<ng;g++){
uint64_t parity0=0,parity1=0;
for (int t=0;t<64;t+=32){
__m256i va=_mm256_load_si256((const __m256i*)(a0+64*g+t));
__m256i vb=_mm256_load_si256((const __m256i*)(a1+64*g+t));
BQ_TERM(1); BQ_TERM(2); BQ_TERM(3);
__m256i lo=_mm256_unpacklo_epi8(va,vb), hi=_mm256_unpackhi_epi8(va,vb);
int8_t *o=BI3b+((size_t)g*(h/2)+p)*128;
_mm256_store_si256((__m256i*)(o+2*t),    (h==256?lo:_mm256_permute2x128_si256(lo,hi,0x20)));
_mm256_store_si256((__m256i*)(o+2*t+32), (h==256?hi:_mm256_permute2x128_si256(lo,hi,0x31)));
uint32_t mm0=(uint32_t)_mm256_movemask_epi8(_mm256_slli_epi16(va,7));
uint32_t mm1=(uint32_t)_mm256_movemask_epi8(_mm256_slli_epi16(vb,7));
parity0 |= (uint64_t)mm0<<t; parity1 |= (uint64_t)mm1<<t;
}
w0p[g]=parity0; w1p[g]=parity1;
}
}
}
AVX2 static void buildB_q4(int h,int n,const int8_t*P,
int r0,int c0,int r1,int c1,int s1,
int r2,int c2,int s2, int r3,int c3,int s3){
#define PACKB(S1,S2,S3) return buildB_q4t<S1,S2,S3>(h,n,P,r0,c0,r1,c1,r2,c2,r3,c3)
if (s1==-1&&s2==-1&&s3== 1){ PACKB(-1,-1, 1); }
if (s1==-1&&s2== 0&&s3== 0){ PACKB(-1, 0, 0); }
if (s1==-1&&s2== 1&&s3==-1){ PACKB(-1, 1,-1); }
if (s1== 0&&s2==-1&&s3== 0){ PACKB( 0,-1, 0); }
if (s1== 0&&s2== 0&&s3== 0){ PACKB( 0, 0, 0); }
if (s1== 0&&s2== 1&&s3== 0){ PACKB( 0, 1, 0); }
if (s1== 1&&s2==-1&&s3==-1){ PACKB( 1,-1,-1); }
if (s1== 1&&s2== 0&&s3== 0){ PACKB( 1, 0, 0); }
if (s1== 1&&s2== 1&&s3== 1){ PACKB( 1, 1, 1); }
buildB_q4g(h,n,P,r0,c0,r1,c1,s1,r2,c2,s2,r3,c3,s3);
}
template<int Q>
AVX2 static void cmb4(int8_t *C,int n,int h,const int8_t (&M)[7][Q]){
for(int i=0;i<h;i++){
int8_t *r0=C+(size_t)i*n, *r1=C+(size_t)(i+h)*n;
for(int j=0;j<h;j+=64){
size_t t=(size_t)i*h+j;
const __m256i a0=_mm256_load_si256((const __m256i*)(M[0]+t)), b0=_mm256_load_si256((const __m256i*)(M[0]+t+32));
const __m256i a1=_mm256_load_si256((const __m256i*)(M[1]+t)), b1=_mm256_load_si256((const __m256i*)(M[1]+t+32));
const __m256i a2=_mm256_load_si256((const __m256i*)(M[2]+t)), b2=_mm256_load_si256((const __m256i*)(M[2]+t+32));
const __m256i a3=_mm256_load_si256((const __m256i*)(M[3]+t)), b3=_mm256_load_si256((const __m256i*)(M[3]+t+32));
const __m256i a4=_mm256_load_si256((const __m256i*)(M[4]+t)), b4=_mm256_load_si256((const __m256i*)(M[4]+t+32));
const __m256i a5=_mm256_load_si256((const __m256i*)(M[5]+t)), b5=_mm256_load_si256((const __m256i*)(M[5]+t+32));
const __m256i a6=_mm256_load_si256((const __m256i*)(M[6]+t)), b6=_mm256_load_si256((const __m256i*)(M[6]+t+32));
_mm256_stream_si256((__m256i*)(r0+j),
_mm256_add_epi8(_mm256_sub_epi8(_mm256_add_epi8(a0,a3),a4),a6));
_mm256_stream_si256((__m256i*)(r0+j+32),
_mm256_add_epi8(_mm256_sub_epi8(_mm256_add_epi8(b0,b3),b4),b6));
_mm256_stream_si256((__m256i*)(r0+h+j), _mm256_add_epi8(a2,a4));
_mm256_stream_si256((__m256i*)(r0+h+j+32), _mm256_add_epi8(b2,b4));
_mm256_stream_si256((__m256i*)(r1+j),   _mm256_add_epi8(a1,a3));
_mm256_stream_si256((__m256i*)(r1+j+32), _mm256_add_epi8(b1,b3));
_mm256_stream_si256((__m256i*)(r1+h+j),
_mm256_add_epi8(_mm256_sub_epi8(_mm256_add_epi8(a0,a2),a1),a5));
_mm256_stream_si256((__m256i*)(r1+h+j+32),
_mm256_add_epi8(_mm256_sub_epi8(_mm256_add_epi8(b0,b2),b1),b5));
}
}
}
static __attribute__((always_inline)) inline __m256i iv366_bc(const uint64_t*p){__m256i r;asm volatile("vmovq %1,%x0;vpbroadcastq %x0,%0":"=x"(r):"r"(*p));return r;}
template<int Q>
AVX2 static void cmb4wb(int8_t*C,int n,int h,const int8_t(&M)[7][Q]){const __m256i ctrl0=_mm256_load_si256((const __m256i*)PSCTRL),ctrl1=_mm256_add_epi8(ctrl0,_mm256_set1_epi8(4)),bit=_mm256_load_si256((const __m256i*)PSBIT),sign=_mm256_load_si256((const __m256i*)PS80);for(int i=0;i<h;i++){int8_t*r0=C+(size_t)i*n,*r1=C+(size_t)(i+h)*n;for(int j=0;j<h;j+=64){size_t pos=(size_t)i*4+(j>>6);uint64_t b0=iv363_pb[0][pos],b1=iv363_pb[1][pos],b2=iv363_pb[2][pos],b3=iv363_pb[3][pos],b4=iv363_pb[4][pos],b5=iv363_pb[5][pos],b6=iv363_pb[6][pos];alignas(32)uint64_t iv366c[4]={b0^b3^b4^b6,b2^b4,b1^b3,b0^b2^b1^b5};for(int half=0;half<2;half++){size_t off=(size_t)i*h+j+32*half;__m256i m0=_mm256_load_si256((const __m256i*)(M[0]+off)),m1=_mm256_load_si256((const __m256i*)(M[1]+off)),m2=_mm256_load_si256((const __m256i*)(M[2]+off)),m3=_mm256_load_si256((const __m256i*)(M[3]+off)),m4=_mm256_load_si256((const __m256i*)(M[4]+off)),m5=_mm256_load_si256((const __m256i*)(M[5]+off)),m6=_mm256_load_si256((const __m256i*)(M[6]+off));__m256i v0=_mm256_add_epi8(_mm256_sub_epi8(_mm256_add_epi8(m0,m3),m4),m6),v1=_mm256_add_epi8(m2,m4),v2=_mm256_add_epi8(m1,m3),v3=_mm256_add_epi8(_mm256_sub_epi8(_mm256_add_epi8(m0,m2),m1),m5);__m256i ctrl=half?ctrl1:ctrl0;__m256i e0=_mm256_sign_epi8(sign,_mm256_and_si256(_mm256_shuffle_epi8(iv366_bc(iv366c+0),ctrl),bit)),e1=_mm256_sign_epi8(sign,_mm256_and_si256(_mm256_shuffle_epi8(iv366_bc(iv366c+1),ctrl),bit)),e2=_mm256_sign_epi8(sign,_mm256_and_si256(_mm256_shuffle_epi8(iv366_bc(iv366c+2),ctrl),bit)),e3=_mm256_sign_epi8(sign,_mm256_and_si256(_mm256_shuffle_epi8(iv366_bc(iv366c+3),ctrl),bit));_mm256_store_si256((__m256i*)(r0+j+32*half),_mm256_xor_si256(v0,e0));_mm256_store_si256((__m256i*)(r0+h+j+32*half),_mm256_xor_si256(v1,e1));_mm256_store_si256((__m256i*)(r1+j+32*half),_mm256_xor_si256(v2,e2));_mm256_store_si256((__m256i*)(r1+h+j+32*half),_mm256_xor_si256(v3,e3));}}}}
AVX2 static inline __m256i c4_fuse_expand(unsigned b){return _mm256_sign_epi8(_mm256_load_si256((const __m256i*)PS80),_mm256_and_si256(_mm256_shuffle_epi8(_mm256_set1_epi32(b),_mm256_load_si256((const __m256i*)PSCTRL)),_mm256_load_si256((const __m256i*)PSBIT)));}
AVX2 static inline void c4_fuse_words(size_t p,uint64_t*w){uint64_t a=iv363_pb[0][p],b=iv363_pb[1][p],c=iv363_pb[2][p],d=iv363_pb[3][p],e=iv363_pb[4][p],f=iv363_pb[5][p],g=iv363_pb[6][p];w[0]=a^d^e^g;w[1]=c^e;w[2]=b^d;w[3]=a^b^c^f;}
template<int Q,int Q2>
AVX2 static void cmb4fu(int8_t*C,int n,int h,const int8_t (&M)[7][Q],const int8_t (&S)[7][Q2]){
const int h2=h>>1;
for(int i=0;i<h2;i++){
const size_t t2=(size_t)i*h2;
for(int j=0;j<h2;j+=64){
const __m256i p0=_mm256_load_si256((const __m256i*)(S[0]+t2+j)), P0=_mm256_load_si256((const __m256i*)(S[0]+t2+j+32));
const __m256i p1=_mm256_load_si256((const __m256i*)(S[1]+t2+j)), P1=_mm256_load_si256((const __m256i*)(S[1]+t2+j+32));
const __m256i p2=_mm256_load_si256((const __m256i*)(S[2]+t2+j)), P2=_mm256_load_si256((const __m256i*)(S[2]+t2+j+32));
const __m256i p3=_mm256_load_si256((const __m256i*)(S[3]+t2+j)), P3=_mm256_load_si256((const __m256i*)(S[3]+t2+j+32));
const __m256i p4=_mm256_load_si256((const __m256i*)(S[4]+t2+j)), P4=_mm256_load_si256((const __m256i*)(S[4]+t2+j+32));
const __m256i p5=_mm256_load_si256((const __m256i*)(S[5]+t2+j)), P5=_mm256_load_si256((const __m256i*)(S[5]+t2+j+32));
const __m256i p6=_mm256_load_si256((const __m256i*)(S[6]+t2+j)), P6=_mm256_load_si256((const __m256i*)(S[6]+t2+j+32));
__m256i v0=_mm256_add_epi8(_mm256_sub_epi8(_mm256_add_epi8(p0,p3),p4),p6), W0=_mm256_add_epi8(_mm256_sub_epi8(_mm256_add_epi8(P0,P3),P4),P6);
__m256i v1=_mm256_add_epi8(p2,p4), W1=_mm256_add_epi8(P2,P4);
__m256i v2=_mm256_add_epi8(p1,p3), W2=_mm256_add_epi8(P1,P3);
__m256i v3=_mm256_add_epi8(_mm256_sub_epi8(_mm256_add_epi8(p0,p2),p1),p5), W3=_mm256_add_epi8(_mm256_sub_epi8(_mm256_add_epi8(P0,P2),P1),P5);
uint64_t bits[4];c4_fuse_words(i*4+(j>>6),bits);
v0=_mm256_xor_si256(v0,c4_fuse_expand((unsigned)bits[0]));W0=_mm256_xor_si256(W0,c4_fuse_expand((unsigned)(bits[0]>>32)));
v1=_mm256_xor_si256(v1,c4_fuse_expand((unsigned)bits[1]));W1=_mm256_xor_si256(W1,c4_fuse_expand((unsigned)(bits[1]>>32)));
v2=_mm256_xor_si256(v2,c4_fuse_expand((unsigned)bits[2]));W2=_mm256_xor_si256(W2,c4_fuse_expand((unsigned)(bits[2]>>32)));
v3=_mm256_xor_si256(v3,c4_fuse_expand((unsigned)bits[3]));W3=_mm256_xor_si256(W3,c4_fuse_expand((unsigned)(bits[3]>>32)));
#define POS(R,CC,VA,VB) { const int8_t*m0=M[0]+(size_t)(R)*h+(CC), *m1=M[1]+(size_t)(R)*h+(CC), \
*m2=M[2]+(size_t)(R)*h+(CC), *m3=M[3]+(size_t)(R)*h+(CC), *m4=M[4]+(size_t)(R)*h+(CC), *m5=M[5]+(size_t)(R)*h+(CC); \
const __m256i a0=_mm256_load_si256((const __m256i*)m0), A0=_mm256_load_si256((const __m256i*)(m0+32)); \
const __m256i a1=_mm256_load_si256((const __m256i*)m1), A1=_mm256_load_si256((const __m256i*)(m1+32)); \
const __m256i a2=_mm256_load_si256((const __m256i*)m2), A2=_mm256_load_si256((const __m256i*)(m2+32)); \
const __m256i a3=_mm256_load_si256((const __m256i*)m3), A3=_mm256_load_si256((const __m256i*)(m3+32)); \
const __m256i a4=_mm256_load_si256((const __m256i*)m4), A4=_mm256_load_si256((const __m256i*)(m4+32)); \
const __m256i a5=_mm256_load_si256((const __m256i*)m5), A5=_mm256_load_si256((const __m256i*)(m5+32)); \
int8_t*d0=C+(size_t)(R)*n+(CC), *d1=d0+h, *d2=C+(size_t)((R)+h)*n+(CC), *d3=d2+h; \
_mm256_stream_si256((__m256i*)d0, _mm256_add_epi8(VA,_mm256_sub_epi8(_mm256_add_epi8(a0,a3),a4))); \
_mm256_stream_si256((__m256i*)(d0+32), _mm256_add_epi8(VB,_mm256_sub_epi8(_mm256_add_epi8(A0,A3),A4))); \
_mm256_stream_si256((__m256i*)d1, _mm256_add_epi8(a2,a4)); \
_mm256_stream_si256((__m256i*)(d1+32), _mm256_add_epi8(A2,A4)); \
_mm256_stream_si256((__m256i*)d2, _mm256_add_epi8(a1,a3)); \
_mm256_stream_si256((__m256i*)(d2+32), _mm256_add_epi8(A1,A3)); \
_mm256_stream_si256((__m256i*)d3, _mm256_add_epi8(_mm256_sub_epi8(_mm256_add_epi8(a0,a2),a1),a5)); \
_mm256_stream_si256((__m256i*)(d3+32), _mm256_add_epi8(_mm256_sub_epi8(_mm256_add_epi8(A0,A2),A1),A5)); }
POS(i, j, v0, W0)
POS(i, j+h2, v1, W1)
POS(i+h2, j, v2, W2)
POS(i+h2, j+h2, v3, W3)
#undef POS
}
}
}
AVX2 static void cmb(int8_t*C,int n,int r0,int c0,int m,
const int8_t*M0,int s0,const int8_t*M1,int s1,
const int8_t*M2,int s2,const int8_t*M3,int s3){
for (int i=0;i<m;i++){
int8_t *d=C+(size_t)(r0+i)*n+c0;
const int8_t *p0=M0?M0+(size_t)i*m:0, *p1=M1?M1+(size_t)i*m:0;
const int8_t *p2=M2?M2+(size_t)i*m:0, *p3=M3?M3+(size_t)i*m:0;
for (int j=0;j<m;j+=32){
__m256i v;
if (p0){ __m256i t=_mm256_load_si256((const __m256i*)(p0+j));
v = s0>0?t:_mm256_sub_epi8(_mm256_setzero_si256(),t); }
else    v = _mm256_setzero_si256();
if (p1){ __m256i t=_mm256_load_si256((const __m256i*)(p1+j));
v = _mm256_add_epi8(v, s1>0?t:_mm256_sub_epi8(_mm256_setzero_si256(),t)); }
if (p2){ __m256i t=_mm256_load_si256((const __m256i*)(p2+j));
v = _mm256_add_epi8(v, s2>0?t:_mm256_sub_epi8(_mm256_setzero_si256(),t)); }
if (p3){ __m256i t=_mm256_load_si256((const __m256i*)(p3+j));
v = _mm256_add_epi8(v, s3>0?t:_mm256_sub_epi8(_mm256_setzero_si256(),t)); }
_mm256_stream_si256((__m256i*)(d+j), v);
}
}
}
template<int OP>
AVX2 static void blk_t(int8_t*dst,const int8_t*X,int n,int r0,int c0,int h,
const int8_t*Y,int r1,int c1){
const int B = h>>2, NB = 4;
const size_t BB = (size_t)B*(size_t)B;
for (int i=0;i<h;i++){
const int8_t *xa=X+(size_t)(r0+i)*n+c0, *ya=Y+(size_t)(r1+i)*n+c1;
int8_t *d0 = dst + (size_t)((i/B)*NB)*BB + (size_t)(i%B)*B;
for (int J=0;J<NB;J++){
int8_t *d = d0 + (size_t)J*BB;
const int8_t *xaj = xa + J*B, *yaj = ya + J*B;
int j=0;
if (OP==0)      for(;j+32<=B;j+=32) _mm256_store_si256((__m256i*)(d+j),
_mm256_load_si256((const __m256i*)(xaj+j)));
else if (OP==1) for(;j+32<=B;j+=32) _mm256_store_si256((__m256i*)(d+j),
_mm256_add_epi8(_mm256_load_si256((const __m256i*)(xaj+j)),
_mm256_load_si256((const __m256i*)(yaj+j))));
else            for(;j+32<=B;j+=32) _mm256_store_si256((__m256i*)(d+j),
_mm256_sub_epi8(_mm256_load_si256((const __m256i*)(xaj+j)),
_mm256_load_si256((const __m256i*)(yaj+j))));
if (OP) for(;j<B;j++) d[j]=(int8_t)(xaj[j]+(OP==1?yaj[j]:-yaj[j]));
}
}
}
AVX2 static void blk(int8_t*dst,const int8_t*X,int n,int r0,int c0,int h,
const int8_t*Y,int r1,int c1,int op){
if (op==0)      blk_t<0>(dst,X,n,r0,c0,h,Y,r1,c1);
else if (op==1) blk_t<1>(dst,X,n,r0,c0,h,Y,r1,c1);
else            blk_t<2>(dst,X,n,r0,c0,h,Y,r1,c1);
}
template<int OP>
AVX2 static void blk_nt_t(int8_t*dst,const int8_t*X,int n,int r0,int c0,int h,
const int8_t*Y,int r1,int c1){
for (int i=0;i<h;i++){
const int8_t *xa=X+(size_t)(r0+i)*n+c0, *ya=Y+(size_t)(r1+i)*n+c1;
int8_t *d=dst+(size_t)i*h;
int j=0;
if (OP==0)      for(;j+32<=h;j+=32) _mm256_stream_si256((__m256i*)(d+j),
_mm256_load_si256((const __m256i*)(xa+j)));
else if (OP==1) for(;j+32<=h;j+=32) _mm256_stream_si256((__m256i*)(d+j),
_mm256_add_epi8(_mm256_load_si256((const __m256i*)(xa+j)),
_mm256_load_si256((const __m256i*)(ya+j))));
else            for(;j+32<=h;j+=32) _mm256_stream_si256((__m256i*)(d+j),
_mm256_sub_epi8(_mm256_load_si256((const __m256i*)(xa+j)),
_mm256_load_si256((const __m256i*)(ya+j))));
if (OP) for(;j<h;j++) d[j]=(int8_t)(xa[j]+(OP==1?ya[j]:-ya[j]));
}
}
AVX2 static void blk_nt(int8_t*dst,const int8_t*X,int n,int r0,int c0,int h,
const int8_t*Y,int r1,int c1,int op){
if (op==0)      blk_nt_t<0>(dst,X,n,r0,c0,h,Y,r1,c1);
else if (op==1) blk_nt_t<1>(dst,X,n,r0,c0,h,Y,r1,c1);
else            blk_nt_t<2>(dst,X,n,r0,c0,h,Y,r1,c1);
}
AVX2BMI static void corr1024(void);
AVX2BMI static void corr1024(void){
const int n=1024, W=16;
memset(Pbb,0,(size_t)n*W*8);
const int KB=64;
for (int kb=0; kb<n; kb+=KB){
int w0=kb>>6;
const char *bp0=(const char*)Bpb+(size_t)w0*8192;
for (int i=0;i<n;i++){
const uint64_t*row=A1b+(size_t)i*W+w0;
uint64_t b0=row[0], b1=0;
uint64_t *o=Pbb+(size_t)i*W;
asm volatile(
"vpxor %%xmm0,%%xmm0,%%xmm0\n\t"
"vpxor %%xmm1,%%xmm1,%%xmm1\n\t"
"vpxor %%xmm2,%%xmm2,%%xmm2\n\t"
"vpxor %%xmm3,%%xmm3,%%xmm3\n\t"
"testq %[b0], %[b0]\n\t"
"jz 3%=f\n\t"
"1%=:\n\t"
"tzcntq %[b0], %%r11\n\t"
"shlq $7, %%r11\n\t"
"addq %[bp0], %%r11\n\t"
"vpxor 0(%%r11), %%ymm0, %%ymm0\n\t"
"vpxor 32(%%r11), %%ymm1, %%ymm1\n\t"
"vpxor 64(%%r11), %%ymm2, %%ymm2\n\t"
"vpxor 96(%%r11), %%ymm3, %%ymm3\n\t"
"blsrq %[b0], %[b0]\n\t"
"jnz 1%=b\n\t"
"3%=:\n\t"
"testq %[b1], %[b1]\n\t"
"jz 4%=f\n\t"
"2%=:\n\t"
"tzcntq %[b1], %%r11\n\t"
"shlq $7, %%r11\n\t"
"addq %[bp0], %%r11\n\t"
"vpxor 0(%%r11), %%ymm0, %%ymm0\n\t"
"vpxor 32(%%r11), %%ymm1, %%ymm1\n\t"
"vpxor 64(%%r11), %%ymm2, %%ymm2\n\t"
"vpxor 96(%%r11), %%ymm3, %%ymm3\n\t"
"blsrq %[b1], %[b1]\n\t"
"jnz 2%=b\n\t"
"4%=:\n\t"
"vpxor   0(%[o]), %%ymm0, %%ymm0\n\t"
"vmovdqu %%ymm0, 0(%[o])\n\t"
"vpxor   32(%[o]), %%ymm1, %%ymm1\n\t"
"vmovdqu %%ymm1, 32(%[o])\n\t"
"vpxor   64(%[o]), %%ymm2, %%ymm2\n\t"
"vmovdqu %%ymm2, 64(%[o])\n\t"
"vpxor   96(%[o]), %%ymm3, %%ymm3\n\t"
"vmovdqu %%ymm3, 96(%[o])\n\t"
: [b0]"+r"(b0), [b1]"+r"(b1)
: [bp0]"r"(bp0), [o]"r"(o)
: "r11","cc","memory","ymm0","ymm1","ymm2","ymm3");
}
}
}
#define R2M_BMI2 __attribute__((target("avx2,bmi2")))
#define R2M_ROR(w,r) ((uint64_t)(((w)>>(r))|((w)<<(64-(r)))))
static uint64_t TB[((size_t)NMAX/64)*256] asm("c4_fixed_tb") __attribute__((aligned(64),used));
AVX2BMI static void nibtab_build(const uint64_t *__restrict r, uint64_t *__restrict T, int W){
const int U = W>>2;
const __m256i *rp = (const __m256i*)r;
__m256i *tp = (__m256i*)T;
for (int q=0;q<U;q++){
const __m256i a1=rp[q], a2=rp[U+q], a4=rp[2*U+q], a8=rp[3*U+q];
const __m256i x12=_mm256_xor_si256(a1,a2), x14=_mm256_xor_si256(a1,a4),
x18=_mm256_xor_si256(a1,a8);
__m256i *o = tp+q;
o[ 0*U]=_mm256_setzero_si256();
o[ 1*U]=a1;      o[ 2*U]=a2;      o[ 4*U]=a4;      o[ 8*U]=a8;
o[ 3*U]=x12;     o[ 5*U]=x14;
o[ 6*U]=_mm256_xor_si256(a2,a4);   o[ 7*U]=_mm256_xor_si256(x12,a4);
o[ 9*U]=x18;     o[10*U]=_mm256_xor_si256(a2,a8);
o[12*U]=_mm256_xor_si256(a4,a8);
o[11*U]=_mm256_xor_si256(a2,x18);  o[13*U]=_mm256_xor_si256(a4,x18);
o[14*U]=_mm256_xor_si256(_mm256_xor_si256(a2,a4),a8);
o[15*U]=_mm256_xor_si256(_mm256_xor_si256(x12,a4),a8);
}
}
#define DEF_NIBCORR(TAG, M)                                                    \
AVX2BMI static void nibcorr##TAG(void){                                        \
const int n=M, W=n/64, G=n>>2, GB=16, NY=W>>2;                               \
memset(Pbb,0,(size_t)n*W*8);                                                 \
for (int g0=0; g0<G; g0+=GB){                                                \
for (int t=0;t<GB;t++){                                                    \
const int g=g0+t;                                                        \
uint64_t *T=TB+(size_t)t*16*W;                                           \
const uint64_t *r=Bpb+(size_t)(g*4)*W;                                   \
nibtab_build(r,T,W);                                                     \
\
}                                                                          \
for (int i=0;i<n;i++){                                                     \
const uint64_t *arow=A1b+(size_t)i*W+(g0>>4);                            \
__m256i acc[NY];                                                         \
for (int q=0;q<NY;q++) acc[q]=_mm256_setzero_si256();                    \
const uint64_t *T=TB;                                                    \
for (int wb=0; wb<(GB>>4); wb++){                                        \
const uint64_t word=arow[wb];                                          \
_Pragma("GCC unroll 16")                                               \
for (int b=0;b<16;b++, T+=(size_t)16*W){                               \
const uint64_t *e=T+(size_t)(((word>>(4*b))&15u)*W);                 \
for (int q=0;q<NY;q++)                                               \
acc[q]=_mm256_xor_si256(acc[q],_mm256_load_si256((const __m256i*)(e+4*q))); \
}                                                                      \
}                                                                        \
uint64_t *o=Pbb+(size_t)i*W;                                             \
for (int q=0;q<NY;q++){                                                  \
__m256i *po=(__m256i*)(o+4*q);                                         \
_mm256_store_si256(po,_mm256_xor_si256(_mm256_load_si256((const __m256i*)po),acc[q])); \
}                                                                        \
}                                                                          \
}                                                                            \
}
DEF_NIBCORR(512,512)
DEF_NIBCORR(256,256)
DEF_NIBCORR(1024,1024)
AVX2 static inline __attribute__((always_inline)) void mixed_build5(const uint64_t*r,__m256i*T){asm volatile(
"vmovdqa 0(%[r]),%%ymm0\nvmovdqa 32(%[r]),%%ymm1\nvmovdqa 64(%[r]),%%ymm2\nvmovdqa 96(%[r]),%%ymm3\nvmovdqa 128(%[r]),%%ymm4\nvpxor %%ymm6,%%ymm6,%%ymm6\nvmovdqa %%ymm6,0(%[t])\nvmovdqa %%ymm0,32(%[t])\nvpxor %%ymm1,%%ymm0,%%ymm6\nvmovdqa %%ymm6,96(%[t])\nvmovdqa %%ymm1,64(%[t])\nvpxor %%ymm2,%%ymm1,%%ymm6\nvmovdqa %%ymm6,192(%[t])\nvpxor %%ymm0,%%ymm6,%%ymm6\nvmovdqa %%ymm6,224(%[t])\nvpxor %%ymm1,%%ymm6,%%ymm6\nvmovdqa %%ymm6,160(%[t])\nvmovdqa %%ymm2,128(%[t])\nvpxor %%ymm3,%%ymm2,%%ymm6\nvmovdqa %%ymm6,384(%[t])\nvpxor %%ymm0,%%ymm6,%%ymm6\nvmovdqa %%ymm6,416(%[t])\nvpxor %%ymm1,%%ymm6,%%ymm6\nvmovdqa %%ymm6,480(%[t])\nvpxor %%ymm0,%%ymm6,%%ymm6\nvmovdqa %%ymm6,448(%[t])\nvpxor %%ymm2,%%ymm6,%%ymm6\nvmovdqa %%ymm6,320(%[t])\nvpxor %%ymm0,%%ymm6,%%ymm6\nvmovdqa %%ymm6,352(%[t])\nvpxor %%ymm1,%%ymm6,%%ymm6\nvmovdqa %%ymm6,288(%[t])\nvmovdqa %%ymm3,256(%[t])\nvpxor %%ymm4,%%ymm3,%%ymm6\nvmovdqa %%ymm6,768(%[t])\nvpxor %%ymm0,%%ymm6,%%ymm6\nvmovdqa %%ymm6,800(%[t])\nvpxor %%ymm1,%%ymm6,%%ymm6\nvmovdqa %%ymm6,864(%[t])\nvpxor %%ymm0,%%ymm6,%%ymm6\nvmovdqa %%ymm6,832(%[t])\nvpxor %%ymm2,%%ymm6,%%ymm6\nvmovdqa %%ymm6,960(%[t])\nvpxor %%ymm0,%%ymm6,%%ymm6\nvmovdqa %%ymm6,992(%[t])\nvpxor %%ymm1,%%ymm6,%%ymm6\nvmovdqa %%ymm6,928(%[t])\nvpxor %%ymm0,%%ymm6,%%ymm6\nvmovdqa %%ymm6,896(%[t])\nvpxor %%ymm3,%%ymm6,%%ymm6\nvmovdqa %%ymm6,640(%[t])\nvpxor %%ymm0,%%ymm6,%%ymm6\nvmovdqa %%ymm6,672(%[t])\nvpxor %%ymm1,%%ymm6,%%ymm6\nvmovdqa %%ymm6,736(%[t])\nvpxor %%ymm0,%%ymm6,%%ymm6\nvmovdqa %%ymm6,704(%[t])\nvpxor %%ymm2,%%ymm6,%%ymm6\nvmovdqa %%ymm6,576(%[t])\nvpxor %%ymm0,%%ymm6,%%ymm6\nvmovdqa %%ymm6,608(%[t])\nvpxor %%ymm1,%%ymm6,%%ymm6\nvmovdqa %%ymm6,544(%[t])\nvmovdqa %%ymm4,512(%[t])\n"
::[r]"r"(r),[t]"r"(T):"memory","ymm0","ymm1","ymm2","ymm3","ymm4","ymm5","ymm6");}

AVX2 static inline __attribute__((always_inline)) void mixed_build6(const uint64_t*r,__m256i*T){asm volatile(
"vmovdqa 0(%[r]),%%ymm0\nvmovdqa 32(%[r]),%%ymm1\nvmovdqa 64(%[r]),%%ymm2\nvmovdqa 96(%[r]),%%ymm3\nvmovdqa 128(%[r]),%%ymm4\nvmovdqa 160(%[r]),%%ymm5\nvpxor %%ymm6,%%ymm6,%%ymm6\nvmovdqa %%ymm6,0(%[t])\nvmovdqa %%ymm0,32(%[t])\nvpxor %%ymm1,%%ymm0,%%ymm6\nvmovdqa %%ymm6,96(%[t])\nvmovdqa %%ymm1,64(%[t])\nvpxor %%ymm2,%%ymm1,%%ymm6\nvmovdqa %%ymm6,192(%[t])\nvpxor %%ymm0,%%ymm6,%%ymm6\nvmovdqa %%ymm6,224(%[t])\nvpxor %%ymm1,%%ymm6,%%ymm6\nvmovdqa %%ymm6,160(%[t])\nvmovdqa %%ymm2,128(%[t])\nvpxor %%ymm3,%%ymm2,%%ymm6\nvmovdqa %%ymm6,384(%[t])\nvpxor %%ymm0,%%ymm6,%%ymm6\nvmovdqa %%ymm6,416(%[t])\nvpxor %%ymm1,%%ymm6,%%ymm6\nvmovdqa %%ymm6,480(%[t])\nvpxor %%ymm0,%%ymm6,%%ymm6\nvmovdqa %%ymm6,448(%[t])\nvpxor %%ymm2,%%ymm6,%%ymm6\nvmovdqa %%ymm6,320(%[t])\nvpxor %%ymm0,%%ymm6,%%ymm6\nvmovdqa %%ymm6,352(%[t])\nvpxor %%ymm1,%%ymm6,%%ymm6\nvmovdqa %%ymm6,288(%[t])\nvmovdqa %%ymm3,256(%[t])\nvpxor %%ymm4,%%ymm3,%%ymm6\nvmovdqa %%ymm6,768(%[t])\nvpxor %%ymm0,%%ymm6,%%ymm6\nvmovdqa %%ymm6,800(%[t])\nvpxor %%ymm1,%%ymm6,%%ymm6\nvmovdqa %%ymm6,864(%[t])\nvpxor %%ymm0,%%ymm6,%%ymm6\nvmovdqa %%ymm6,832(%[t])\nvpxor %%ymm2,%%ymm6,%%ymm6\nvmovdqa %%ymm6,960(%[t])\nvpxor %%ymm0,%%ymm6,%%ymm6\nvmovdqa %%ymm6,992(%[t])\nvpxor %%ymm1,%%ymm6,%%ymm6\nvmovdqa %%ymm6,928(%[t])\nvpxor %%ymm0,%%ymm6,%%ymm6\nvmovdqa %%ymm6,896(%[t])\nvpxor %%ymm3,%%ymm6,%%ymm6\nvmovdqa %%ymm6,640(%[t])\nvpxor %%ymm0,%%ymm6,%%ymm6\nvmovdqa %%ymm6,672(%[t])\nvpxor %%ymm1,%%ymm6,%%ymm6\nvmovdqa %%ymm6,736(%[t])\nvpxor %%ymm0,%%ymm6,%%ymm6\nvmovdqa %%ymm6,704(%[t])\nvpxor %%ymm2,%%ymm6,%%ymm6\nvmovdqa %%ymm6,576(%[t])\nvpxor %%ymm0,%%ymm6,%%ymm6\nvmovdqa %%ymm6,608(%[t])\nvpxor %%ymm1,%%ymm6,%%ymm6\nvmovdqa %%ymm6,544(%[t])\nvmovdqa %%ymm4,512(%[t])\nvpxor %%ymm5,%%ymm4,%%ymm6\nvmovdqa %%ymm6,1536(%[t])\nvpxor %%ymm0,%%ymm6,%%ymm6\nvmovdqa %%ymm6,1568(%[t])\nvpxor %%ymm1,%%ymm6,%%ymm6\nvmovdqa %%ymm6,1632(%[t])\nvpxor %%ymm0,%%ymm6,%%ymm6\nvmovdqa %%ymm6,1600(%[t])\nvpxor %%ymm2,%%ymm6,%%ymm6\nvmovdqa %%ymm6,1728(%[t])\nvpxor %%ymm0,%%ymm6,%%ymm6\nvmovdqa %%ymm6,1760(%[t])\nvpxor %%ymm1,%%ymm6,%%ymm6\nvmovdqa %%ymm6,1696(%[t])\nvpxor %%ymm0,%%ymm6,%%ymm6\nvmovdqa %%ymm6,1664(%[t])\nvpxor %%ymm3,%%ymm6,%%ymm6\nvmovdqa %%ymm6,1920(%[t])\nvpxor %%ymm0,%%ymm6,%%ymm6\nvmovdqa %%ymm6,1952(%[t])\nvpxor %%ymm1,%%ymm6,%%ymm6\nvmovdqa %%ymm6,2016(%[t])\nvpxor %%ymm0,%%ymm6,%%ymm6\nvmovdqa %%ymm6,1984(%[t])\nvpxor %%ymm2,%%ymm6,%%ymm6\nvmovdqa %%ymm6,1856(%[t])\nvpxor %%ymm0,%%ymm6,%%ymm6\nvmovdqa %%ymm6,1888(%[t])\nvpxor %%ymm1,%%ymm6,%%ymm6\nvmovdqa %%ymm6,1824(%[t])\nvpxor %%ymm0,%%ymm6,%%ymm6\nvmovdqa %%ymm6,1792(%[t])\nvpxor %%ymm4,%%ymm6,%%ymm6\nvmovdqa %%ymm6,1280(%[t])\nvpxor %%ymm0,%%ymm6,%%ymm6\nvmovdqa %%ymm6,1312(%[t])\nvpxor %%ymm1,%%ymm6,%%ymm6\nvmovdqa %%ymm6,1376(%[t])\nvpxor %%ymm0,%%ymm6,%%ymm6\nvmovdqa %%ymm6,1344(%[t])\nvpxor %%ymm2,%%ymm6,%%ymm6\nvmovdqa %%ymm6,1472(%[t])\nvpxor %%ymm0,%%ymm6,%%ymm6\nvmovdqa %%ymm6,1504(%[t])\nvpxor %%ymm1,%%ymm6,%%ymm6\nvmovdqa %%ymm6,1440(%[t])\nvpxor %%ymm0,%%ymm6,%%ymm6\nvmovdqa %%ymm6,1408(%[t])\nvpxor %%ymm3,%%ymm6,%%ymm6\nvmovdqa %%ymm6,1152(%[t])\nvpxor %%ymm0,%%ymm6,%%ymm6\nvmovdqa %%ymm6,1184(%[t])\nvpxor %%ymm1,%%ymm6,%%ymm6\nvmovdqa %%ymm6,1248(%[t])\nvpxor %%ymm0,%%ymm6,%%ymm6\nvmovdqa %%ymm6,1216(%[t])\nvpxor %%ymm2,%%ymm6,%%ymm6\nvmovdqa %%ymm6,1088(%[t])\nvpxor %%ymm0,%%ymm6,%%ymm6\nvmovdqa %%ymm6,1120(%[t])\nvpxor %%ymm1,%%ymm6,%%ymm6\nvmovdqa %%ymm6,1056(%[t])\nvmovdqa %%ymm5,1024(%[t])\n"
::[r]"r"(r),[t]"r"(T):"memory","ymm0","ymm1","ymm2","ymm3","ymm4","ymm5","ymm6");}

template<bool First>
R2M_BMI2 static void c4_corr_i_loop(const char*tb,int wb){
(void)tb;for(int i=0;i<256;i+=4){asm volatile(
"mov 0(%[a]),%%r8\nmov 8(%[a]),%%r9\n.if !%c[first]\nvmovdqa 0(%[c]),%%ymm0\nvmovdqa 32(%[c]),%%ymm1\n.endif\nrorx $59,%%r8,%%rax\nrorx $59,%%r9,%%rdx\nand $2016,%%eax\nand $2016,%%edx\n.if %c[first]\nvmovdqa c4_fixed_tb+0(%%eax),%%ymm0\nvmovdqa c4_fixed_tb+0(%%edx),%%ymm1\n.else\nvpxor c4_fixed_tb+0(%%eax),%%ymm0,%%ymm0\nvpxor c4_fixed_tb+0(%%edx),%%ymm1,%%ymm1\n.endif\nrorx $1,%%r8,%%rax\nrorx $1,%%r9,%%rdx\nand $2016,%%eax\nand $2016,%%edx\nvmovdqa c4_fixed_tb+2048(%%eax),%%ymm2\nvmovdqa c4_fixed_tb+2048(%%edx),%%ymm3\nrorx $7,%%r8,%%rax\nrorx $7,%%r9,%%rdx\nand $2016,%%eax\nand $2016,%%edx\nvpxor c4_fixed_tb+4096(%%eax),%%ymm0,%%ymm0\nvpxor c4_fixed_tb+4096(%%edx),%%ymm1,%%ymm1\nrorx $13,%%r8,%%rax\nrorx $13,%%r9,%%rdx\nand $2016,%%eax\nand $2016,%%edx\nvpxor c4_fixed_tb+6144(%%eax),%%ymm2,%%ymm2\nvpxor c4_fixed_tb+6144(%%edx),%%ymm3,%%ymm3\nrorx $19,%%r8,%%rax\nrorx $19,%%r9,%%rdx\nand $992,%%eax\nand $992,%%edx\nvpxor c4_fixed_tb+8192(%%eax),%%ymm0,%%ymm0\nvpxor c4_fixed_tb+8192(%%edx),%%ymm1,%%ymm1\nrorx $24,%%r8,%%rax\nrorx $24,%%r9,%%rdx\nand $992,%%eax\nand $992,%%edx\nvpxor c4_fixed_tb+9216(%%eax),%%ymm2,%%ymm2\nvpxor c4_fixed_tb+9216(%%edx),%%ymm3,%%ymm3\nrorx $29,%%r8,%%rax\nrorx $29,%%r9,%%rdx\nand $992,%%eax\nand $992,%%edx\nvpxor c4_fixed_tb+10240(%%eax),%%ymm0,%%ymm0\nvpxor c4_fixed_tb+10240(%%edx),%%ymm1,%%ymm1\nrorx $34,%%r8,%%rax\nrorx $34,%%r9,%%rdx\nand $992,%%eax\nand $992,%%edx\nvpxor c4_fixed_tb+11264(%%eax),%%ymm2,%%ymm2\nvpxor c4_fixed_tb+11264(%%edx),%%ymm3,%%ymm3\nrorx $39,%%r8,%%rax\nrorx $39,%%r9,%%rdx\nand $992,%%eax\nand $992,%%edx\nvpxor c4_fixed_tb+12288(%%eax),%%ymm0,%%ymm0\nvpxor c4_fixed_tb+12288(%%edx),%%ymm1,%%ymm1\nrorx $44,%%r8,%%rax\nrorx $44,%%r9,%%rdx\nand $992,%%eax\nand $992,%%edx\nvpxor c4_fixed_tb+13312(%%eax),%%ymm2,%%ymm2\nvpxor c4_fixed_tb+13312(%%edx),%%ymm3,%%ymm3\nrorx $49,%%r8,%%rax\nrorx $49,%%r9,%%rdx\nand $992,%%eax\nand $992,%%edx\nvpxor c4_fixed_tb+14336(%%eax),%%ymm0,%%ymm0\nvpxor c4_fixed_tb+14336(%%edx),%%ymm1,%%ymm1\nrorx $54,%%r8,%%rax\nrorx $54,%%r9,%%rdx\nand $992,%%eax\nand $992,%%edx\nvpxor c4_fixed_tb+15360(%%eax),%%ymm2,%%ymm2\nvpxor c4_fixed_tb+15360(%%edx),%%ymm3,%%ymm3\nvpxor %%ymm2,%%ymm0,%%ymm0\nvpxor %%ymm3,%%ymm1,%%ymm1\nvmovdqa %%ymm0,0(%[c])\nvmovdqa %%ymm1,32(%[c])\n"
::[a]"r"(A1b+wb*256+i),[c]"r"(iv363_out+i*4),[first]"i"(First):"rax","rdx","r8","r9","ymm0","ymm1","ymm2","ymm3","memory","cc");asm volatile(
"mov 0(%[a]),%%r8\nmov 8(%[a]),%%r9\n.if !%c[first]\nvmovdqa 0(%[c]),%%ymm0\nvmovdqa 32(%[c]),%%ymm1\n.endif\nrorx $59,%%r8,%%rax\nrorx $59,%%r9,%%rdx\nand $2016,%%eax\nand $2016,%%edx\n.if %c[first]\nvmovdqa c4_fixed_tb+0(%%eax),%%ymm0\nvmovdqa c4_fixed_tb+0(%%edx),%%ymm1\n.else\nvpxor c4_fixed_tb+0(%%eax),%%ymm0,%%ymm0\nvpxor c4_fixed_tb+0(%%edx),%%ymm1,%%ymm1\n.endif\nrorx $1,%%r8,%%rax\nrorx $1,%%r9,%%rdx\nand $2016,%%eax\nand $2016,%%edx\nvmovdqa c4_fixed_tb+2048(%%eax),%%ymm2\nvmovdqa c4_fixed_tb+2048(%%edx),%%ymm3\nrorx $7,%%r8,%%rax\nrorx $7,%%r9,%%rdx\nand $2016,%%eax\nand $2016,%%edx\nvpxor c4_fixed_tb+4096(%%eax),%%ymm0,%%ymm0\nvpxor c4_fixed_tb+4096(%%edx),%%ymm1,%%ymm1\nrorx $13,%%r8,%%rax\nrorx $13,%%r9,%%rdx\nand $2016,%%eax\nand $2016,%%edx\nvpxor c4_fixed_tb+6144(%%eax),%%ymm2,%%ymm2\nvpxor c4_fixed_tb+6144(%%edx),%%ymm3,%%ymm3\nrorx $19,%%r8,%%rax\nrorx $19,%%r9,%%rdx\nand $992,%%eax\nand $992,%%edx\nvpxor c4_fixed_tb+8192(%%eax),%%ymm0,%%ymm0\nvpxor c4_fixed_tb+8192(%%edx),%%ymm1,%%ymm1\nrorx $24,%%r8,%%rax\nrorx $24,%%r9,%%rdx\nand $992,%%eax\nand $992,%%edx\nvpxor c4_fixed_tb+9216(%%eax),%%ymm2,%%ymm2\nvpxor c4_fixed_tb+9216(%%edx),%%ymm3,%%ymm3\nrorx $29,%%r8,%%rax\nrorx $29,%%r9,%%rdx\nand $992,%%eax\nand $992,%%edx\nvpxor c4_fixed_tb+10240(%%eax),%%ymm0,%%ymm0\nvpxor c4_fixed_tb+10240(%%edx),%%ymm1,%%ymm1\nrorx $34,%%r8,%%rax\nrorx $34,%%r9,%%rdx\nand $992,%%eax\nand $992,%%edx\nvpxor c4_fixed_tb+11264(%%eax),%%ymm2,%%ymm2\nvpxor c4_fixed_tb+11264(%%edx),%%ymm3,%%ymm3\nrorx $39,%%r8,%%rax\nrorx $39,%%r9,%%rdx\nand $992,%%eax\nand $992,%%edx\nvpxor c4_fixed_tb+12288(%%eax),%%ymm0,%%ymm0\nvpxor c4_fixed_tb+12288(%%edx),%%ymm1,%%ymm1\nrorx $44,%%r8,%%rax\nrorx $44,%%r9,%%rdx\nand $992,%%eax\nand $992,%%edx\nvpxor c4_fixed_tb+13312(%%eax),%%ymm2,%%ymm2\nvpxor c4_fixed_tb+13312(%%edx),%%ymm3,%%ymm3\nrorx $49,%%r8,%%rax\nrorx $49,%%r9,%%rdx\nand $992,%%eax\nand $992,%%edx\nvpxor c4_fixed_tb+14336(%%eax),%%ymm0,%%ymm0\nvpxor c4_fixed_tb+14336(%%edx),%%ymm1,%%ymm1\nrorx $54,%%r8,%%rax\nrorx $54,%%r9,%%rdx\nand $992,%%eax\nand $992,%%edx\nvpxor c4_fixed_tb+15360(%%eax),%%ymm2,%%ymm2\nvpxor c4_fixed_tb+15360(%%edx),%%ymm3,%%ymm3\nvpxor %%ymm2,%%ymm0,%%ymm0\nvpxor %%ymm3,%%ymm1,%%ymm1\nvmovdqa %%ymm0,0(%[c])\nvmovdqa %%ymm1,32(%[c])\n"
::[a]"r"(A1b+wb*256+i+2),[c]"r"(iv363_out+i*4+8),[first]"i"(First):"rax","rdx","r8","r9","ymm0","ymm1","ymm2","ymm3","memory","cc");}}
R2M_BMI2 static void corr_six256(){
const char*tb=(const char*)TB;
for(int wb=0;wb<4;wb++){
for(int g=0;g<4;g++)mixed_build6(Bpb+(wb*64+g*6)*4,(__m256i*)TB+g*64);
for(int g=0;g<8;g++)mixed_build5(Bpb+(wb*64+24+g*5)*4,(__m256i*)TB+256+g*32);
if(wb==0) c4_corr_i_loop<true>(tb,0); else c4_corr_i_loop<false>(tb,wb);
}

}
AVX2 static void submm_tail(int m_,int8_t*Cp){constexpr int m=256;
GLDA = 2*(m) + PADA; GLDB = m + PADB;
if (m==1024) nibcorr1024(); else if (m==512) nibcorr512();
else if (m==256) corr_six256(); else corr_pass3(m);
const int GB=1024, IB=1024;
for (int ib=0; ib<m; ib+=IB){
const int ihi = (ib+IB<m)? ib+IB : m;
for (int gb=0; gb<m; gb+=GB){
const int ghi = (gb+GB<m)? gb+GB : m;
for (int g=gb; g<ghi; g+=64){
const int8_t *bi = BI3b+(size_t)(g>>6)*(m/2)*128;
{ int rev=0;
for (int i=ib; i<ihi; i+=(m==256?256:2)){
const int8_t *ap=A0b+(size_t)i*GLDA;
const uint8_t *pw = (const uint8_t*)(Pbb+(size_t)i*(m>>6)+(g>>6));
c4_cross_row_micro(m,ap,bi,Cp+(size_t)i*m+g);
rev^=1; }
}
}
}
}
}
AVX2 static void submm_plain(int m,const int8_t*Ap,const int8_t*Bp,int8_t*Cp){
GLDA = 2*(m) + PADA; GLDB = m + PADB;
build_A0A1(m,Ap); build_BI3Bp(m,Bp); submm_tail(m,Cp);
}
AVX2 static void submm(int m,const int8_t*Ap,const int8_t*Bp,int8_t*Cp);
#define S2MIN 1024
AVX2 static void submm_s4(int m,const int8_t*X,const int8_t*Y,int8_t*dst);
AVX2 static void submm_s5f(int M,const int8_t*Ap,int Aa0,int Aa1,int Aa2,int Aa3,int Aop,
const int8_t*Bp,int Ba0,int Ba1,int Ba2,int Ba3,int Bop,int8_t*Cp);
alignas(4096) static int8_t outerA[6*65536+64],outerB[6*65536+64];
 AVX2 static inline void foldOuter(const int8_t*&src,int&x,int&y,int&u,int&v,int op,int8_t*dst,int h){
 if(!op)return;
 for(int yy=0;yy<2;yy++)for(int xx=0;xx<2;xx++){
  size_t q=4*yy+xx,area=(size_t)h*h;const int8_t*a=src+(4*(x/h)+y/h+q)*area,*b=src+(4*(u/h)+v/h+q)*area;
  int8_t*out=dst+q*area;
  if(op>0){for(size_t k=0;k<area;k+=32)_mm256_store_si256((__m256i*)(out+k),_mm256_add_epi8(_mm256_load_si256((const __m256i*)(a+k)),_mm256_load_si256((const __m256i*)(b+k))));}
  else{for(size_t k=0;k<area;k+=32)_mm256_store_si256((__m256i*)(out+k),_mm256_sub_epi8(_mm256_load_si256((const __m256i*)(a+k)),_mm256_load_si256((const __m256i*)(b+k))));}
 }src=dst;x=y=u=v=0;
 }
AVX2 static void submm_s5f(int M,const int8_t*Ap,int Aa0,int Aa1,int Aa2,int Aa3,int Aop,
const int8_t*Bp,int Ba0,int Ba1,int Ba2,int Ba3,int Bop,int8_t*Cp){
const int h=M>>2, o=h;
#define NRB(rr,cc) ((((rr)/h)*4 + ((cc)/h))*(h))
const int SQA=(Aop==1)?1:((Aop==2)?-1:0);
int SQB=(Bop==1)?1:((Bop==2)?-1:0);
foldOuter(Bp,Ba0,Ba1,Ba2,Ba3,SQB,outerB+0,h);SQB=0;
const int dz=M>>1;
GLDA = 2*h + PADA; GLDB = h + PADB;
for (int k=0;k<7;k++){iv363_out=iv363_pb[k];
switch(k){
case 0: buildA_q4(h,h,Ap,NRB(Aa0,Aa1),0, NRB(Aa2,Aa3),0,SQA, NRB(Aa0+o,Aa1+o),0,+1, NRB(Aa2+o,Aa3+o),0,(SQA)*(+1));
buildB_q4(h,h,Bp,NRB(Ba0,Ba1),0, NRB(Ba2,Ba3),0,SQB, NRB(Ba0+o,Ba1+o),0,+1, NRB(Ba2+o,Ba3+o),0,(SQB)*(+1));
submm_tail(h,S4M[0]); break;
case 1: buildA_q4(h,h,Ap,NRB(Aa0+o,Aa1),0, NRB(Aa2+o,Aa3),0,SQA, NRB(Aa0+o,Aa1+o),0,+1, NRB(Aa2+o,Aa3+o),0,(SQA)*(+1));
buildB_q4(h,h,Bp,NRB(Ba0,Ba1),0, NRB(Ba2,Ba3),0,SQB, NRB(Ba0,Ba1),0,+0, NRB(Ba2,Ba3),0,(SQB)*(+0));
submm_tail(h,S4M[1]); break;
case 2: buildA_q4(h,h,Ap,NRB(Aa0,Aa1),0, NRB(Aa2,Aa3),0,SQA, NRB(Aa0,Aa1),0,+0, NRB(Aa2,Aa3),0,(SQA)*(+0));
buildB_q4(h,h,Bp,NRB(Ba0,Ba1+o),0, NRB(Ba2,Ba3+o),0,SQB, NRB(Ba0+o,Ba1+o),0,-1, NRB(Ba2+o,Ba3+o),0,(SQB)*(-1));
submm_tail(h,S4M[2]); break;
case 3: buildA_q4(h,h,Ap,NRB(Aa0+o,Aa1+o),0, NRB(Aa2+o,Aa3+o),0,SQA, NRB(Aa0+o,Aa1+o),0,+0, NRB(Aa2+o,Aa3+o),0,(SQA)*(+0));
buildB_q4(h,h,Bp,NRB(Ba0+o,Ba1),0, NRB(Ba2+o,Ba3),0,SQB, NRB(Ba0,Ba1),0,-1, NRB(Ba2,Ba3),0,(SQB)*(-1));
submm_tail(h,S4M[3]); break;
case 4: buildA_q4(h,h,Ap,NRB(Aa0,Aa1),0, NRB(Aa2,Aa3),0,SQA, NRB(Aa0,Aa1+o),0,+1, NRB(Aa2,Aa3+o),0,(SQA)*(+1));
buildB_q4(h,h,Bp,NRB(Ba0+o,Ba1+o),0, NRB(Ba2+o,Ba3+o),0,SQB, NRB(Ba0+o,Ba1+o),0,+0, NRB(Ba2+o,Ba3+o),0,(SQB)*(+0));
submm_tail(h,S4M[4]); break;
case 5: buildA_q4(h,h,Ap,NRB(Aa0+o,Aa1),0, NRB(Aa2+o,Aa3),0,SQA, NRB(Aa0,Aa1),0,-1, NRB(Aa2,Aa3),0,(SQA)*(-1));
buildB_q4(h,h,Bp,NRB(Ba0,Ba1),0, NRB(Ba2,Ba3),0,SQB, NRB(Ba0,Ba1+o),0,+1, NRB(Ba2,Ba3+o),0,(SQB)*(+1));
submm_tail(h,S4M[5]); break;
case 6: buildA_q4(h,h,Ap,NRB(Aa0,Aa1+o),0, NRB(Aa2,Aa3+o),0,SQA, NRB(Aa0+o,Aa1+o),0,-1, NRB(Aa2+o,Aa3+o),0,(SQA)*(-1));
buildB_q4(h,h,Bp,NRB(Ba0+o,Ba1),0, NRB(Ba2+o,Ba3),0,SQB, NRB(Ba0+o,Ba1+o),0,+1, NRB(Ba2+o,Ba3+o),0,(SQB)*(+1));
submm_tail(h,S4M[6]); break;
}
}
#undef NRB
iv363_out=Pbb;
_mm_sfence();
if(!c4_skip_last)cmb4wb(Cp,dz,h,S4M);
_mm_sfence();
}
AVX2 static void submm_s5(int m,const int8_t*Ap,const int8_t*Bp,int8_t*Cp);
AVX2 static void submm_s5(int m,const int8_t*Ap,const int8_t*Bp,int8_t*Cp){
GLDA = 2*((m>>1)) + PADA; GLDB = (m>>1) + PADB;
const int h=m>>1, o=h;
for (int k=0;k<7;k++){
switch(k){
case 0: buildA_q(h,m,Ap,0,0,Ap,o,o,1); buildB_q(h,m,Bp,0,0,Bp,o,o,1);  submm_tail(h,S4M[0]); break;
case 1: buildA_q(h,m,Ap,o,0,Ap,o,o,1); buildB_q(h,m,Bp,0,0,Bp,0,0,0);  submm_tail(h,S4M[1]); break;
case 2: buildA_q(h,m,Ap,0,0,Ap,0,0,0); buildB_q(h,m,Bp,0,o,Bp,o,o,2);  submm_tail(h,S4M[2]); break;
case 3: buildA_q(h,m,Ap,o,o,Ap,o,o,0); buildB_q(h,m,Bp,o,0,Bp,0,0,2);  submm_tail(h,S4M[3]); break;
case 4: buildA_q(h,m,Ap,0,0,Ap,0,o,1); buildB_q(h,m,Bp,o,o,Bp,o,o,0);  submm_tail(h,S4M[4]); break;
case 5: buildA_q(h,m,Ap,o,0,Ap,0,0,2); buildB_q(h,m,Bp,0,0,Bp,0,o,1);  submm_tail(h,S4M[5]); break;
case 6: buildA_q(h,m,Ap,0,o,Ap,o,o,2); buildB_q(h,m,Bp,o,0,Bp,o,o,1);  submm_tail(h,S4M[6]); break;
}
}
_mm_sfence();
cmb(Cp,m,0,0,h, S4M[0],1,S4M[3],1,S4M[4],-1,S4M[6],1);
cmb(Cp,m,0,o,h, S4M[2],1,S4M[4],1,0,0,0,0);
cmb(Cp,m,o,0,h, S4M[1],1,S4M[3],1,0,0,0,0);
cmb(Cp,m,o,o,h, S4M[0],1,S4M[2],1,S4M[1],-1,S4M[5],1);
_mm_sfence();
}
AVX2 static void submm_s4(int m,const int8_t*Ap,const int8_t*Bp,int8_t*Cp){
GLDA = 2*((m>>1)) + PADA; GLDB = (m>>1) + PADB;
const int h=m>>1, o=h;
for (int k=0;k<7;k++){
switch(k){
case 0: submm_s5f(m,Ap,0,0,o,o,1, Bp,0,0,o,o,1, S3M[0]); break;
case 1: submm_s5f(m,Ap,o,0,o,o,1, Bp,0,0,0,0,0, S3M[1]); break;
case 2: submm_s5f(m,Ap,0,0,0,0,0, Bp,0,o,o,o,2, S3M[2]); break;
case 3: submm_s5f(m,Ap,o,o,o,o,0, Bp,o,0,0,0,2, S3M[3]); break;
case 4: submm_s5f(m,Ap,0,0,0,o,1, Bp,o,o,o,o,0, S3M[4]); break;
case 5: submm_s5f(m,Ap,o,0,0,0,2, Bp,0,0,0,o,1, S3M[5]); break;
case 6: c4_skip_last=1;submm_s5f(m,Ap,0,o,o,o,2, Bp,o,0,o,o,1, S3M[6]);c4_skip_last=0;break;
}
}
_mm_sfence();
cmb4fu(Cp,m,h,S3M,S4M);
_mm_sfence();
}
AVX2 static void build_combsA5t(const int8_t*X,int n,int h,int8_t*o0,int8_t*o1,int8_t*o4,int8_t*o5,int8_t*o6){
const int B=h>>2, BB=B*B;
for(int i=0;i<h;i++){
const int8_t*p00=X+(size_t)i*n, *p01=p00+h, *p10=p00+(size_t)h*n, *p11=p10+h;
const size_t base=(size_t)((i/B)*4)*BB+(size_t)(i%B)*B;
for(int J=0;J<4;J++){
const int8_t*x00=p00+J*B,*x01=p01+J*B,*x10=p10+J*B,*x11=p11+J*B;
const size_t o=base+(size_t)J*BB;
for(int j=0;j<B;j+=64){
__m256i v00=_mm256_load_si256((const __m256i*)(x00+j)), w00=_mm256_load_si256((const __m256i*)(x00+j+32));
__m256i v01=_mm256_load_si256((const __m256i*)(x01+j)), w01=_mm256_load_si256((const __m256i*)(x01+j+32));
__m256i v10=_mm256_load_si256((const __m256i*)(x10+j)), w10=_mm256_load_si256((const __m256i*)(x10+j+32));
__m256i v11=_mm256_load_si256((const __m256i*)(x11+j)), w11=_mm256_load_si256((const __m256i*)(x11+j+32));
_mm256_stream_si256((__m256i*)(o0+o+j),_mm256_add_epi8(v00,v11));
_mm256_stream_si256((__m256i*)(o0+o+j+32),_mm256_add_epi8(w00,w11));
_mm256_stream_si256((__m256i*)(o1+o+j),_mm256_add_epi8(v10,v11));
_mm256_stream_si256((__m256i*)(o1+o+j+32),_mm256_add_epi8(w10,w11));
_mm256_stream_si256((__m256i*)(o4+o+j),_mm256_add_epi8(v00,v01));
_mm256_stream_si256((__m256i*)(o4+o+j+32),_mm256_add_epi8(w00,w01));
_mm256_stream_si256((__m256i*)(o5+o+j),_mm256_sub_epi8(v10,v00));
_mm256_stream_si256((__m256i*)(o5+o+j+32),_mm256_sub_epi8(w10,w00));
_mm256_stream_si256((__m256i*)(o6+o+j),_mm256_sub_epi8(v01,v11));
_mm256_stream_si256((__m256i*)(o6+o+j+32),_mm256_sub_epi8(w01,w11));
}
}
}
_mm_sfence();
}
AVX2 static void submm_s2(int m,const int8_t*Ap,const int8_t*Bp,int8_t*Cp,int la,int lb){
GLDA = 2*((m>>1)) + PADA; GLDB = (m>>1) + PADB;
const int h=m>>1, o=h;
build_combsA5t(Ap,la,h,S2M[0],S2M[1],S2M[4],S2M[5],S2M[6]);
for (int k=0;k<7;k++){
switch(k){
case 0: blk(S3B,Bp,lb,0,0,h,Bp,o,o,1); submm_s4(h,S2M[0],S3B,S2M[0]); break;
case 1: blk(S3B,Bp,lb,0,0,h,Bp,0,0,0); submm_s4(h,S2M[1],S3B,S2M[1]); break;
case 2: blk(S3A,Ap,la,0,0,h,Ap,0,0,0); blk(S3B,Bp,lb,0,o,h,Bp,o,o,2); submm_s4(h,S3A,S3B,S2M[2]); break;
case 3: blk(S3A,Ap,la,o,o,h,Ap,o,o,0); blk(S3B,Bp,lb,o,0,h,Bp,0,0,2); submm_s4(h,S3A,S3B,S2M[3]); break;
case 4: blk(S3B,Bp,lb,o,o,h,Bp,o,o,0); submm_s4(h,S2M[4],S3B,S2M[4]); break;
case 5: blk(S3B,Bp,lb,0,0,h,Bp,0,o,1); submm_s4(h,S2M[5],S3B,S2M[5]); break;
case 6: blk(S3B,Bp,lb,o,0,h,Bp,o,o,1); submm_s4(h,S2M[6],S3B,S2M[6]); break;
}
}
_mm_sfence();
cmb4(Cp,m,h,S2M);
_mm_sfence();
}
AVX2 static void submm(int m,const int8_t*Ap,const int8_t*Bp,int8_t*Cp){
if (m>=S2MIN) { submm_s2(m,Ap,Bp,Cp,m,m); return; }
submm_plain(m,Ap,Bp,Cp);
}
static volatile char r9c4k_pad64[312] __attribute__((used));
AVX2 static void build_combsA5(const int8_t*X,int n,int m,int8_t*o0,int8_t*o1,int8_t*o4,int8_t*o5,int8_t*o6){
for(int i=0;i<m;i++){
const int8_t*p00=X+(size_t)i*n, *p01=p00+m, *p10=p00+(size_t)m*n, *p11=p10+m;
int8_t*d0=o0+(size_t)i*m,*d1=o1+(size_t)i*m,*d4=o4+(size_t)i*m,*d5=o5+(size_t)i*m,*d6=o6+(size_t)i*m;
for(int j=0;j<m;j+=64){
__m256i v00=_mm256_loadu_si256((const __m256i*)(p00+j)), w00=_mm256_loadu_si256((const __m256i*)(p00+j+32));
__m256i v01=_mm256_loadu_si256((const __m256i*)(p01+j)), w01=_mm256_loadu_si256((const __m256i*)(p01+j+32));
__m256i v10=_mm256_loadu_si256((const __m256i*)(p10+j)), w10=_mm256_loadu_si256((const __m256i*)(p10+j+32));
__m256i v11=_mm256_loadu_si256((const __m256i*)(p11+j)), w11=_mm256_loadu_si256((const __m256i*)(p11+j+32));
_mm256_stream_si256((__m256i*)(d0+j),_mm256_add_epi8(v00,v11));
_mm256_stream_si256((__m256i*)(d0+j+32),_mm256_add_epi8(w00,w11));
_mm256_stream_si256((__m256i*)(d1+j),_mm256_add_epi8(v10,v11));
_mm256_stream_si256((__m256i*)(d1+j+32),_mm256_add_epi8(w10,w11));
_mm256_stream_si256((__m256i*)(d4+j),_mm256_add_epi8(v00,v01));
_mm256_stream_si256((__m256i*)(d4+j+32),_mm256_add_epi8(w00,w01));
_mm256_stream_si256((__m256i*)(d5+j),_mm256_sub_epi8(v10,v00));
_mm256_stream_si256((__m256i*)(d5+j+32),_mm256_sub_epi8(w10,w00));
_mm256_stream_si256((__m256i*)(d6+j),_mm256_sub_epi8(v01,v11));
_mm256_stream_si256((__m256i*)(d6+j+32),_mm256_sub_epi8(w01,w11));
}
}
_mm_sfence();
}
AVX2 static void build_combsB5(const int8_t*X,int n,int m,int8_t*o0,int8_t*o2,int8_t*o3,int8_t*o5,int8_t*o6){
for(int i=0;i<m;i++){
const int8_t*p00=X+(size_t)i*n, *p01=p00+m, *p10=p00+(size_t)m*n, *p11=p10+m;
int8_t*d0=o0+(size_t)i*m,*d2=o2+(size_t)i*m,*d3=o3+(size_t)i*m,*d5=o5+(size_t)i*m,*d6=o6+(size_t)i*m;
for(int j=0;j<m;j+=64){
__m256i v00=_mm256_loadu_si256((const __m256i*)(p00+j)), w00=_mm256_loadu_si256((const __m256i*)(p00+j+32));
__m256i v01=_mm256_loadu_si256((const __m256i*)(p01+j)), w01=_mm256_loadu_si256((const __m256i*)(p01+j+32));
__m256i v10=_mm256_loadu_si256((const __m256i*)(p10+j)), w10=_mm256_loadu_si256((const __m256i*)(p10+j+32));
__m256i v11=_mm256_loadu_si256((const __m256i*)(p11+j)), w11=_mm256_loadu_si256((const __m256i*)(p11+j+32));
_mm256_stream_si256((__m256i*)(d0+j),_mm256_add_epi8(v00,v11));
_mm256_stream_si256((__m256i*)(d0+j+32),_mm256_add_epi8(w00,w11));
_mm256_stream_si256((__m256i*)(d2+j),_mm256_sub_epi8(v01,v11));
_mm256_stream_si256((__m256i*)(d2+j+32),_mm256_sub_epi8(w01,w11));
_mm256_stream_si256((__m256i*)(d3+j),_mm256_sub_epi8(v10,v00));
_mm256_stream_si256((__m256i*)(d3+j+32),_mm256_sub_epi8(w10,w00));
_mm256_stream_si256((__m256i*)(d5+j),_mm256_add_epi8(v00,v01));
_mm256_stream_si256((__m256i*)(d5+j+32),_mm256_add_epi8(w00,w01));
_mm256_stream_si256((__m256i*)(d6+j),_mm256_add_epi8(v10,v11));
_mm256_stream_si256((__m256i*)(d6+j+32),_mm256_add_epi8(w10,w11));
}
}
_mm_sfence();
}
AVX2 static void strassen1(int n,const int8_t*A,const int8_t*B,int8_t*C){
const int m=n>>1, o=m;
int8_t*a0=MMS[0],*a1=MMS[1],*a4=MMS[4],*a5=MMS[5],*a6=MMS[6];
build_combsA5(A,n,m,a0,a1,a4,a5,a6);
const size_t SZ=(size_t)m*m;
int8_t*b2=MMS[2],*b3=MMS[3],*b0=C,*b5=C+SZ,*b6=C+2*SZ;
build_combsB5(B,n,m,b0,b2,b3,b5,b6);
for (int k=0;k<7;k++){
const int8_t*pa=a0;int la=m;const int8_t*pb=b2;int lb=m;
switch(k){
case 0: pb=b0; break;
case 1: pa=a1; pb=B; lb=n; break;
case 2: pa=A; la=n; pb=b2; break;
case 3: pa=A+(size_t)o*n+o; la=n; pb=b3; break;
case 4: pa=a4; pb=B+(size_t)o*n+o; lb=n; break;
case 5: pa=a5; pb=b5; break;
case 6: pa=a6; pb=b6; break;
}
submm_s2(m,pa,pb,MMS[k],la,lb);
}
_mm_sfence();
cmb4(C,n,m,MMS);
_mm_sfence();
}
AVX2 void matrix_multiply(int n,const int8_t*A,const int8_t*B,int8_t*C){if(n==4096)strassen1(n,A,B,C);else generic_mm(n,A,B,C);}

CompilationN/AN/ACompile OKScore: N/A

Testcase #1319.014 ms55 MB + 620 KBAcceptedScore: 100


Judge Duck Online | 评测鸭在线
Server Time: 2026-11-19 09:41:17 | Loaded in 1 ms | Server Status
个人娱乐项目,仅供学习交流使用 | 捐赠