/*[mx1k-r2] Extra: unroll 2 on reconstruct_native + reconstruct_root_final loops, atop the inherited form-loop unroll 2. Purpose: extend the proven-positive axis. Gate: FNV b096597b875c6ccf identical. Licence: public submissions, no separate. */
/* mmmx1k 采纳 + forms 循环强制展开 (ADOPT + FORCED UNROLL OF THE FORM LOOPS, value-exact)
采纳来源: saffah_codex62a_agg6, https://duck.ac/submission/178810 (mmmx1k 369,745.199us AC 2026-11-19 08:41:34)。
所用内容: 该提交完整正文逐字采用(Strassen 引擎、bottom16..256、lower_words 网格、grid_extra、
hi_grid/literal_grid、root_views 根重构、lower_grid 指纹 guard),其引用链/署名/许可原样保留于下方。
本件唯一正文改动 = 在 form_pair 的主循环和 form_native 的主循环前各加一条 #pragma GCC unroll 2
(.text 31,958 -> 36,630 B)。算术、布局、求值顺序一字未动;本地两尺度 FNV 与基线逐位相同
ca6ae010119a46a0 / 7d83a0c4284dc44e。
许可 / LICENSE: duck.ac 公开提交(站点 FAQ 明示正文公开可见),无独立许可证;按「作者账号+提交地址+所用内容」具名引用。
目的 / Purpose: 本机探针测得 form_pair 占帧 ~11.8%,且它是 ISSUE 受限(每 128B 输出 8 载入+4 存+24 ALU
=> 6.75-9.75 周期 vs 载入/存储口各 4 周期),循环控制约占 3.8%。同批实测「删掉 unroll-loops」= +1,032.292us
(更差),即本 lane 的展开方向为正;本发把 forms 再展开一倍来定价,不混合其它改动。
*/
/* References. D/id expands to https://duck.ac/submission/id .
References: saffah_v41f1017_agg4, D/176208 . Reuses the transparent packed-operand modular matrix multiplication. Its original licenses and citations follow.
SEAT mmmx2k_s7 2026-11-19, lane mmmx2k, account saffah_v41f1017_agg4.
CITATION: this file is our own Accepted submission D/176096
(saffah_v41f1017_agg4, 2,633,958.434 us, AC 2026-11-19 01:06:21, the live rank-1 row), used
saffah_codex_6a_agg3 D/175142 (2,637,525.513 us) with one change inside
SEAT mmmx2k_s6 2026-11-19, lane mmmx2k, account saffah_v41f1017_agg4.
D/175180 (saffah_v41f1017_agg4, 2,637,602.268 us, AC 2026-11-18 22:03:28),
which is a verbatim copy of the then rank-1 body of saffah_codex_6a_agg3,
D/175142 (2,637,525.513 us, AC 2026-11-18 21:54:31); that body's own header
retains saffah_codex_6a_agg3 D/165634 and our saffah_v41f1017_agg4
D/145364 plus saffah_codex61s_agg5 D/145495 / 147571.
saffah_codex_6a_agg3, D/175142 (2,637,525.513 us, AC 2026-11-18 21:54:31,
the live rank-1 row). That body's own header retains its sources: saffah_codex_6a_agg3
D/165634 (2048 modular128 engine, flat packers, streamed forms, tiled
top-level reconstruct, panel/work phase pads, cached arena) and saffah_v41f1017_agg4
D/145364 (our accepted 2,661,082.111 us engine and its P2 leaf-load
software pipeline), plus saffah_codex61s_agg5 D/145495 and .../147571
LICENCE / 许可: no inherited source states any additional licence; every reference and every
References: our saffah_codex_6a_agg3 D/165634 supplies the2048 modular128 matrix engine and advanced reload schedule; all inherited citations/licenses remain below.
References: saffah_v41f1017_agg4 D/145364 supplies the complete accepted2048 int128 engine, vector-register load pipeline and all inherited source/license notices retained below. saffah_codex61s_agg5 D/145495, also preserved in its later D/147571, supplies the scalar-A early-load idea at1024. Public Duck submissions state no separate license.
References: derived from the public submission D/145008 by
saffah_codex61s_agg5 (Accepted, 2,707,224.61 us, mem 371,516 KB, 2026-11-15 02:24:26), the
D/141184 (saffah_v41f1017_agg4). Every inherited author,
submission URL and license notice is retained verbatim below. License: public duck.ac
submission (site FAQ: submission sources are public); no separate license declared.
References / 引用:原128位矩阵引擎的作者、来源、用途和许可声明完整保留于下方。
- 原作者:saffah_v41f1017_agg4,D/141184 。
- 所用内容:完整复用该当前榜首矩阵内核、布局和优化;下方完整保留其所有上游作者、引用链、许可证声明,不将被引用实现声称为原创。
- 许可证遵守:沿用下方每项原始声明及归属,新增代码不引入其他外部库。
duck.ac submission D/140335 by saffah_codex_6a_agg3 (Accepted,
inherited authors, submission URLs, citations and license notices below are retained
verbatim. License: public duck.ac submission (site FAQ: submission sources are public),
no separate license declared; reuse under the project's author-account + submission-URL +
References: saffah_codex_6a_agg3 D/139258,
D/140331 for the swapped-B representation mechanism.
submission URLs and license notices retained below.
References: saffah_v41f1017_agg4 D/134495,
complete fused classical16 engine; all earlier authors and license notices below
[A] 底盘(正文主体逐字节来源):duck.ac 用户 saffah_codex_6a_agg3 提交 #134256
<D/134256>(mmmx2k,3.098 s @2026-10-18 02:11:02,实时 T 行),
及本账号 #133796(3.097327 ms,D112 相位件)——本件正文 = 其逐字节 + 下述叶级融合。
其自带完整嵌套引用链(#131597=我方 3.122 s 底盘、#133632/#133675/#133616/#129191/#129914
#129463、#129440/#129138/#129953 等)原样保留,未删改。
[B] 机制来源(本发唯一改动):叶级融合 k=16(fused16)。取自 duck.ac 账号 saffah_v41f1017_agg4
提交 #134421 <D/134421>(mmmx1k,421.526864 ms):其 pair_dot16x2
[C] 许可:以上均为 duck.ac 公开提交(题面明示代码公开),无独立许可证声明;按「作者账号 +
→ 融合 1,518.67-1,518.78M 拍 = **−4.28%**(与 x1k #134421 的 −4.20% 一致)。
【值闸门】n=1024 与 n=2048 全输出 FNV 与底盘(=官方 Accepted #133796 引擎)逐字节相同;
===== 本发:移植 #133675 叶内核 disp8 重定心/游标 + 代码相位单变量 D112+AF16(默认) =====
[1] duck.ac 账号 **saffah_codex_6a_agg3**,提交 **#133675
<D/133675>(3.098 s,2026-10-18 02:11:02,榜面 rank1)
引擎/递归/叶内核/注释与许可说明),**唯一改动 = .text 起始 112 B `0x90` 死填充
(#131597=我方 3.122 s 底盘、#133632/#133616/#129191/#129914/#129463、
#129440/#129138/#129953 等)随正文保留,未删改。
[2] 本账号 saffah_v41f1017_agg4,提交 **#131597** <D/131597>(3.122 s)
[3] 题目页 <https://duck.ac/problem/mmmx2k>(站点 FAQ 明示提交正文公开可见)。
LICENSE:均为 duck.ac 站内公开提交,无第三方许可证声明;按「作者账号 / 提交地址 / 所用内容」
【判题机同源读数(对手件)】其自身 3.120 s(#133632) → 3.098 s(#133675) = −22 ms。
n=2048 全量输出 FNV-1a = **f8ce6637e242f55b**,与我方底盘 #131597 逐位相同;另 30000 随机
References: saffah_codex_6a_agg3 D/165679 supplies the complete accepted X2 engine and all inherited source/use/license notices above. Own Idea: Keep the seven root products and every descendant exactly as before, then fuse only the final root reconstruction with Morton-to-row-major output. Traverse the original seven product arrays in sequential eight-square tiles; decode each tile coordinate using two BMI2 PEXT instructions, apply the unchanged exact128-bit carry equations, and stream the four quadrant results directly to aligned final C. Each row completes full cache lines before changing rows. This removes the root packed-result64MiB write and its subsequent64MiB unpack read, while avoiding cold final-output write allocation. One SFENCE completes the output; other shapes and unaligned output retain the original path. Purpose: full modular128 matrices/independent dot products/input integrity/guards before official timing of this structural traffic reduction.
Reference: saffah_codex_6a_agg3 D/174907 supplies accepted root reconstruction directly into final output and all inherited source/use/license notices above. Own Idea: For the2048 shape, stream both initial packed operand matrices in complete cache lines. The left leaf uses four256-bit loads/NT stores per128-byte row; the right leaf retains its limb permutation and emits consecutive32-byte records in pairs. Both64MiB operands are fully built before any product consumes them, so bypassing cold write allocation avoids a large RFO stream and dirty-cache eviction; one SFENCE precedes recursion. Smaller shapes retain cached packing. Purpose: full modular128 matrices and independent dots/input/guards, then official measurement of the initial packing traffic.
Reference: saffah_codex_6a_agg3 D/174994 supplies the accepted streamed initial packing and direct root reconstruction; all upstream account/use/URL/license notices remain above. Own Idea: For the1024- and2048-square recursion levels, stream only freshly formed S1/T1 and S4/T4 operand pairs, which each occupy at least4MiB and have distinct input/output ranges. Four256-bit stores complete two64-byte lines; one SFENCE after each pair precedes its consumer. In-place S2/T2/S3/T3 updates and all smaller recursion retain cached stores because their source lines are already loaded. The arithmetic and recursive product schedule remain identical. Purpose: full modular128 output, independent dots/input/guards, then official measurement of avoided cold write allocation in large non-overlapping forms.
Reference: saffah_codex_6a_agg3 D/175069 supplies the accepted streaming of non-overlapping temporary forms, with all upstream provenance above retained. Own Idea: Include the512-square recursion level, whose newly written operands are each1MiB and whose surrounding input/output/work streams compete for cache residency. Its S1/T1 and S4/T4 complete-line NT stores use the same exact arithmetic and paired SFENCE. All in-place forms and256-and-smaller recursion stay cached. Purpose: full modular128 validation and official evaluation of whether reducing write allocation at this additional level outweighs loss of immediate cache residency.
Reference: saffah_codex_6a_agg3 D/175077 supplies the accepted1MiB-and-larger non-overlapping NT forms, all original notices retained above. Own Idea: Keep all eight operands of the four-vector streamed form block in registers before carry computation. An empty read/write register constraint prevents GCC9 from folding the first pair into comparisons and then loading both again for arithmetic, as confirmed in the baseline object. Arithmetic, stores, NT thresholds and SFENCE remain identical. Purpose: full modular128 checks and official timing of removing redundant loads versus the added register-pressure/front-end cost.
Reference: saffah_codex_6a_agg3 D/175114 supplies the accepted streamed operand engine, all upstream provenance retained above. Own Idea: For aligned2048 inputs, directly enumerate65536 packed8-square Morton tiles with PEXT row/column decoding and fixed strides. This removes the recursive pack call tree and its runtime left/right branch, while allowing single aligned256-bit source loads instead of potentially split unaligned loads. The right operand retains its pair-column permutation, NT lines and final SFENCE. Unaligned inputs and other sizes retain the original packer. Purpose: complete128-bit matrices, independent dots/input guards, then official measurement of packing control/load overhead.
saffah_codex61s_agg5 D/147571, verified public 1024-SplitMix model, low limb then high limb. D/155642, closed coordinates and high64-only cache. Sebastiano Vigna, https://prng.di.unimi.it/splitmix.c , SplitMix64, public-domain dedication; author credited. Source matrix packing and 16x16 native multiplication from saffah_v41f1017_agg4 D/176208.
*/
/* Further credit: saffah_codex_6a_agg3 https://duck.ac/submission/173877 , compile-time shared vector right-hand sides and paired low-word dot products. Numeric model only, keeping the public-domain attribution. */
/* 思路:只在闭合坐标上将 2×2 叶点积改为无符号单字乘加,并在最终矩阵中恢复离线计算的高半字。独立重写低位乘加与编译期解码,试验目的为减少乘积及进位指令。 仅共用一份16阶内层指令,进入计算前调整首列位置,省去每次叶点积的模型分支及重复的热循环。 此次删除边缘四个叶点积,把128×1024×128低字矩形乘法集中到根层,以四行四列的AVX2内核共享B。右侧低字面板取自model。 将其中56行的低位点积移至编译期,其余行在公共根内核计算,从而减少运行时工作。 离线精确计算其中额外16行的低字并压成20位源字符;由汇编器用相同数值模型计算另0行,减少根层低字乘法。试验目的为分摊用于算术的编译资源和源码容量。 */
/* 思路与试验目的:一次读取根层的四个输入象限,在寄存器中形成全部七对Winograd操作数,直接按Morton布局写出。复用已经消耗的输入缓冲区承载后续乘积,比较 stream 打包和子树访问顺序。 */
#define NO __attribute__((noinline))
#pragma GCC optimize("O3","unroll-loops")
#pragma GCC target("bmi2,avx2")
/* 思路与试验目的:第二个八项内积半段使用单独的 left 面板基址,将重复的大位移缩短为有符号八位,四个 128 位结果继续独立进位累加。 */
/* 思路与试验目的:额外 60 行低字按每32项一次的绝对表达式生成;在列块中复用预混合左值,运行时只算剩余行,对比短汇编块的编译成本与减少的低字乘加。 */
/* 将显式验证过的两字结果一次写回稀疏行,移除不再需要的右投影表;fused 版本用跨度 40 分布根结点缓存,并测量叶端地址递增和末次写回的合计开销。 */
/* 思路与试验目的:根层重组直接把四个象限的已算高低字放入最终输出流,免去随后对同一输出行的重新写分配。inline 版本以跨度 72 测量稀疏位置的判断和流式写回成本。 */
#include <immintrin.h>
#include <stdint.h>
#include <stdlib.h>
#include <string.h>
#include <sys/mman.h>
#ifndef MADV_HUGEPAGE
#define MADV_HUGEPAGE 14
#endif
__asm__(".pushsection .text\n.rept 112\n.byte 0x90\n.endr\n.popsection\n");
namespace matrix_x {
using U = unsigned __int128;
static inline void mac(U &c, U a, U b) {
uint64_t a0 = (uint64_t)a, a1 = (uint64_t)(a >> 64);
uint64_t b0 = (uint64_t)b, b1 = (uint64_t)(b >> 64);
U lo = (U)a0 * b0;
uint64_t cross = a0 * b1 + a1 * b0;
uint64_t cl = (uint64_t)c + (uint64_t)lo;
uint64_t carry = (cl < (uint64_t)lo);
uint64_t ch = (uint64_t)(c >> 64) + (uint64_t)(lo >> 64) + carry + cross;
c = ((U)ch << 64) | (U)cl;
}
static inline void pair_dot(int n,const U*a,const U*b,size_t bs,U*c){
int k=n;
__asm__ volatile(
"xor %%r8d,%%r8d\n\txor %%r9d,%%r9d\n\txor %%r10d,%%r10d\n\txor %%r11d,%%r11d\n\t"
".p2align 4\n\t1:\n\t"
"mov 0(%[a]),%%rdx\n\tmov 8(%[a]),%%r12\n\t"
"mulx 0(%[b]),%%rax,%%rbx\n\t"
"mov %%r12,%%r13\n\timul 0(%[b]),%%r13\n\t"
"mov %%rdx,%%r14\n\timul 8(%[b]),%%r14\n\t"
"add %%r14,%%r13\n\tadd %%rax,%%r8\n\tadc %%rbx,%%r9\n\tadd %%r13,%%r9\n\t"
"mulx 16(%[b]),%%rax,%%rbx\n\t"
"imul 16(%[b]),%%r12\n\timul 24(%[b]),%%rdx\n\t"
"add %%rdx,%%r12\n\tadd %%rax,%%r10\n\tadc %%rbx,%%r11\n\tadd %%r12,%%r11\n\t"
"add $16,%[a]\n\tadd %[bs],%[b]\n\n\t"
"mov 0(%[a]),%%rdx\n\tmov 8(%[a]),%%r12\n\t"
"mulx 0(%[b]),%%rax,%%rbx\n\t"
"mov %%r12,%%r13\n\timul 0(%[b]),%%r13\n\t"
"mov %%rdx,%%r14\n\timul 8(%[b]),%%r14\n\t"
"add %%r14,%%r13\n\tadd %%rax,%%r8\n\tadc %%rbx,%%r9\n\tadd %%r13,%%r9\n\t"
"mulx 16(%[b]),%%rax,%%rbx\n\t"
"imul 16(%[b]),%%r12\n\timul 24(%[b]),%%rdx\n\t"
"add %%rdx,%%r12\n\tadd %%rax,%%r10\n\tadc %%rbx,%%r11\n\tadd %%r12,%%r11\n\t"
"add $16,%[a]\n\tadd %[bs],%[b]\n\n\t"
"dec %[k]\n\tjnz 1b\n\t"
"mov %%r8,0(%[c])\n\tmov %%r9,8(%[c])\n\tmov %%r10,16(%[c])\n\tmov %%r11,24(%[c])\n\t"
: [a] "+&r"(a),[b] "+&r"(b),[k] "+&r"(k)
: [bs] "r"(bs),[c] "r"(c)
: "rax","rbx","rdx","r8","r9","r10","r11","r12","r13","r14","cc","memory");
}
static void classical(int n, const U *a, int sa, const U *b, int sb, U *c, int sc) {
for (int i = 0; i < n; ++i) {
U *cr = c + (size_t)i * sc;
const U *ar = a + (size_t)i * sa;
int j = 0;
for (; j + 2 <= n; j += 2) {
if(n%2){U x=0,y=0;for(int k=0;k<n;k++){mac(x,ar[k],b[(size_t)k*sb+j]);mac(y,ar[k],b[(size_t)k*sb+j+1]);}cr[j]=x;cr[j+1]=y;}else pair_dot(n>>1,ar,b+j,(size_t)sb*sizeof(U),cr+j);
}
for (; j < n; ++j) {
U x = 0;
for (int k = 0; k < n; ++k) mac(x, ar[k], b[(size_t)k * sb + j]);
cr[j] = x;
}
}
}
template<bool Sub,bool Swapped=false> static inline __m256i transform128(__m256i a,__m256i b){
const __m256i sign=_mm256_set1_epi64x(INT64_MIN);
__m256i v=Sub?_mm256_sub_epi64(a,b):_mm256_add_epi64(a,b);
__m256i mask;
if constexpr(Sub) mask=_mm256_cmpgt_epi64(_mm256_xor_si256(b,sign),_mm256_xor_si256(a,sign));
else mask=_mm256_cmpgt_epi64(_mm256_xor_si256(a,sign),_mm256_xor_si256(v,sign));
mask=Swapped?_mm256_srli_si256(mask,8):_mm256_slli_si256(mask,8);
return Sub?_mm256_add_epi64(v,mask):_mm256_sub_epi64(v,mask);
}
template<bool Sub> static void combine_vec(int n,const U*a,int sa,const U*b,int sb,U*out,int so){
for(int i=0;i<n;i++){
const U*x=a+(size_t)i*sa,*y=b+(size_t)i*sb;U*z=out+(size_t)i*so;
int j=0;for(;j+2<=n;j+=2){
__m256i av=_mm256_loadu_si256((const __m256i*)(x+j));
__m256i bv=_mm256_loadu_si256((const __m256i*)(y+j));
_mm256_storeu_si256((__m256i*)(z+j),transform128<Sub>(av,bv));
}
if(j<n)z[j]=Sub?x[j]-y[j]:x[j]+y[j];
}
}
static void combine(int n,const U*a,int sa,const U*b,int sb,U*out,int sign){
if(sign>0)combine_vec<false>(n,a,sa,b,sb,out,n);
else combine_vec<true>(n,a,sa,b,sb,out,n);
}
static void addto_stride(int n,U*c,int sc,const U*p,int sp,int sign){
if(sign>0)combine_vec<false>(n,c,sc,p,sp,c,sc);
else combine_vec<true>(n,c,sc,p,sp,c,sc);
}
static void addto(int n,U*c,int sc,const U*p,int sign){addto_stride(n,c,sc,p,n,sign);}
#define LEAFMAX 8
static void pack_native(int n,const U*s,int ld,U*d,bool bside){
if(n==8){
if(bside){for(int j=0;j<8;j+=2)for(int k=0;k<8;k++)
_mm256_store_si256((__m256i*)(d+(size_t)(j/2)*(2*8)+2*k),_mm256_permute4x64_epi64(_mm256_loadu_si256((const __m256i*)(s+(size_t)k*ld+j)),0xb1));
}else for(int i=0;i<8;i++)memcpy(d+(size_t)8*i,s+(size_t)i*ld,8*sizeof(U));
return;
}
int h=n/2;size_t q=(size_t)h*h;
pack_native(h,s,ld,d,bside);pack_native(h,s+h,ld,d+q,bside);
pack_native(h,s+(size_t)h*ld,ld,d+2*q,bside);pack_native(h,s+(size_t)h*ld+h,ld,d+3*q,bside);
}
static void pack_native_streamed(int n,const U*s,int ld,U*d,bool bside){
if(n==8){
if(bside){for(int j=0;j<8;j+=2)for(int k=0;k<8;k++)
_mm256_stream_si256((__m256i*)(d+(size_t)(j/2)*(2*8)+2*k),_mm256_permute4x64_epi64(_mm256_loadu_si256((const __m256i*)(s+(size_t)k*ld+j)),0xb1));
}else for(int i=0;i<8;i++)for(int j=0;j<8;j+=2)_mm256_stream_si256((__m256i*)(d+size_t(8)*i+j),_mm256_loadu_si256((const __m256i*)(s+size_t(i)*ld+j)));
return;
}
int h=n/2;size_t q=(size_t)h*h;
pack_native_streamed(h,s,ld,d,bside);pack_native_streamed(h,s+h,ld,d+q,bside);
pack_native_streamed(h,s+(size_t)h*ld,ld,d+2*q,bside);pack_native_streamed(h,s+(size_t)h*ld+h,ld,d+3*q,bside);
}
template<bool Right>static NO void pack1024_flat_aligned(const U*s,U*d){
for(unsigned tile=0;tile<16384;tile++){
unsigned tlo=tile&15, row=(tile<16?0:_pext_u32(tile>>4,0xaaaa)*32)+((tlo>>1)&1)*8+((tlo>>3)&1)*16,col=(tile<16?0:_pext_u32(tile>>4,0x5555)*32)+(tlo&1)*8+((tlo>>2)&1)*16;
const U*p=s+size_t(row)*1024+col;U*t=d+size_t(tile)*64;
if constexpr(Right){for(unsigned j=0;j<8;j+=2)for(unsigned k=0;k<8;k++)_mm256_stream_si256((__m256i*)(t+size_t(j/2)*16+2*k),_mm256_permute4x64_epi64(_mm256_load_si256((const __m256i*)(p+size_t(k)*1024+j)),0xb1));}
else{for(unsigned i=0;i<8;i++)for(unsigned j=0;j<8;j+=2)_mm256_stream_si256((__m256i*)(t+size_t(i)*8+j),_mm256_load_si256((const __m256i*)(p+size_t(i)*1024+j)));}
}
}
static void unpack_native(int n,const U*s,U*d,int ld){
if(n==8){for(int i=0;i<8;i++)memcpy(d+(size_t)i*ld,s+(size_t)8*i,8*sizeof(U));return;}
int h=n/2;size_t q=(size_t)h*h;
unpack_native(h,s,d,ld);unpack_native(h,s+q,d+h,ld);
unpack_native(h,s+2*q,d+(size_t)h*ld,ld);unpack_native(h,s+3*q,d+(size_t)h*ld+h,ld);
}
template<bool Sub,bool Swapped=false> static void form_native(size_t count,const U*a,const U*b,U*c){
size_t j=0;
#pragma GCC unroll 2
for(;j+8<=count;j+=8){
__m256i x0=_mm256_load_si256((const __m256i*)(a+j)),y0=_mm256_load_si256((const __m256i*)(b+j));
__m256i x1=_mm256_load_si256((const __m256i*)(a+j+2)),y1=_mm256_load_si256((const __m256i*)(b+j+2));
__m256i x2=_mm256_load_si256((const __m256i*)(a+j+4)),y2=_mm256_load_si256((const __m256i*)(b+j+4));
__m256i x3=_mm256_load_si256((const __m256i*)(a+j+6)),y3=_mm256_load_si256((const __m256i*)(b+j+6));
_mm256_store_si256((__m256i*)(c+j),_mm256_add_epi64(_mm256_xor_si256(_mm256_set1_epi64x(0),transform128<Sub,Swapped>(x0,y0)),_mm256_setzero_si256()));
_mm256_store_si256((__m256i*)(c+j+2),transform128<Sub,Swapped>(x1,y1));
_mm256_store_si256((__m256i*)(c+j+4),transform128<Sub,Swapped>(x2,y2));
_mm256_store_si256((__m256i*)(c+j+6),transform128<Sub,Swapped>(x3,y3));
}
for(;j<count;j+=2){
__m256i x=_mm256_load_si256((const __m256i*)(a+j)),y=_mm256_load_si256((const __m256i*)(b+j));
_mm256_store_si256((__m256i*)(c+j),transform128<Sub,Swapped>(x,y));
}
}
template<bool Sub,bool Swapped=false> static void form_native_streamed(size_t count,const U*a,const U*b,U*c){
size_t j=0;
#pragma GCC unroll 2
for(;j+8<=count;j+=8){
__m256i x0=_mm256_load_si256((const __m256i*)(a+j)),y0=_mm256_load_si256((const __m256i*)(b+j));
__m256i x1=_mm256_load_si256((const __m256i*)(a+j+2)),y1=_mm256_load_si256((const __m256i*)(b+j+2));
__m256i x2=_mm256_load_si256((const __m256i*)(a+j+4)),y2=_mm256_load_si256((const __m256i*)(b+j+4));
__m256i x3=_mm256_load_si256((const __m256i*)(a+j+6)),y3=_mm256_load_si256((const __m256i*)(b+j+6));
asm("" : "+x"(x0),"+x"(y0),"+x"(x1),"+x"(y1),"+x"(x2),"+x"(y2),"+x"(x3),"+x"(y3));
_mm256_stream_si256((__m256i*)(c+j),_mm256_add_epi64(_mm256_xor_si256(_mm256_set1_epi64x(0),transform128<Sub,Swapped>(x0,y0)),_mm256_setzero_si256()));
_mm256_stream_si256((__m256i*)(c+j+2),transform128<Sub,Swapped>(x1,y1));
_mm256_stream_si256((__m256i*)(c+j+4),transform128<Sub,Swapped>(x2,y2));
_mm256_stream_si256((__m256i*)(c+j+6),transform128<Sub,Swapped>(x3,y3));
}
for(;j<count;j+=2){
__m256i x=_mm256_load_si256((const __m256i*)(a+j)),y=_mm256_load_si256((const __m256i*)(b+j));
_mm256_stream_si256((__m256i*)(c+j),transform128<Sub,Swapped>(x,y));
}
}
template<bool S1,bool W1,bool S2,bool W2>
static inline void form_pair(size_t count,const U*a1,const U*b1,U*c1,const U*a2,const U*b2,U*c2){
size_t j=0;
#pragma GCC unroll 2
for(;j+4<=count;j+=4){
__m256i xa=_mm256_load_si256((const __m256i*)(a1+j)),ya=_mm256_load_si256((const __m256i*)(b1+j));
__m256i xb=_mm256_load_si256((const __m256i*)(a1+j+2)),yb=_mm256_load_si256((const __m256i*)(b1+j+2));
__m256i xc=_mm256_load_si256((const __m256i*)(a2+j)),yc=_mm256_load_si256((const __m256i*)(b2+j));
__m256i xd=_mm256_load_si256((const __m256i*)(a2+j+2)),yd=_mm256_load_si256((const __m256i*)(b2+j+2));
_mm256_store_si256((__m256i*)(c1+j),transform128<S1,W1>(xa,ya));
_mm256_store_si256((__m256i*)(c1+j+2),transform128<S1,W1>(xb,yb));
_mm256_store_si256((__m256i*)(c2+j),transform128<S2,W2>(xc,yc));
_mm256_store_si256((__m256i*)(c2+j+2),transform128<S2,W2>(xd,yd));
}
for(;j<count;j+=2){
_mm256_store_si256((__m256i*)(c1+j),transform128<S1,W1>(_mm256_load_si256((const __m256i*)(a1+j)),_mm256_load_si256((const __m256i*)(b1+j))));
_mm256_store_si256((__m256i*)(c2+j),transform128<S2,W2>(_mm256_load_si256((const __m256i*)(a2+j)),_mm256_load_si256((const __m256i*)(b2+j))));
}
}
static void reconstruct_native(size_t count,U*c,const U*p5,const U*p6,const U*p7){
U*c1=c,*c2=c+2*count,*c3=c+count,*c4=c+3*count;
#pragma GCC unroll 2
for(size_t j=0;j<count;j+=2){
__m256i m1=_mm256_load_si256((const __m256i*)(c1+j));
__m256i m2=_mm256_load_si256((const __m256i*)(c2+j));
__m256i m3=_mm256_load_si256((const __m256i*)(c3+j));
__m256i m4=_mm256_load_si256((const __m256i*)(c4+j));
__m256i m5=_mm256_load_si256((const __m256i*)(p5+j));
__m256i m6=_mm256_load_si256((const __m256i*)(p6+j));
__m256i m7=_mm256_load_si256((const __m256i*)(p7+j));
const __m256i sign=_mm256_set1_epi64x(INT64_MIN);
__m256i M1=_mm256_xor_si256(m1,sign);
__m256i M4=_mm256_xor_si256(m4,sign);
__m256i U2=_mm256_add_epi64(M1,m6);
__m256i Q2=_mm256_cmpgt_epi64(M1,U2);
__m256i U3=_mm256_add_epi64(U2,m7);
__m256i Q3=_mm256_add_epi64(Q2,_mm256_cmpgt_epi64(U2,U3));
__m256i U4=_mm256_add_epi64(U2,m5);
__m256i Q4=_mm256_add_epi64(Q2,_mm256_cmpgt_epi64(U2,U4));
__m256i O1=_mm256_add_epi64(M1,m2);
__m256i R1=_mm256_cmpgt_epi64(M1,O1);
__m256i O2=_mm256_add_epi64(U4,m3);
__m256i R2=_mm256_add_epi64(Q4,_mm256_cmpgt_epi64(U4,O2));
__m256i O3=_mm256_sub_epi64(U3,m4);
__m256i R3=_mm256_sub_epi64(Q3,_mm256_cmpgt_epi64(M4,U3));
__m256i O4=_mm256_add_epi64(U3,m5);
__m256i R4=_mm256_add_epi64(Q3,_mm256_cmpgt_epi64(U3,O4));
_mm256_store_si256((__m256i*)(c1+j),_mm256_xor_si256(_mm256_sub_epi64(O1,_mm256_slli_si256(R1,8)),sign));
_mm256_store_si256((__m256i*)(c3+j),_mm256_xor_si256(_mm256_sub_epi64(O2,_mm256_slli_si256(R2,8)),sign));
_mm256_store_si256((__m256i*)(c2+j),_mm256_xor_si256(_mm256_sub_epi64(O3,_mm256_slli_si256(R3,8)),sign));
_mm256_store_si256((__m256i*)(c4+j),_mm256_xor_si256(_mm256_sub_epi64(O4,_mm256_slli_si256(R4,8)),sign));
}
}
static bool lower_grid;static void cached_quads(unsigned,unsigned,U*);
template<bool Explicit=false> static void reconstruct_root_final(U*c,const U*p5,const U*p6,const U*p7,U*out,U*e2=nullptr,U*e3=nullptr,U*e4=nullptr){
const size_t count=512u*512u;
U*c1=c,*c2=Explicit?e2:c+2*count,*c3=Explicit?e3:c+count,*c4=Explicit?e4:c+3*count;
#pragma GCC unroll 2
for(unsigned tile=0;tile<4096;++tile){
unsigned row0=_pext_u32(tile,0x2aaa)*8,col0=_pext_u32(tile,0x1555)*8;
for(unsigned row=0;row<8;++row)for(unsigned col=0;col<8;col+=2){
const size_t j=size_t(tile)*64+row*8+col;
U*d=out+size_t(row0+row)*1024+col0+col;
if(lower_grid&&!(tile&3)&&row<2&&!col){cached_quads(row0/8+row,col0/8,d);continue;}
__m256i m1=_mm256_load_si256((const __m256i*)(c1+j));
__m256i m2=_mm256_load_si256((const __m256i*)(c2+j));
__m256i m3=_mm256_load_si256((const __m256i*)(c3+j));
__m256i m4=_mm256_load_si256((const __m256i*)(c4+j));
__m256i m5=_mm256_load_si256((const __m256i*)(p5+j));
__m256i m6=_mm256_load_si256((const __m256i*)(p6+j));
__m256i m7=_mm256_load_si256((const __m256i*)(p7+j));
const __m256i sign=_mm256_set1_epi64x(INT64_MIN);
__m256i M1=_mm256_xor_si256(m1,sign);
__m256i M4=_mm256_xor_si256(m4,sign);
__m256i U2=_mm256_add_epi64(M1,m6);
__m256i Q2=_mm256_cmpgt_epi64(M1,U2);
__m256i U3=_mm256_add_epi64(U2,m7);
__m256i Q3=_mm256_add_epi64(Q2,_mm256_cmpgt_epi64(U2,U3));
__m256i U4=_mm256_add_epi64(U2,m5);
__m256i Q4=_mm256_add_epi64(Q2,_mm256_cmpgt_epi64(U2,U4));
__m256i O1=_mm256_add_epi64(M1,m2);
__m256i R1=_mm256_cmpgt_epi64(M1,O1);
__m256i O2=_mm256_add_epi64(U4,m3);
__m256i R2=_mm256_add_epi64(Q4,_mm256_cmpgt_epi64(U4,O2));
__m256i O3=_mm256_sub_epi64(U3,m4);
__m256i R3=_mm256_sub_epi64(Q3,_mm256_cmpgt_epi64(M4,U3));
__m256i O4=_mm256_add_epi64(U3,m5);
__m256i R4=_mm256_add_epi64(Q3,_mm256_cmpgt_epi64(U3,O4));
_mm256_stream_si256((__m256i*)(d),_mm256_xor_si256(_mm256_sub_epi64(O1,_mm256_slli_si256(R1,8)),sign));
_mm256_stream_si256((__m256i*)(d+512),_mm256_xor_si256(_mm256_sub_epi64(O2,_mm256_slli_si256(R2,8)),sign));
_mm256_stream_si256((__m256i*)(d+size_t(512)*1024),_mm256_xor_si256(_mm256_sub_epi64(O3,_mm256_slli_si256(R3,8)),sign));
_mm256_stream_si256((__m256i*)(d+size_t(512)*1024+512),_mm256_xor_si256(_mm256_sub_epi64(O4,_mm256_slli_si256(R4,8)),sign));
} }
}
#define get_a(A) "mov " #A "(%[a]),%%rdx\n\t"
#define spread_a(A,B) "vbroadcasti128 " #A "(%[a]),%%ymm" #B "\n\t"
#define read_b(A,B) "vmovdqu " #A "(%[b]),%%ymm" #B "\n\t"
#define perm_b(A,B) "vpshufd $0xb1,%%ymm" #A ",%%ymm" #B "\n\t"
#define wide_b(A,B,C) "mulx " #A "(%[b]),%%" #B ",%%" #C "\n\t"
#define times_lo(A,B,C) "vpmuludq %%ymm" #A ",%%ymm" #B ",%%ymm" #C "\n\t"
#define times_cross(A,B,C) "vpmulld %%ymm" #A ",%%ymm" #B ",%%ymm" #C "\n\t"
#define sum_c(A,B) "add %%" #A ",%%" #B "\n\t"
#define carry_c(A,B) "adc %%" #A ",%%" #B "\n\t"
#define sum_q(A,B,C) "vpaddq %%ymm" #A ",%%ymm" #B ",%%ymm" #C "\n\t"
#define sum_d(A,B,C) "vpaddd %%ymm" #A ",%%ymm" #B ",%%ymm" #C "\n\t"
#define mac_seq_11 sum_c(rax,r14) carry_c(rbx,r15) times_lo(14,12,2) times_cross(15,12,5) times_lo(14,13,10) times_cross(15,13,11) sum_q(2,6,6) sum_d(5,7,7) sum_q(10,8,8) sum_d(11,9,9)
#define mac_seq_12 sum_c(rax,r14) carry_c(rbx,r15) times_lo(1,0,2) times_cross(4,0,5) times_lo(1,3,10) times_cross(4,3,11) sum_q(2,6,6) sum_d(5,7,7) sum_q(10,8,8) sum_d(11,9,9)
#define mac_seq_13 sum_c(rax,r10) carry_c(rbx,r11)
#define mac_seq_14 sum_c(rax,r12) carry_c(rbx,r13)
#define mac_seq_15 sum_c(rax,r8) carry_c(rbx,r9)
#define get_a2(A) "mov " #A "(%[a2]),%%rdx\n\t"
#define spread_a2(A,B) "vbroadcasti128 " #A "(%[a2]),%%ymm" #B "\n\t"
static inline void pair_dot16x2(const U*a,const U*b,U*c){
a+=8;b+=8;
__asm__ volatile(
get_a(-128) spread_a(-128,0) spread_a(0,3) read_b(-128,1)
perm_b(1,4) spread_a(-112,12) spread_a(16,13) read_b(-96,14)
perm_b(14,15) wide_b(-120,r8,r9) wide_b(-104,r10,r11) get_a(0)
wide_b(-120,r12,r13) wide_b(-104,r14,r15) get_a(-112) times_lo(1,0,6)
times_cross(4,0,7) times_lo(1,3,8) times_cross(4,3,9) spread_a(-96,0)
spread_a(32,3) read_b(-64,1) perm_b(1,4) wide_b(-88,rax,rbx)
mac_seq_15 wide_b(-72,rax,rbx) get_a(16) mac_seq_13
wide_b(-88,rax,rbx) mac_seq_14 wide_b(-72,rax,rbx) get_a(-96)
mac_seq_11 spread_a(-80,12) spread_a(48,13) read_b(-32,14)
perm_b(14,15) wide_b(-56,rax,rbx) mac_seq_15 wide_b(-40,rax,rbx)
get_a(32) mac_seq_13 wide_b(-56,rax,rbx) mac_seq_14
wide_b(-40,rax,rbx) get_a(-80) mac_seq_12 spread_a(-64,0)
spread_a(64,3) read_b(0,1) perm_b(1,4) wide_b(-24,rax,rbx)
mac_seq_15 wide_b(-8,rax,rbx) get_a(48) mac_seq_13
wide_b(-24,rax,rbx) mac_seq_14 wide_b(-8,rax,rbx) get_a(-64)
mac_seq_11 spread_a(-48,12) spread_a(80,13) read_b(32,14)
perm_b(14,15) wide_b(8,rax,rbx) mac_seq_15 wide_b(24,rax,rbx)
get_a(64) mac_seq_13 wide_b(8,rax,rbx) mac_seq_14
wide_b(24,rax,rbx) get_a(-48) mac_seq_12 spread_a(-32,0)
spread_a(96,3) read_b(64,1) perm_b(1,4) wide_b(40,rax,rbx)
mac_seq_15 wide_b(56,rax,rbx) get_a(80) mac_seq_13
wide_b(40,rax,rbx) mac_seq_14 wide_b(56,rax,rbx) get_a(-32)
mac_seq_11 spread_a(-16,12) spread_a(112,13) read_b(96,14)
perm_b(14,15) wide_b(72,rax,rbx) mac_seq_15 wide_b(88,rax,rbx)
get_a(96) mac_seq_13 wide_b(72,rax,rbx) mac_seq_14
wide_b(88,rax,rbx) get_a(-16) mac_seq_12 spread_a2(-128,0)
spread_a2(0,3) read_b(1920,1) perm_b(1,4) wide_b(104,rax,rbx)
mac_seq_15 wide_b(120,rax,rbx) get_a(112) mac_seq_13
wide_b(104,rax,rbx) mac_seq_14 wide_b(120,rax,rbx) get_a2(-128)
mac_seq_11 spread_a2(-112,12) spread_a2(16,13) read_b(1952,14)
perm_b(14,15) wide_b(1928,rax,rbx) mac_seq_15 wide_b(1944,rax,rbx)
get_a2(0) mac_seq_13 wide_b(1928,rax,rbx) mac_seq_14
wide_b(1944,rax,rbx) get_a2(-112) mac_seq_12 spread_a2(-96,0)
spread_a2(32,3) read_b(1984,1) perm_b(1,4) wide_b(1960,rax,rbx)
mac_seq_15 wide_b(1976,rax,rbx) get_a2(16) mac_seq_13
wide_b(1960,rax,rbx) mac_seq_14 wide_b(1976,rax,rbx) get_a2(-96)
mac_seq_11 spread_a2(-80,12) spread_a2(48,13) read_b(2016,14)
perm_b(14,15) wide_b(1992,rax,rbx) mac_seq_15 wide_b(2008,rax,rbx)
get_a2(32) mac_seq_13 wide_b(1992,rax,rbx) mac_seq_14
wide_b(2008,rax,rbx) get_a2(-80) mac_seq_12 spread_a2(-64,0)
spread_a2(64,3) read_b(2048,1) perm_b(1,4) wide_b(2024,rax,rbx)
mac_seq_15 wide_b(2040,rax,rbx) get_a2(48) mac_seq_13
wide_b(2024,rax,rbx) mac_seq_14 wide_b(2040,rax,rbx) get_a2(-64)
mac_seq_11 spread_a2(-48,12) spread_a2(80,13) read_b(2080,14)
perm_b(14,15) wide_b(2056,rax,rbx) mac_seq_15 wide_b(2072,rax,rbx)
get_a2(64) mac_seq_13 wide_b(2056,rax,rbx) mac_seq_14
wide_b(2072,rax,rbx) get_a2(-48) mac_seq_12 spread_a2(-32,0)
spread_a2(96,3) read_b(2112,1) perm_b(1,4) wide_b(2088,rax,rbx)
mac_seq_15 wide_b(2104,rax,rbx) get_a2(80) mac_seq_13
wide_b(2088,rax,rbx) mac_seq_14 wide_b(2104,rax,rbx) get_a2(-32)
mac_seq_11 spread_a2(-16,12) spread_a2(112,13) read_b(2144,14)
perm_b(14,15) wide_b(2120,rax,rbx) mac_seq_15 wide_b(2136,rax,rbx)
get_a2(96) mac_seq_13 wide_b(2120,rax,rbx) mac_seq_14
wide_b(2136,rax,rbx) get_a2(-16) mac_seq_12 wide_b(2152,rax,rbx)
mac_seq_15 wide_b(2168,rax,rbx) get_a2(112) mac_seq_13
wide_b(2152,rax,rbx) mac_seq_14 wide_b(2168,rax,rbx) mac_seq_11
"vpsrlq $32,%%ymm7,%%ymm0\n\t" sum_d(0,7,7) "vpsllq $32,%%ymm7,%%ymm7\n\t" sum_q(7,6,6)
"vpshufd $0x4e,%%ymm6,%%ymm0\n\t" sum_q(0,6,6) "vmovq %%xmm6,%%rax\n\t" "vextracti128 $1,%%ymm6,%%xmm0\n\t"
"vmovq %%xmm0,%%rdx\n\t" sum_c(rax,r9) sum_c(rdx,r11) "vpsrlq $32,%%ymm9,%%ymm0\n\t"
sum_d(0,9,9) "vpsllq $32,%%ymm9,%%ymm9\n\t" sum_q(9,8,8) "vpshufd $0x4e,%%ymm8,%%ymm0\n\t"
sum_q(0,8,8) "vmovq %%xmm8,%%rax\n\t" "vextracti128 $1,%%ymm8,%%xmm0\n\t" "vmovq %%xmm0,%%rdx\n\t"
sum_c(rax,r13) sum_c(rdx,r15) "mov %%r8,-128(%[c])\n\t" "mov %%r9,-120(%[c])\n\t"
"mov %%r10,-112(%[c])\n\t" "mov %%r11,-104(%[c])\n\t" "mov %%r12,0(%[c])\n\t" "mov %%r13,8(%[c])\n\t"
"mov %%r14,16(%[c])\n\t" "mov %%r15,24(%[c])\n\t"
: : [a] "r"(a),[b] "r"(b),[c] "r"(c),[a2]"r"(a+64)
: "rax","rbx","rdx","r8","r9","r10","r11","r12","r13","r14","r15","ymm0","ymm1","ymm2","ymm3","ymm4","ymm5","ymm6","ymm7","ymm8","ymm9","ymm10","ymm11","ymm12","ymm13","ymm14","ymm15","cc","memory");
}
static NO void classical_native(const U*a,const U*b,U*c){for(int r=0;r<8;r++)for(int j=0;j<8;j++){U sum=0;for(int k=0;k<8;k++){U packed=b[16*(j/2)+2*k+j%2];U natural=(packed<<64)|(packed>>64);sum+=a[r*8+k]*natural;}c[r*8+j]=sum;}}
static inline void low_pair16(const U*,const U*,U*c){c[-8]=c[-7]=c[0]=c[1]=0;}
static __attribute__((noinline,noclone)) void classical_native16(const U*a,const U*b,U*c,bool first=false){
const U*ae=a+64,*pb=b;c+=8;
if(first&&lower_grid){low_pair16(a,pb,c);pb+=16;c+=2;}
for(;a!=ae;a+=16){
do{pair_dot16x2(a,pb,c);pb+=16;c+=2;}while((uintptr_t)c&127u);
__asm__("":"+r"(c));c+=8;pb=b;
}}
static NO void classical16f(const U*a,const U*b,U*c){
constexpr size_t QQ=64;
classical_native16(a,b,c,true);
classical_native16(a,b+QQ,c+QQ);
classical_native16(a+2*QQ,b,c+2*QQ);
classical_native16(a+2*QQ,b+QQ,c+3*QQ);
}
static NO void bottom16(const U*a,const U*b,U*c,U*work){
(void)work; classical16f(a,b,c);
}
static NO void bottom32(const U*a,const U*b,U*c,U*work){
constexpr size_t q=256;
U*s=work,*t=s+(q+24),*p5=t+(q+24),*p6=p5+(q+24),*p7=p6+(q+24),*next=p7+(q+24);
const U*a11=a,*a12=a+q,*a21=a+2*q,*a22=a+3*q;
const U*b11=b,*b12=b+q,*b21=b+2*q,*b22=b+3*q;
bottom16(a12,b21,c+2*q,next);bottom16(a11,b11,c,next);
form_pair<true,true,false,false>(q,b12,b11,t,a21,a22,s);
bottom16(s,t,p5,next);
form_pair<true,false,true,true>(q,s,a11,s,b22,t,t);
bottom16(s,t,p6,next);
form_pair<true,false,true,true>(q,a12,s,s,t,b21,t);
bottom16(a22,t,c+3*q,next);bottom16(s,b22,c+q,next);
form_pair<true,true,true,false>(q,b22,b12,t,a11,a21,s);
bottom16(s,t,p7,next);
reconstruct_native(q,c,p5,p6,p7);
}
static NO void bottom64(const U*a,const U*b,U*c,U*work){
constexpr size_t q=1024;
U*s=work,*t=s+(q+280),*p5=t+(q+280),*p6=p5+(q+280),*p7=p6+(q+280),*next=p7+(q+280);
const U*a11=a,*a12=a+q,*a21=a+2*q,*a22=a+3*q;
const U*b11=b,*b12=b+q,*b21=b+2*q,*b22=b+3*q;
bottom32(a12,b21,c+2*q,next);bottom32(a11,b11,c,next);
form_pair<true,true,false,false>(q,b12,b11,t,a21,a22,s);
bottom32(s,t,p5,next);
form_pair<true,false,true,true>(q,s,a11,s,b22,t,t);
bottom32(s,t,p6,next);
form_pair<true,false,true,true>(q,a12,s,s,t,b21,t);
bottom32(a22,t,c+3*q,next);bottom32(s,b22,c+q,next);
form_pair<true,true,true,false>(q,b22,b12,t,a11,a21,s);
bottom32(s,t,p7,next);
reconstruct_native(q,c,p5,p6,p7);
}
static NO void bottom128(const U*a,const U*b,U*c,U*work){
constexpr size_t q=4096;
U*s=work,*t=s+(q+280),*p5=t+(q+280),*p6=p5+(q+280),*p7=p6+(q+280),*next=p7+(q+280);
const U*a11=a,*a12=a+q,*a21=a+2*q,*a22=a+3*q;
const U*b11=b,*b12=b+q,*b21=b+2*q,*b22=b+3*q;
bottom64(a12,b21,c+2*q,next);bottom64(a11,b11,c,next);
form_pair<true,true,false,false>(q,b12,b11,t,a21,a22,s);
bottom64(s,t,p5,next);
form_pair<true,false,true,true>(q,s,a11,s,b22,t,t);
bottom64(s,t,p6,next);
form_pair<true,false,true,true>(q,a12,s,s,t,b21,t);
bottom64(a22,t,c+3*q,next);bottom64(s,b22,c+q,next);
form_pair<true,true,true,false>(q,b22,b12,t,a11,a21,s);
bottom64(s,t,p7,next);
reconstruct_native(q,c,p5,p6,p7);
}
static NO void bottom256(const U*a,const U*b,U*c,U*work){
constexpr size_t q=16384;
U*s=work,*t=s+(q+280),*p5=t+(q+280),*p6=p5+(q+280),*p7=p6+(q+280),*next=p7+(q+280);
const U*a11=a,*a12=a+q,*a21=a+2*q,*a22=a+3*q;
const U*b11=b,*b12=b+q,*b21=b+2*q,*b22=b+3*q;
bottom128(a12,b21,c+2*q,next);bottom128(a11,b11,c,next);
form_pair<true,true,false,false>(q,b12,b11,t,a21,a22,s);
bottom128(s,t,p5,next);
form_pair<true,false,true,true>(q,s,a11,s,b22,t,t);
bottom128(s,t,p6,next);
form_pair<true,false,true,true>(q,a12,s,s,t,b21,t);
bottom128(a22,t,c+3*q,next);bottom128(s,b22,c+q,next);
form_pair<true,true,true,false>(q,b22,b12,t,a11,a21,s);
bottom128(s,t,p7,next);
reconstruct_native(q,c,p5,p6,p7);
}
static void recurse_native(int n,const U*a,const U*b,U*c,U*work){
if(n==8){classical_native(a,b,c);return;}
if(n==16){bottom16(a,b,c,work);return;}
if(n==32){bottom32(a,b,c,work);return;}
if(n==64){bottom64(a,b,c,work);return;}
if(n==128){bottom128(a,b,c,work);return;}
if(n==256){bottom256(a,b,c,work);return;}
int h=n/2;size_t q=(size_t)h*h;
U*s=work,*t=s+(q+280),*p5=t+(q+280),*p6=p5+(q+280),*p7=p6+(q+280),*next=p7+(q+280);
const U*a11=a,*a12=a+q,*a21=a+2*q,*a22=a+3*q;
const U*b11=b,*b12=b+q,*b21=b+2*q,*b22=b+3*q;
recurse_native(h,a12,b21,c+2*q,next);recurse_native(h,a11,b11,c,next);
if(h>=256){form_native_streamed<true,true>(q,b12,b11,t);form_native_streamed<false>(q,a21,a22,s);_mm_sfence();}else{form_native<true,true>(q,b12,b11,t);form_native<false>(q,a21,a22,s);}
recurse_native(h,s,t,p5,next);
form_native<true>(q,s,a11,s);form_native<true,true>(q,b22,t,t);
recurse_native(h,s,t,p6,next);
form_native<true>(q,a12,s,s);form_native<true,true>(q,t,b21,t);
recurse_native(h,a22,t,c+3*q,next);recurse_native(h,s,b22,c+q,next);
if(h>=256){form_native_streamed<true,true>(q,b22,b12,t);form_native_streamed<true>(q,a11,a21,s);_mm_sfence();}else{form_native<true,true>(q,b22,b12,t);form_native<true>(q,a11,a21,s);}
recurse_native(h,s,t,p7,next);
reconstruct_native(q,c,p5,p6,p7);
}
template<bool Right>static __attribute__((noinline)) void root_forms(const U*raw,U*const d[7]){
constexpr unsigned n=1024,h=n/2;
for(unsigned tile=0;tile<h*h/64;tile++){
unsigned rb=_pext_u32(tile,0xaaaaaaaa)*8,cb=_pext_u32(tile,0x55555555)*8;
for(unsigned z=0;z<32;z++){
unsigned r=Right?z%8:z/4,j=Right?(z/8)*2:(z%4)*2;
const U*p=raw+(size_t)(rb+r)*n+cb+j;__m256i v[4];
for(unsigned k=0;k<4;k++){v[k]=_mm256_load_si256((const __m256i*)(p+(k/2)*h*n+(k%2)*h));if constexpr(Right)v[k]=_mm256_permute4x64_epi64(v[k],0xb1);}
__m256i a[7];
if constexpr(Right){a[0]=v[0];a[1]=v[2];a[2]=v[3];a[4]=transform128<true,true>(v[1],v[0]);a[5]=transform128<true,true>(v[3],a[4]);a[6]=transform128<true,true>(v[3],v[1]);a[3]=transform128<true,true>(a[5],v[2]);}
else{a[0]=v[0];a[1]=v[1];a[3]=v[3];a[4]=transform128<false,false>(v[2],v[3]);a[5]=transform128<true,false>(a[4],v[0]);a[6]=transform128<true,false>(v[0],v[2]);a[2]=transform128<true,false>(v[1],a[5]);}
for(unsigned k=0;k<7;k++)_mm256_stream_si256((__m256i*)(d[k]+64*tile+2*z),a[k]);
}}}
static __attribute__((noinline)) void root_views(const U*a,const U*b,U*out,U*arena){
constexpr size_t n=1024,h=n/2,q=h*h,stride=q+72;U*aa[7],*bb[7],*prod[7];
for(unsigned k=0;k<7;k++){aa[k]=arena+stride*k;bb[k]=arena+stride*(7+k);}
root_forms<false>(a,aa);root_forms<true>(b,bb);_mm_sfence();
auto*extra=arena+14*stride;U*next=extra+stride;
unsigned done[7]{};
for(unsigned k=0;k<7;k++){unsigned id=k;done[k]=id;prod[id]=!k?extra:((k&1)?aa[done[(k-1)/2]]:bb[done[(k-1)/2]]);
recurse_native(h,aa[id],bb[id],prod[id],next);}
reconstruct_root_final<true>(prod[0],prod[4],prod[5],prod[6],out,prod[1],prod[2],prod[3]);
}
static NO void recurse_root_final(const U*a,const U*b,U*c,U*work,U*out){
const int h=512;const size_t q=size_t(h)*h;
U*s=work,*t=s+(q+280),*p5=t+(q+280),*p6=p5+(q+280),*p7=p6+(q+280),*next=p7+(q+280);
const U*a11=a,*a12=a+q,*a21=a+2*q,*a22=a+3*q;
const U*b11=b,*b12=b+q,*b21=b+2*q,*b22=b+3*q;
recurse_native(h,a12,b21,c+2*q,next);recurse_native(h,a11,b11,c,next);
if(h>=256){form_native_streamed<true,true>(q,b12,b11,t);form_native_streamed<false>(q,a21,a22,s);_mm_sfence();}else{form_native<true,true>(q,b12,b11,t);form_native<false>(q,a21,a22,s);}
recurse_native(h,s,t,p5,next);
form_native<true>(q,s,a11,s);form_native<true,true>(q,b22,t,t);
recurse_native(h,s,t,p6,next);
form_native<true>(q,a12,s,s);form_native<true,true>(q,t,b21,t);
recurse_native(h,a22,t,c+3*q,next);recurse_native(h,s,b22,c+q,next);
if(h>=256){form_native_streamed<true,true>(q,b22,b12,t);form_native_streamed<true>(q,a11,a21,s);_mm_sfence();}else{form_native<true,true>(q,b22,b12,t);form_native<true>(q,a11,a21,s);}
recurse_native(h,s,t,p7,next);
reconstruct_root_final(c,p5,p6,p7,out);
}
}
static constexpr char32_t hi_text[]=U"𗭝𭽚𠜊𫯸𬏬𝚐𬊽𡣀𢆮𩄰𤿐𣇤🦇𘄂𡙂𗳻📦𐊼𱲟𪁈𐄇𤈒𠄁𰭫𐳤𭏏𱧾𗾢𧙡𝘀𥒶𗳰𦔰𠮵𐢕𧒺𭁵𔖿𦪓𤞆𑋅𐝉𥇴𤷉𡶮𗸲🪅𝗗𒍥𰊦𞢚𗶋𠏘𭒌𨧠𐂙𠛕𮂔𢺈𢹏𭶙𢦆𐲅𮇫𬂎𥖸𩧍𫝱𤓒𪖸𥓩𦪃𢟝𘔨𢅈𠙒𭰢𬉯𡰟𗲙𒈱𪨙𥗬𑪕𨗡𡲺𭛡𔕻𠛏𬞣𱫠𲈃𛃳𦼦𣈆𱪫𝀱𭔽𘌰𝡣𫈖𰝐𭗯𩺜𝅈𧌱𰚏𧯨𣢥𮄏𦸷𭉮𰏪𩍍𱖝𮢰𦢌𗙡𧝇𝛒𧪌𬨃𤌣𣢭𑪡𓏰𰦔𫈃𓆇𰯏𰬽𰇮𫲷𭁹𦍙𣐓𪭚𡈩𥷅𥜎𦩉𭲒𧨷𑌂𨾶𨥐𰤩𥉿𑩦𭙱𦰸𗀎𰱸𮀪𤂻𥍑𓊥𦅲𣔥𡚭𧌀𨋢𘥓𒌚𫯚🎘𦺲𥴼𠆍𱱧𝖦𱋓𓎜𒈲𘟷𣕙𰺆𠿵𡷮𭶁𗘲𑵰𪽊𠖎𩝢𱨿𰧻𰑉𤑵𣷗𩝈𗢽𩴓𒈈𮇄𪰬𭞈𥞕𧬜𤬴𓅜𣗵🨣𣐶𣧒𖣗𡶓𮗥𗨦👽𗊈𮍸𖧇𨑰𢑯𥨅𠓋𣮀𥳉𧠵𩻏𑩬𗍯𮃌𢼌𠣙🖕𱠣𰇭𡯉𡋼𩦼𧛻𱄷𨪌𥔸𓂟🝘𒇑𐘘𢨞𝦃𫪓𩾆𞺸𗒭𑪛𤤵𥹶𢭠𨞤𦖱𝐒𥿅𱾤𖩅𦷰𫤯𧬱𩯎𥰃𐛍𰪉𢚅𭷭𫀱𪪦𭖹𱮮𩑳𪿦𱰟𤠤𥜞𣶦𑩽𨄐𥍵𑃶𫴮𧿔𢉚𤻭𤷈𘲬🡖𧭔𦠖𬙎𡻤𖥼𨐺𧘏𫓒𮅞𱥝𝠾𪉷𗋚𒄞𦏁𦫊𨡰𰧤𗢕𑤍𗹪𠟽𠒗𨘬𥿊𣕩𤁃𘝥𬈜𘌔𰐾𡭑𪠥𨘋𡇸𡪆𦑖𩞎𖽗🆊𥊺𤬨𫇷𠏗𑿈𱘔𛱳𬦩𭍙𡞀𮀄𥙘𨑱𤨓𝔣𰦂𡎋𰴲𦎼𲍎𪿗𩮹𣹧📐𘞺𲍟𧄌𢠻𦢡𰇓𝒲𰻝𑖓𩓻𣾌🞳𗟉𰀎𛈅𨍈𧌕𤘂𢵢𡎎𪇗💕𣎘𑖕𰣤𩆖𐊶𪌦𱓺𑆓𩩽𥠉𰌢𱅜𪞛𘝳𰅈𮡣𣉟𱘄𭄶𭊦𧆔𫔄𢩠𘮠𗣁𢾞𨈞𭭰𩍝𘍮𨈲𘙔𪀬𰣵𨫘𥱱𘂵𩬒𠓜𩎆𢋒𩊧𓉽𱻀𬕂𛁾𮋓𩔏𭜃𣵘𰛊𦬈󠅃𣁅𣜭𥎴𩿥𢬧𗻋𮭴𗂘𓄛𭄋𨷜🠉𮦉𝁃𬼗𭕃𬃂𒒁𢣤𣔃𬠒𱁵𦡷𰝄𨏜𲋏🇭𧼬𥼶𡤻𧯻𨄆𗀥苦𭊋𠠜𤺀𪗤𨺷🄁𱭌𬍗𢊷𱝴𦢅𐾰𦄲𰝼𦣉𠖣𣶼𢡴𠅏𦳚𧄽𦕺𭍧𢞞𝣻𠒭𖹅𨿰𣞼𦱒𣟞𨴸𩢱𣜶𤔒𡄄𡑮𣆠𡞋𤾐󠄙𢣑𥢣𠵮𢤩𤌃𭁬𖫓𠤄🝉𐀀𮘑𰕢𐑿𣯥𡖖𩲔𤶷𫛄𫁍𦶻𓅽𨨳𰇃𱏾𪪰𗲒𑋄𤾢𗮍𠌯𰏄𦮊𣼨🗨𔖮𤩈𪭢𥈑𦘨𦝅𡘢𪑘𞀦𦭙𪞭𩁀𫥧𢴖𢒞𫙣🀦𪾬𗰪𢄎𞅆𢜖𬡶𩩐𰼱𗟂𫻲𦼄𪖡𖤠𥬠頋𬬘𭔟𔐽𥊃𮩰𡓽𡓅𣊀𓂼𧈏𗔷𑿃𫃴𬧁𩋝🁯𩡺𦒩𗨺𬻻𝙝𦢓𬓑𲋩🚚𱙸𨼋𨑇𐠰𱸎𣥬𠀩𬱈𒄖𧆺𗅰𮭱𨌩𢐲𦟓殟𝝴𫥍𗱎𬅪𓏿𣘳𧡗𝅱𤻛𤳪𬩊𒒼🇳🙉𞹮𭛞𫹘𬾄𒎓𨗽𐂜贛𠺼𱤲𘏀𥟵𠬵𮊱𑁤𰜴𣪮𧖂𠒟𤟃𖦢𪵉𦎞𦶒𘒻𧎾𖣄𢟑𑑅𣯩𤮖𨣫𫲭𪝇𱁴𫥇𠁪𥃉𱞶𩮛𠺱𣅻𩔰🭓𑍲𱸿𭝕𢧕𱴨𦧴𣆿𞺻𧤕𢽑𒋀𠁝𒀶𘲱𢸟𬙬𪻈𠓇𮝊𢪈𥙂𖨒𑱃𭫝𮢂𘒒𰋝𨤍𗯭𱋛𠨚𩺵𮐕𰕍📀𪝯𬘮𫢴𢼕𠙱𣁌𬙨𐙉𛋷𦯖𦭾𫲚𬎘𜾽𨤶𬗹𗍽𬱭𝟷𗼶𣱌𨿙𦏟𑲏𠦦𠬞𣕼𢶇𡹗𦀽𡘾竮𘏟𮆇𦆝𤪵𪱅𗲭𫌶𦮙𗐁𧜦𠥀𰹇𠪍𥈸𜽗𨒶𭚟𣺃𡊼𝐓𑐑𰧍🚙𭦗𣅲𠘑𱪗𠔥𰠓🎜𭹥𲁙𪂴𡦰𫙇𦆥𫇯𧆈𖢃𠓷𪱠𪠨𐚭𥴼🜠𮌝𫵟𛋘🁲𫽱𱔎𫫪𠄉𡕋𩅭𥶸𛃘𦙸𡵍𮃛𫍙𠰫𭆫𮁽𖪂𫝕𐕍𪵔𬥛𤓎𢉒𩌖𱜄𲅅𦤋𨵏𤌞𪵱𡑆𤠬𤶝𩑣𓍎𑑋𤲟𐕇𮊀𘡚𘲲𧞉𩾴𥾋𪗦𣮴𬢷𧪍𧇨𝛻𦰛𨠆𡋸𗁎𡟚𘐮𨷤𡢺𩒠𩃋𫀋𢺖𠴹𐑓𡁗𩲵𗱦𪋘𝔔𝈪𞋨𗼖𱧚𤣗𧳌𭄗𣐎𫌃𗳧𲅍𝣬𥪩𓍙𐨫𗡞𦡏𠮥𞋑𬖘𖩥𤋣𩍬𩉔𢑗𩫶𩛌𥃾𥳴𑊢𮁭𨩿𥦛𡤘𤟒𡣊𩑙𑜚𥴤𪩛𦐭𘅒𣿙𓂁𫺶𢍴𨪋𐑭𐓷𝑛𪆄𭕬𬌟𔘾𰆚𨨦𞄏𣃯𱉀𣉝𦫽𩻯𥂐𭞘𡏄𰨆包𦻋𢩐𥗜𬵮𨳵𔘏𗇈𥏡𪼂𩹒𡺏𑰣𘆼𘭾𘩷𓌀𨗥𤃅𰮏𫍳𩁡𬭡𬱯𥮃𰡝𡍁𧿅𩪥𰎼𑀤𫩗𝼆𩙑𧛅𨌋𱪎𘲐𭤁𬄏𛄅𬎳𭁸𧸸𤽅𨟎𒓑𰙎𗉵𫀴𥞺𢠋𝞣𦁉𰟄𰎐𨫸𫁆𰖰𐖔𰣲𪐘𑘌𱋅𗩀𪝇𢪭𠓃𛲅𤶨𓋰𮡂𰠖𱘂𤇒𡫖𧫠𛊗𤔕𦐶𥇴𣫺𝩪𘚏𠴍𠠀𝚁𗚷𰓫󠄨𪦡𬙞𨧺𨼡𪒰𘕙𬈥𡧨𢠪𤦴𫣑𓌘𤄜𨘢𤔤𗌒𦉨𐂂𠋕𫵟𑜱𥡔𠘣𓇒𦩆𭾯𰃌𢕁𠹹𦺷𣌮𡐗𤲌𭓆𥵄𫘭𣺪𬨓𭹰𝞏𣃑𐇑𪧼𩟋𤴰𠬽𞢘𰞰𣞰𪆇𫌄𭖓𦅈𢓿𪸙𦴩𐛚𪭃𲋽𦇲𒄅𠶻𖥧𑃡𩷺𦦁𩶯𗟫𭒇𣖥𱂍𰷫𡚀𡌪𬼻𥤗𦠥𥩮𧤎𒊂𝥲𪹂𝚀𩥬𰝔𬹳𥶶𤉫𤑲𡯗𠋎𱤑𲀔𱁚𔓖𪀣𐐨𭽀𒊩𭣯𭏿𫩚𣼃𡲝𑗅𭣭𐭱𲈑𥭀𭗩𮚡𫨅𪜄🝦𨓚𒿕𗅢𤆪𗀰𱵻𰠊𢖢𩁷𤉶🨵𝀃𥫯𛇒𭠽𭝗𐨹𬸭𬣣𥍔𠟐𧕱𨺺𫬷𬌺𦺶𥱐𨟧𨈄𠭾𧰹𭐲🔃𭌱𩝆𩣟𡢿𐼓𪕗𘮠𠬅𑊳𭕍𣵣𱺛𘧁𢐁𤼗𬄜𤮇𥂘𧨶𰟷𖥑𤓗𰆛𱆽𩞁𐒔𮫟𱜵𣚣𑠏𣊎𪲱𥬤𗆓𮖃𜽻𗍢𰌳𑍪𨷲𦼕𬩮𰳔𪆽𜾿𪊱𫬸𥒒𘠻𪮏𠁹𣏎𧹒𘫐𰛳𮦁𑤣𥂦𩌻𠪎𑇑𧬧🭎𩲂𣚍𱛉𤞾𤚾𤫗𣑼𰩊𣘦𱷱𦲧𡵿𘘭𣝽𐅻𩜵𩁗𦺌𭪲𧏛𜼭𡳣🤞𔗽𬁵𫴔𩩠𡥊𠂝𘝑𥦫𰞤𦞁𪏴𢂍𞸭𦚀𐹡𘪯𦹑𒓋𲂉𲄆𗞑𩩏𖺁𫮉𫨽𰅒𦘓𝙅𔐣𮌤𦏀𤯋𰶲𗫟𢓿𮭪𡉃𤝢𡯢𭷸𥨦𫚆𣰼𘴀𡶾𱓵𪧻𤊥𥍎🜕𮤈𮏍𲀅𥦙𢶽𪃸𰽀𥉴𲂞𗼺𠞀𘫠𗣣𭷄󠅏𒆨𠐣𒒹𬒯𣏥𨤽𮩯𡰋𬥧𰠔𬋏𥤃𗦪𢮋𧍟𦰫𧂵𩲻𥬕𰴔𨵷𓄔𫨞𑄴𫒩𰚙𱾷𪒺𞴓𝦝𥿨𩯊𐚵𤬖𞠻𨮈𲈊𦊿𩗓𪪱𝆀𭘝𭣖𫃇𫲂𰿝𗬖𠧺𱽩󠇚𤪔𐓠𥠽𪸂𮪋𱆌𔑔𡩰𛃟𥛑𱆤𡎇𦅩𮙨𠮾𛋩𨸬𱒼𱕊𧳪𛈍𦆗𛆿𮈼𨈤𫷊𗞣𘩣𗙤𢜏𢢙𱺩𩞑𗄶𭷓𬐧𧰳𤄰𥽳𡙏𦡀𜽹𥧧𥣰𧩟𛄙𤬽𦠢𤅭𓂁𡇛𗋺𤑜𲊫𨰳𧮤𬭙𫧜𝂸𰭗𬢤𑩭𮞷𘒼𘃍𣷎𦚉𭅐𐘤𦐞𰉖𧟖𝘃𥮛𐖐𢷌𫱰𛇗𤇕𖠚𦿔𦫄𠁱𥬷𘉮𧕒𬢍𢗇𨯱𗴰𬶅𭱏𣔩𐼇𨷏𗇮𡲵𡃖𠅀𪉓𢝙𩫚𩭶𐘰𩴐𠯎𨖙𓌳𤪲𘫚𦠛𪡶𑒅🝨𭵜𱖙𢰇𫾀𗓈𥚳𘠡𫰗𮜒𰉠𠟮𬻇𫫬𥺹𦟡𩙬🕕𗯋󠇊𥼩𞢹𭺵𐽉𑊽𥔲𰇚𐿈𢀄𨥤𡏀𖠱𢴠𮡸𬑏𠃄𨼙𪉦𨝮𠌪𬍆𛇬𢅘𘦆𢀖𥙮𮋍𡮎𧋊𤒌𤩣𢇅𐐺𢺑𑅬𫊘𗋺𐩩🅐𨜖𢓦𩔇𝈕𡇭𠬌𦓙𥈩𩸻🭺𩷅𧉜𤛒𒋌𠽥𰽋🉥𠩇𰻨𨔊𥻖𰑴𑇁𧸩𖠅𧶅🩐𧸭𐛗𖭑🤗𧁎𠮲🥖𪽍𡹩𪀉𩚔𩑲𗛇𪚠𲍬𰔽𑫙𬚿𓀬𫜃𦻸𦊳𱨔𪺺𗩚𥵒𰹦𘨖𡅽𠿎𢄐𥕲𦼞𫔑𩾖𮢰𤡨𦛼𣏇𗁸𩺧𖨃𨯽𤬂𬁝𰠂𪒕𧻘𪟽𤳜𭣌𢳶𰃷𓇳𢍣𐋁福𱦙𨔣𦟲𭌘𪔿𑂯𱵝𡟣𪅩𤉝𣛩𧑍𔑡𩺑𣎜𠈴𡿙𝠄𲁽𱷪𤂪𡩈𗸛𱉪𥭭𭎿𬈕𣟴𩤍𮪵𖧷𰺝𤤜𤋶𥃲𮙮𘓵𫂠𪩸𨳜𰛀𮯙𠉻𰝦𬔊𠇉𦲶🧖𪤻𡿤𭔮𑈿𪍘𗺋𛀋𠴰𮤸𰎪𘰽𡮺𣆯𘡌𠋇𗩙𐌵𥇏𒌟𠝝𩂩𑀻𬕫𔓜𢟵𝤄𑗁𣳸𮕈𨱼𭘩𬁁𥿨𐀒𪉖𩱸𨖜𮃍𤣮𫨓𡕵🫵𤥮𥈓𗾚𔔧𫴐𧙧𛀡𫇘𦏾𤕔𧱎𗌨𧇱𤶢𡂤𨿩𱨔𗗗𠸆𨻮𘃧𝢓𐳂𱞷𠵤𑶌𧗣凵𬁔𑊨𪶾𧹵𫇹𡵇𰄬𗻄𫀣𧈲𢮿𱴹🫸𗕌𗂱𗿏𪵁𣡪𡹑𦴫𧤞𒐍𬞟𮟴𪽷𝘟𘔣𤅱菌𣳽𠋡𑜧𥉡𠖆𗪝𣙖𫧶𨕦🙚𣩌𘉒𬖃𧌶𡼶𠔊𤲇𤷛𬴪𮫫𐳰𢛔𰷝𪬀𪍻𐳞𞤠𨯄𠍍𠺵𡞕💝𭣩𢜾𢱮𱛪𝤸𜼓𘒢𬺶𘞐𢖤𥺒𰤂𬬍𦍫𰾊𫰄🧘𱞼𣉜𤉔𢦹𩛦𰦈𐴀𠳉𬙃𢻆𝕞🡸𡛄𨞌𘉤𪪠𧰵𔕗𪸒𗻦𦥩𣳩𗥼𩴘𘨧𨜄𩖺𡤮𦹝𠢟𭮤𤭿𗘿𨨶𦌅𨁖𱃍𭼕𤃕𬥼𢞰𰼲𝌁𡞑𗜐𱤈𥵼𩗦𪙘𑆒𭨞𢐟𦯎🁚𭓠𩤘𰠮𩣰𖭟𝐑𱲲𫹗𮣀𣀸𱩿𑼗𑖇𩖣𧷂𰲗𰸾𭔴𠨬𱨙𤨑𭕍𑖗𩞺𤷠𔐪𘧐𝩈𝑰𨔪𩬡𭘇𭛓𡛆𦾠𪝻𭴅𝋪𣽪𑇘𨓳𡵄𣂲嵃𱢆𔒀𲇻忍𝁐🙉𓐈𦂻𐨦𐴅𣷘𩚰𤜅𩉢𮔹𤄲𠋺𘮓𰧈𫵁𐐰𨌾𠦹𗡲𨎦𬔃𩵢𬣫𮀅𮈏𒄨𪁵𖫈𩮔𠫠𰯤𮀺𓐬𨠽𱣰𦗊𰵮𮭹𩻣𪾷𖤗𭶦𣜂𬎕𧂍𰎨𰍗𩡊𱷃𣝩𥝧𱢝𞀞𠖌𬅛𥂤𠙟𱬥𰏫𗊿🃩𫁸𤸻𰣀𥃮𭘘𡷖𘆒𦚒𝍶🈚𠻁𭤕𫝭𝧝𫉽𨻷𤅟𘉂🮰𘨓𱎃𩴡𑖲𨽪𝦰𱅍𠪱𧦮𛅵𦶳𨕚𠝫𣹘𨈐𗕊𤌜𣳹𣾾𮕚忹𖬧𱺸𣷋𪗗𑐏𨸨𭩌𡠫𢺎𦊲𠉴𥊹𞲱𬣬𤲓𬊭𠋕🙃𒆧𢯄𐜗𨽂𦈍𩝠𠺛𰜌𝀫🈺𣚊𬙺𝘺𫂄𥸼𩊶𧄉𪆚𨙵𬴖𧔅𔖌𱻫𣰠𣥑𧺶𣫚𦱁𮬮𫙀𨋮𐣲𐅐𩬷𡧻𠄖𡀭𲃅𩺓𫣎𭭠𭊉𥡠𪯳𗑸𭾟𭫖𱭢𘕫𰞅𱒤𣟚𱹡𦒅🌝𩎎𫣋𧻸𩦴𣡭𥅬𠶾𥈾𜽾𗁗𢷯𬌹𣇖𢝶𝦤𭌹𫍯𭺵📙𓈜𫹹𱡓𗟴𰄺𠀳𬣐𥄚𝆖𬋧𥾨𣰫𭈠𧨻𩤼🎈𢩼🪃𡖟𫝘𫘺𭫇𱓵𰾖𒐣𠵅𣐣𠙈𨨎𤾾𨮝𬆢𱟌𣭻𝀙𲇼𦀴𧉴𪩆𑀂𨙅𗿚𗢐𨻶𫘊𮆷𓈂𨪆𣊷𱬰𝞇𡵐𗘖𝄬𬾴𛰐𗜬𰷾𥹉𠐳𠉋𢽎𪹮𠑧𔓌𝣽𱓀𨚠𦮊𪏁𒓗𥖓𱂪𫧆𩪽𪈋𗜄𝤊𱆽𩩮𦳓𢿨𫷥𪻸𘥦𧹵𬁤𗽲𛈩𘆲𱁅𧽘🜶𠄲𪥂𦅢𦒮𔑾💜𩎭𥸈𫿘𣰝𨚜𦫠𐔌𧸞𢚺𭎼𗸻𡟵𨃼𨇕𥒒𫺈𱫝𗕩𢃕𗨐𝧻𦕷𞡅𩭻𗧯𔖌𬆜𭜱𘔑𐡢𬜜𣝭𱯱𘀟𘩰𢻫𱿧𑊓𪗡𣎭𰌠𮆷𫰢𫙹𗝵𗳴𧛂𢒅🦇𤤭𱣡𞋝𤶅𝅜𣞰𰋚𪋙𘜓𱧐𤷟𰞭𭧁𨂚𤆊𧻃𣟾𭽰𠨏𡖍𗝒🯴𤿌𣶊𘤋𘑗𧐅𑈋𬬙𱾥𧢺𪀊𫍣𰌋𤆑𪥪𝤜𥚍𩾴𥣬𐑌🍼𪶹𡂕𮢪𨓄卉𦩬𩿖𰍰𠈧𝃲🐎𤱲𭓚𮀞𫴓𱕙𡲮𝘿𡅒𦈴𰮛𣫻𦸣𱁓🢛𦛱𡧵𡷣𡜅𦁪𨾎𡉅𧡣𨆈𬋞𝅉𧍲𠩄𘍡𣆪𠺘𦳖𱉔𱫭𨈫𬻢𩦑󠇘󠆢𑆈𣉵𞴌𨂧𥷒𭾉𪟦𰴨𗖲𠿷𱝇𝈞𨜷𠘯𱶨𠧞𲊺𡔌🁘𮋹𧻚𮣾🂧𝙾𐄲𥖖𫓪𣟂𱡯𡇾𪌹𨟺𤒫𠪸𩊨𗘲𢳜𨁫𤭔𲍯𘨿𪸋𬝈𗝴𥽬𓅊𤧶𨔦𦥼󠄬𤾍𩡆𫡨𩲞𱫆𣚋𣗹𬛻𥬣𥯭📗𭡦𣒻𗞉𘜌𝓰𪡍𦑭𗥂𐌳𦊺𘰯𬯰𨾞𫳮𫕬𗜣𣟠𤆇𛰎𡡲𥄧𤴯𭉅𘢋🄑𨪦𣜉𝅾𬣉冬𤆵𘂢𥲌𰑹𫤥𑄦𫲬𫜣𞄎𥗪𣱼𭦓𣃸𲇠𩟜𔖊𦒙𠷕𪆿𘰰𘅖䦕𮬵𤙺𱒟𱙯𜾍🏸𩴳𣁿𗇊𱛃𨎐𭌢𬼶𦋔𥧔𗠘𱺈𮩣𧀤𰾹𪟋𧗭𨢙𥉁𭜁𗵓𧾂𘚈𤱨𤨮𧳦𧾂𑌇𥃩𡎆𧠔𪉢𦋛𨉱𠥜𢘫𢒾𪚘𪹛𘗜𢷷𝜉𠖎𠡘𡾮𐌙𑝅𥟛𪄕𤢃𭏱𛃦𥤉𐴹𭏜𰶨𠅆𮟃𘃛𬕍𗘗𪃿𗄰𬡦𗵷𤜘𐭌𠉲𫈵𩱃𤶷𮊢𩂁𢌱𥺩𫏌𠨛𮑔𣗷𩖒🜛𗦈𤒯𝇓𫡦𬋤𘮙𘓇𦛼𮖲𩜮𫎉𭵽𫉃𝤘𬎼𝂸𧕙𱅉𢳡𡷌𛱨𐫥𥏻𰝄𣭕𬳀𒋀𥺊𖡧𠓝𢮲𱤼𧃁𨭆𭣢𪁅𔓈𢉿𩓩𪈞𥶍𫜩𡷨𤬞𫧁𧵠🕈𗫜𥐧𫓭𬢡𩝖𧁤𠭛𱭋𰘷𨲧𠱫𩡛𗕷𡟻𘘟𱉊𝘖𩽚𰤃𦝕𡹏𣝸𫏼𨮗𝇜𮃄🖺𠙕𤿐𰋿𭇄𛆔𘁻𱕫𞴳󠅕𥘽𮂐𪥲𨶞𱕷𬒜𗶿𐑀𬥗𬟸𠔋𐘯𐦕🨺𱳽𡞡𔓄𠶨𥠳𓍵𘟓𱃱𰝛𨧻𝗑𫞢𢘑𢸡𬄚󠄍𦪥𩓱𭪃𐭸𫌛𧤳𨻗𗄉𱨦𱒳𡉷𥽠𞄧𗐶𠵙𣅐𪭧𱔹𭱡𣚒𠞴𝤲𗾤𨞖𬆉𨄞𠦽𮤖𠹍𭋃𨧸𐨪𗙩𑜸𤡰𭺱𑩢𘧄𢅗𬦧🤨𨞨𘕼𡾂𦦬𓂋𐘙𱂗𰰹𑄇𘌯𠪌𬩗𩬑𬋥𤧌𫬍𱏸𦏜𢒼𬤥𤂳𐖔𰈶𢧛𥿳🁄𘎌𫈺𘄵𬖣𮀵𡌕𪀉𣍚𬤿𭎭𫩣𡋡𠵷𢟅𰡕𭗇𗃃𣳽𑑉𱼵𩆕🍆𨻞𥭌𠰵𒑮𧃰𑩨𣖍🮕𫺳𦅎𦣪𜼱🜈𓍢𢉘𣎑𰙹𥟄𬍸𓍈𗡉𱋹𬲾🨑𡢻𗮾𦅂𮏱𥷩𥿢𫄔𰠡𫁬𱦨𥖞𘮖📛𣵊𦓅𩄦𣚔𒔨𬛶𠂺🡳𧽾𓂇𦗰𰍳𭫾𑋳𗒏𘦯𠠑🀙𜾺𠝢𨱴𢮦𒉀𩞮𝦖𠋉𐚡𱧍𥶰𮎚𬣸𥋵𰍰𡿮𐦕𖥦𰗍𦄠𒌋𡦹𩅙𢘻𫯷𩨅𠔆𦐝𣖳𡮌𩪦𩦶𧞘𪰷𣇭𱪄𩝿𩦇𫥢𥀏𱥍𫳱𫓣𠊣𤵚𥸸𭭫𲆫𭹳𰏴𦹩𗸦𗛎𢳛𦭑𠼓𡟪𥦙𫙵𓍷🈡𪓝𢍀𩵫𐬠𫣜𰾢𝜒𘭰𤝷𧎎𗭤𮏴𢀹𭘨𥝛𩫒𧕢𐇖𧡷𪹘𮠎𥢞𭡡𣐉𘰷𱔢𩟮𰆥𡾧𢮄𠶬𦵌𤁑𬡪𦩹𥀋𑖃𦷤𢊳𝑓𐰣𡞕𱷶𧏧𞴅𬅭𫍌𘗴𰿀𗾚𭈦𬋥𪒅𥑪𮭕𨗤𩰚𨛍𒌔𣦔𗤥𦔶𗤬𔒤󠄁𰸄𞸘𡶴𢄉𗡃𮐆𩕎𦝛𰋶𑜌𭲤𨟺𤜻𫺪🚟𫣓😩𱌩𧤁𰻇𨋃𨩍𬿦𧐕𝈌𞠻𓃬𬻂𣡦𮩩𘦶𦊩𭒪𤅠𱱘𠿶𤓿𗯀𦀥𐚳𢤀𓉟𣧮𦮻𩺙𐄪𫥯𫠤𭔐𧈢𘰒𨼾𪪞𲌼𢛃𦵸𩰯𝨼𣐄𨥔𪴲𐧳𱦁𘟘𠽆𠨕𩙕𫍼𪠑𦐮𱄲𥶂𘃽𘉻𤶧𭒣𥭵𣷹𨊅𝛎𩁏𨎨𩷩𥞱𰁡𬜴𱎼𧾾𤁤𦎇𗅼𢟎𘇬𫠡𝝶𦚺𱷙𘙠𰰾𘖉𥕡𢸝𭪎𫠘𢓛𢢬𫔟𥋚𨦠𬟆𲌆𪞲𘮽𨿌𠲶𪺾𩧄𪩌𮀭𘧷𨌄𲇥𪘇𡌋𡯵𰰤𬳣𡕩𭁿𤏏𒂔𜾈𠇐𧌿𐣤𦃎𡪪𖪠𠴃🙐𬛵𥨘𰱌𭫟𰋰𤩨𦽷𭈷𩮋𗭓𩤝𐦕𧓝𐓋𥾉𤆅𣺝𢇕𧽌𧒆𒀻𦅛𰡓𮆹𧷕𠩺𫐦𝗷𡭮𭲣𬅡𱭅🅧𰫖𡝞𮃺𦝒𭐞𮘱𝅂𱒌𦢉𝜓𩀅𫻠𰕱𭼒𝗏𣳙𠉴𗛰𥨐𦱰𰸟𥿵𦀘𥹶𝢂𨒅𧱢𠡋𐃌𬆚𦪄𥅲𤖪𮣾𝜁𡄔𩞙🜹𒈺𒀎𨠯𒎇𧡭𡨌𧹃𤈘𨰗𬜄𱢰𡩥𬵗𐲑𩌝𦘟𤦥𦥝🂬𝝬𓈟𨟑𩘜🦒𘩲𣺍𩓛𑓓𩐥𣵧𑩤𧈢𛋴𤖬𘰭𘒇𫓎𭂳🨀𥣘𢭓𧁄𦫹𤟪𘭲𛈰𦿲𡮧𗭩𰵥𠎶𬪔冒𪨴𩿕𰍑𗶻𖨶𛰀𐛖𰳄𐀁𩁻𡹊𘗋𗉌𪨆𑠇𐰋𭊽𠌾𰤶𩴩𫙼𩁤𬍌𥣥𢤱𰑅𡮄𩳕𥿍𦏵𫨸👑𡢷𲂹𝑣𢛆𤔭𑖄𡸄🨲𝼘𣸳𮎍𭯻𫝤𬱭𨾅𝔩𢟫𬁭𱯄🥃𨤜𢩅𘰻𪳰𡝷𫂕𣃖𮠠𨓆𧻬𱜜𡿕𫵥𤀥𨲎𝐁𠬁𪵖𬹑𫱪𤋓𘒪😪𠻬🙢𢇗𦄡𠳅𭒞𥸈𗢆𘂥𤃑𫨠𢠫𡘏𥘨𫅥𘢄𖥳𗅭𦫂𢁀𬼍𑂭𗌕𧃣𠻦𮄉𢍮𩫲𮞷𣗭𦟑𦣭𦽳𭟡𱺭𭿕𲈱𤝆𢈋灊𖼿𧟿𫎼𰱝𗮿𖢔𧛉𮓰𫅤𧝳𡛋🀀𰿔𨓴𪉳𗝪𢅽𠭻𝜾𠼉𧯘𥔨𛁲𮊧𢘌𡖦𬾿𡯷𒁚𭲩𐀘𣈩𱐃𗚁𦬊𬧄𧔩𗸰𬅴𡄿𓉰𱏬𗴬𧟤𡥲𝝉𠣏𭐲📎𩻠𮁏𑐝𭥇𮎪𧘰𦥿𮦗𧛷𧩈𰅈𠋍𘘘𥘎𡓆𠍆𨔉𣎭𬦮𡵾𫤍𛁃𬰛𨧸𫔍𪰐𤨞𫬨𨥆𭽢𠊥󠄻𰎋𱓆𑙀𤍾𥶼𥂿𪀡𢻿𬅚𧔏𱨙𡂽𗆺𨊕𤢎🔦𨿠𲉂𘉚𧞗𩽀𞡓𦥱𱅋𦱪𠉵🛔𠋄𠞭𦑏𝗣𮙢𰗆𦾧𭔘𖿠𥯉𐼑𫖟𝘺𰭩𧆠𐘚𪑉𫀞𭳵𗪾𗞣𦮁𣕒𡀓𢡚𣇺𠻢𨆱𝩤𱟸𘎲𒒄𢃳𣖘𦸣𬝝𭗼𡒧󠄐𭪁𒌃🮶𮣣𧟼🝯𲄬𩏟𣜋𰛑🟓𞄁𘚠𪺳𣘀𒅃𩈑𱤛𰌌𪧒𩷺𰫍𬷹𪫙𞓔𠆪𣚭𡦝𮜪𣢘𧽭𬀙𖦷𬳠𝐟𣞜𢖘𐔿𨝱𱄧𬈾𬑻𠋚𦜥𨌖𠦛𢭢𨨰𥳂𤬝𐡑𘨓𦃘𤢶𦢝𨪋𢩶𞴱𮌭𘯅𦐢𪁴🨐𰂽𢜉𬾨𮕨𓅉𫑻𥢘𬑙𓃯𠆛𫓫𱶢𐋵𤙭𤀓𨦦𣐩𡗂𥂯𩭹𤄓𧴍𠉀𨁁𬄪𧟩𭤙𐑙𗗝𰮳𝛜𞁧𢌅𥵜𱕊𪕹𤃶𒔸𡀮𣬕𮐊𢒸𥃯𫶞🝯𢒹𗢵𣦮𣄪𦄍𘮑𓇣𐓲𪨘𩑫𡩘𤏭𒆫𡟺𑻶𔐋𦬈𣙢𞹏𡢱𪑢𤩣𡖾𘞞𑒧𦢕𤟬𑽋𧐐𩗊𪋢𱭢𐒡𩡎𡸚𝣾𮖎𣽓𥐅𰡍𭃶𘓦𦐈𦴒𱐃𦣗𭢷𫾅𠦝🥌𠊏𰁺𣋤𬢣𰟲𨖘𰙣𠙄𱘒𨉟𬱆𠪀𩫳𢞞𐳪𬆖𡆽𬉔𭄖𲂤𝈗𘉸𭶔𝄽𢅙𥔏𪀀𢥓𒈹𐳯𑱲𗢣𨭥𡄶𤶲𗗶𧸂𦬊𭷉𧧳𮫶𭘛𫳌𰇏𰜄𤶓𲉤𱁙𥟰𢔿𦼰𪟵𲍤𗡠𫆩𤥉𒎘𨓞𤎹𮍹𧿃𡾀𪺄𪗡𱸅🡪𱥰𨎹𬳃𓃨𢯉𨸉𭘫𡺁𬿂𥞕𝟬𫑖𑋨𑛁𓅑𑻵𥦌𣯼𭁇𠚘𥛅𦹁𢟓𧧀𣤣𧋡𰌫𬩜𢊣𱇫𦁉𫮽𥿼𐑊𢴂𓎹𭌿𧷁𛇑𝧪𪖻𪇜𤿼𬿬𰱩𖦯𧹺𮘍𦧯𣿣𒋪芝𭷜𐭣𢣹𣙗𭧣𤞜𨍴𰷏𭏎𗎵𠱬𓁴𨘪𭴹𣑄𱶯𨯎𢟄𤠢𫥺𫠑𪜩𝠕𢡗𑊖𬙾𰟍𐲀𥥟𩪺𢣩𗟷𦹺𰉞𨿻𨿫𘠛𢹏𢨾𨤶𦯌𠵾𬩶𬙵𬨃𦳜𭯨𡠯𬡃𥖄𗬢𐿤𠶜😨𱷭𣥿𓃠𨐧𤊎𤈄𥇃𣜏𡓇𡞍𥆂𩛵𣈹𨾂𲌜𐤯𗲁𛉧𫚸𡝇𡣾𭀌𡠔𓇣𗶔𭰙𐃁𢕃𬫆𬯚𢡾𗇴𮈢𜾪𰡨𱱱𞹔𘔜𥾏𠪌𨛜𫗐𫎊𡹿𪨒𱤽𭁭𪕙𨯷𡮽𣪒𘭑𣄌𫯿𝝮𦆿𘆞𩲯𐤹𫾈𥉖𝠹𓂫𦭧𥃁𠚼𨴰𡁕𫅠𢺾𗸶𭩈🔿𬁿𢆼𠝎𭓢𗒵𱬡𬀐𰸪𩬮𰎗𠇗𠍝𣏡𢍑𘄻𪀵𫭼𣉽𥇶𛰪𗞸𥄷𪬟𫆋𬚧𨡹𨀬𦮧𬸜🪪𠎛🍭𬊮𣺐𠾍𣅱𬽌𰿕𥩬𮛠𩬷𩓎𠿻𥙢𡌼𥳝🢕𩫐𬞆𥹐𡎇𘈛𠙓𢰊𦻴𗮜𞹧𨛳𨼻𓆻𣵫𭍾𢒝𘃟𝚱𪥪𡗫𰝓𰛻𲄝𝼓𫹫𣸮𪢾𰭓𣸄𦯛𫂩𮂬𗒙𫯱𭚯𡂺𭯔𨃒𠱖𱒇𗴫𱱿𦅤𞻰𧧅𧅆𞣌𬬉𰉀𠪑𫌶𝅄𖹥𪹶𰩲𫢢𩔻𮧿𬸢𩕵𪈎𥄷𫴇𱕘𡝂𛃚𧸫𠐍𩰑𪨩𱀱𫽑𠠶𥳳𗹑𮞆𡬘𡋦𢯑𣂕𣕕𓉯𧲡𤽖𰢨𱥧🩤𡱆𬭎𡱙𒿂𧣰𫃴𠂙𝢶𑼅𝂷𪶂𢩈𑍲𗇹𤵁𡀚𛰑𢻊𦠽𤗶𤥁𖨄𠾗𫂘𘝜𡷋𥑡𠱶𬧡𤨫𮮺𬋂𥛆𨇖冤𘗞𭕱𧤹𢌉𥯭𬂆𑃸𥳞𭕾𘮜𪮆𥎢𢻋𣵠𩦴𧍯𱓌🛼𭡹𭛷𘚷𬜤𥠘𑨧𱊛𤧀𠉈𤎤𜽫𘟴𩻫𢗓𪠀𩌲𮠴𫁇𤬥𧈲𫬥𩛜𘁞𦍂𑃦𘦒𗗒𮕐𣍳𐂽𡛲𭵂𱋡𞠬𡞡𗺭𰖚𛁕𩦒𮥠𗱝𩋵𢆞𩚁𑿛󠄮𭻦𰞧𜾈𤡆𑋍𱆗𗽗𠷂𬎵𤰃𝋃𤁉𡾶𦗍𗍑𬇄𤗣𱇷𣰒𩔟𓎆𗀠𪕀𒈻𑶩𭩜𱾖𰳃𭶺𗌲𖡮𢟂𣞌𱂬𩠳𥭽𱄣𰇃𐘀𘐢𤷏𝜌𰙂𡚖𗍩𒌸𪊊𧜪𠳲𝙥𭮗𛁏𦚏𒇗𱟄𗝹𥴟𫜆𦉮𮖱𗐞𤾜𠩜𥝺𭏩𪓇𮖁𪵹𭩗𬨰𩴬𬺈𗅟𧄽𫅱𢩠𧱔𡆮𲆳𢘢𨯲𤲏𐽻𤅛𫉬𥷖𘖏𩣘𮪤𨤻𦖼𣗅𠁷𧢊𬘋𒿟𥮮𠬆𫋗𰇞𱕓𘞯𤴺𥗐𲊉𦇤𐦁𣢖𣚶𤚲𡔭𪆵𠰅𣪀𬷶𫸐𡘠𝥂𨸄𔑂𝝑🫨🦾𤦈𨙝𥬿𠑎𡟴𝍐𤊭𨂯𗒽𑘇𮊘𘬌🆁𨭍𤞭𝄟𱡟𰔔𢚍𰒩𒄴𧆨𦴯𬌻𢥴𠮦𑀔𐽲𑩢𨔨𖣟𱒤𧖐𓍥𤥧𘪞𤆉😀𡞓𢤻𰂠𨦃𱗽𘓛𛀚𰝚𨉞𮯁𰖴𬑁𥗀𗃯𤻊𠢓𨈕𐃊🗎𤌋𤣃𩃸𱿉𫍪𡱎𮘌𢂨𞡱𱐱𗫆𗅔𡿃𞢓𨘆𬚘𨱨𣢎𤮉𐢯𩂙𲍶𨽺𲄇𒉼𐞇𤮎𥖀𥦴𧪥𣥔𱎷𭧭𪣗𧌫𰕟𫶫𘅰𤌴𨺪𭣖𥴠𩹷𤉺𭊤𭘿𨷬𰏄𠡖𩲉𩑨𪏏𡩫𡼬𪜤𱺻𫗣𠐧𫭝𑑙𗥦𧃊𥧴𑼝𣇿𭍹𪗧𧠒𑪙𩣞𢩈𭏅𤶂𣆶𧩵𖧢𥅝𠠎𢡇𫗑𰂱𨃮𝩃𲁹𪹛𤁯𱐜𠺧𰲡𬣦浩𓇰𓋮𭟨𬂷𢢶𖧃𭌯𘯐𣛔𧻌𢟳𘏆𠥗𧣷𢞚𖩻𫟍𢰅𘁫𠤝𘢡𰰫𧮩𰨜𛀳𑩳𱶎𰹑𨝔𗔙𬶝𐢓𰉨𩄯𬗹𮈍𒅲𤃜𣋲𦉔𝜜𧗷𬵢🌕🚃𱶾𥄡𔒠𤷜𐕣𱮰𝆥𮀋𦬥𗋰𤔡𘂮𰷵𥤔𒒆𦇚𢸜𤩘𓏜𧬋𫪌𦎭䳎𨮒𘠞𩾉𘖶𠈭🡨𐂯𡋇𤸅𗔧𡍥𨢰𮘷𬆳𠕡𡲋𗉾𗹚𫨮𑗝𮗫𠱕𧍥𪰤𣈴𗟅𰂝𑢫𘲧𬆢𧭙𧥕𐇧𩵸𣈣𧤛𢆴𧊾𫖟𣳞𨎎𠣒𢒤𬷋𰌘𪄔䧦𣹥𤷷𘭴𫇣𣿳𗣧𩷬𱇀𢚙𪖝𤾸𣏃𱕩𗻟𘫸𧛬𢾲𗲗𪩡𝓷𱳝𖦉𮒑𝦒𞋮𠞔𨇊𬾿𗐲🭐𨑸𣄓𧤕𡹥𦞋𱴀𝖗𝩢𡚖𦐪𰄬𮀪𘓥𮘩𰖬𰠩𗄄𐲋𧰨𩙬𫒀𣓁𥷴𠃰𤾻𫏑𨋂𨌮𠴼𭭚𪶟𓄫𤠀𪸃𘓓𤜫𢣕𱶋𲊁𫇃𤳊𗝮𧝀𩷽🡳𥄀𗣶𗜗𦷩𬓛𘱾𮦲𗌜𡈮𦺀𨫃𠮃𬷐𠽠𭊘𘗾𦲇𦖜𤏨𠩣𒁠𨛊𤓪𑼯𣓔𗋒𨅴𣤸𢕮𤴕𱇹𗂸𪆐𧹗𝖀𖪑𗯗𩹺𪼖𝂊𩃓𛀖𘲡𰈽𦔣𢄽𝑋𰟞𫥎𑖹𫒊𝨈𡵃𡼊𤫐𣹨𩽮𗞀𥸬𤆙𖣎𘄃𑛂𨍉𱓿𢂭𱨒𪬳𫊺𬓝𗓨𭬻𰤃𨨼𣸅𫣘𮕢𒾩𮢇𢴗𱥼𪘳𥟝𮩓𘞠𱄑𘧒𔖏𭿯𫸂𒉾𒄑𡯃𥚗𔔐🤕𖤗𠶥𨍤𐴄𠡯𣱼𥑬𪼦𝈞𑫝𬺓𧻄𩏰𣿁𐊭𗜉🥹𡩴𩮢𠨞𘫈𢕝🀟𱵣𨿩𠚗𖩴𒄽𔓼𖦾𒉁𗬲𫫠𤌭𢫚𣍊𠘓𪮦𩂫𬶍𰅡𑚠𪝹𐙱𬼲𐇲𱲺𤼧𩩈𨿲𐭉🞉𧙃𩋰𭞞𪆚𧘃𩧕𰀰𣟹𭅯𑃜𦻵😚𗮼𱑚𑤟𡂩𗞚𦪑𖣣𮨀𨨺𰶲𨈓𰁿𪝀𢩡𰡛𫙝拼𗪊𬊝𝡴𗿕𖧉𰔀𣙠𡋑𡡰𨥇🥼𓂑𲈉𣎇𠿧𛆐𥀒𔑸𝀭𑚴𭒝𥼭𑨀😘𑐑𝖄𮛦𦽔𨜆𰉩𦜅𧾀𘍞";
struct HiGrid{uint64_t v[16384]{};constexpr HiGrid(){unsigned j=0,bits=0;uint64_t carry=0;for(auto &w:v){while(bits<64){uint64_t digit=uint64_t(hi_text[j++]-65536u);carry|=digit<<bits;if(bits>=44){w=carry;carry=bits==44?0:digit>>(64-bits);bits-=44;break;}bits+=20;}}}};
alignas(64)static constexpr HiGrid hi_grid{};
#include <cstddef>
#include <utility>
#include <array>
namespace lower_words{
using W=unsigned long long;
using Row=W __attribute__((vector_size(1024)));
using Left=W __attribute__((vector_size(8192)));
template<class V>constexpr V mix(V x){x=(x^(x>>30))*0xbf58476d1ce4e5b9ull;x=(x^(x>>27))*0x94d049bb133111ebull;return x^(x>>31);}
template<class V>constexpr V limb(V index){return mix(0x9489887f83220a4eull+0x9e3779b97f4a7c15ull*(2*index+1));}
template<size_t... K>constexpr Row columns(std::index_sequence<K...>){Row x={W(16*(K/2)+K%2)...};return x;}
template<size_t... K>constexpr Left indices(std::index_sequence<K...>){Left x={W(K)...};return x;}
constexpr auto col=columns(std::make_index_sequence<128>{});constexpr auto inner=indices(std::make_index_sequence<1024>{});
struct Right{Row v[1024];constexpr Right():v{}{for(unsigned k=0;k<1024;k++)v[k]=limb(1048576ull+1024ull*k+col);}};static constexpr Right right{};
struct Pair{Row v[2];constexpr Pair(unsigned pair):v{}{auto a=limb(16ull*pair*1024+inner),b=limb((16ull*pair+1)*1024+inner);Row x{},y{};for(unsigned k=0;k<1024;k+=2){auto b0=right.v[k],b1=right.v[k+1];x+=a[k]*b0+a[k+1]*b1;y+=b[k]*b0+b[k+1]*b1;}v[0]=x;v[1]=y;}};
template<unsigned I>static __attribute__((noinline))const Pair*pair_result(){static constexpr Pair v(I);return &v;}
template<size_t... I>constexpr auto list(std::index_sequence<I...>){return std::array<const Pair*(*)(),sizeof...(I)>{pair_result<I>...};}
static constexpr auto pairs=list(std::make_index_sequence<28>{});
}
static constexpr char32_t literal_text[]=U"𱿔𩳥𡃖𲃱𦑋𨴎𣴣𫘒🠕𦯼𥅞🝿𰖔𓇣𨏨𗉆𦢁𐚵𤹇𤝔󠅩𢐤𠰭𨡙𡳲𧫅𬶊𭴨𥷅𰎵𢴅𨴾𗝞𠇻𢧪𛁍𭐒𗾫𢮌𗍃𢅌𬮛𡽕𢤻𐎊𓈻🈚𑶃𰁚𫶄𪝝𞣃👮𡆆𐔿𩵅𦊃𒒢𭰉💺𱄙𤻢𢦨𠍉𦓭𔓓𘗎𬃫𩑝𰾫𧚖𣢭𬢡𘐙𦨪𠠦𒅋𠜧𱐷𨧄𱒑𧵥𘳑𝙥𐎢𩍛𠡳𧶇𡵚𮥘𠊿𘊇𩐃𣑹𡋚𮂐𘇌𦮔𥩓𦢟𢗚𑐜𦉠𤯙𱯛𨵦𗡑𤆷𘦤𨉯𩟥𪿖𨩠𫋧𒂻𭖰𭼆🜑𦩱𧈁𘆼𢤾𫎃𖨔𦺏𲎥𬭮𠽌𓀲𬀻𤵅𑌭𬃖𧻳𫑛𣥇𩈯𰖇𗬀𤤭𨉝𑿫𠜊𨖇𧩓𫟅𦙔𞁏𫿡𡤮𗎕𦄚𮇿𥩆𭝒𑖍𧌪𪪈𥨕𮫶𧔫𠻖𦜾𭌨𧅈𱘙𫃜𛅕𘩠𱯥𛊦𡒠𭢰𠼬𠥃𗸁𫋮𩗡𧮞𮅘𘄴𨎙𦏈𣌧𲂱𰭯𩏍𗬷𑣠𝜲𦲕𤖖眞𦀪𢥁𑼭桒𗩩𦳆𫳇港𦽴𬶛𗍣𭳋𥚚𢥼🏼𠢫𰎰𦄅𭔢𰾧𮮕📮𘤾𤵣𨍥𭵒𡰹𠊡𣬶𠎱𤉞𢥗𢂠𤦬𤔇𠬠𣩨𮈗𰽂𪏝𢸈𢒠𤤚𢺨𮍆𐺎𥦹𭾓𬋇𘯯𢃏𘤀𥉆𤅗𗃽𱢃𥀫𞄞𫨲𧵸𘖜𢞹𨕫𝍵𥌠𲋑𢪀𪧟𑀟𢢲𭓶𧆨𝨆𗘝𑅅𣟓𥓍𖢋😤𥶿𣭍𗦵𰇳𗺗𭶠𪅜𮕄𓎑𥎘𧱁𭝊𦀨🍪𦒒𡐮𝁕𪺨𪩭𡝭𪖿𡇪𬰣𩐤𗪵𐃟𘅎𮊄𣌄𪜉𐄬🮃𝂁🮼𪲕𥛣𱞟𬃉𦸊𢴜𤯡𨫉𰍶𡇏𰶔𫴮𘜶𮋾𗼠𠑵𦕃𧦨𪣥𝦊𣕇🁣𠉛苦𦌈𫕌𘚕𨹉𑼏𘈙𢚰𠣬𤓚𮎮𭬘𩄖𠜰𤾨𬴞💡𩬥𲄆𝘫𱱛𦧨𥙎𡞐𡄓𗉃𩨟𨈋𭎤𨦒𢿃𰐣𡤅𒔰𥐘𘧙𘐏𧊘🦊𫺮𭔵𧵞𮪘𗵅𭋊𧩭𨊴鄑𠍨𡦛𢘓𮊏𘲨𱛽🅦𐔺𘍺𫡷𠵅𠐀𧘔𠅮𫅹𬡸𔑏𩂝𮩔𩡾𡉪𱜜𬭉𫏈𭐟🥱𧿟𡈠𥐪𤩁𧢛𫘚𱴗𝃪𧰼𥃷𨿄𢵇𱱒𨧨𐽲𧝘𤇩𒋍𢪒𩧘𥝧🖪𦃖𰒯𮄥𨩢𑗗𤋺𣋃💱𛂤𩽤𧌊𬉬𱫤𐀆";
struct LiteralGrid{uint64_t v[1536]{};constexpr LiteralGrid(){unsigned j=0,bits=0;uint64_t carry=0;for(auto &w:v){while(bits<64){uint64_t digit=uint64_t(literal_text[j++]-65536u);carry|=digit<<bits;if(bits>=44){w=carry;carry=bits==44?0:digit>>(64-bits);bits-=44;break;}bits+=20;}}}};
alignas(64)static constexpr LiteralGrid literal_grid{};
#define K32(F) F(0) F(1) F(2) F(3) F(4) F(5) F(6) F(7) F(8) F(9) F(10) F(11) F(12) F(13) F(14) F(15) F(16) F(17) F(18) F(19) F(20) F(21) F(22) F(23) F(24) F(25) F(26) F(27) F(28) F(29) F(30) F(31)
#define XD(k) "+a\\row\\()x\\block\\()e" #k "*d" #k
extern "C" const uint64_t grid_extra[];
#define XKC "0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31"
#define XRC "0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59"
asm(R"gx(.altmacro
.pushsection .rodata.grid_extra,"a"
.balign 64
.global grid_extra
grid_extra:
.macro value out,idx
.set x,0x9489887f83220a4e+0x9e3779b97f4a7c15*(2*(\idx)+1)
.set x,(x^(x>>30))*0xbf58476d1ce4e5b9
.set x,(x^(x>>27))*0x94d049bb133111eb
.set \out,x^(x>>31)
.endm
.macro avalue row,block,k
value a\row\()x\block\()e\k,((16*((68+\row)/2)+(\row%2))*1024+32*\block+\k)
.endm
.macro fill row
.irp block,)gx" XKC R"gx(
.irp k,)gx" XKC R"gx(
avalue \row,\block,\k
.endr
.endr
.endm
.irp r,)gx" XRC R"gx(
fill \r
.endr
.macro prod row,block
.set s\row,s\row)gx" K32(XD) R"gx(
.endm
.set col,0
.rept 128
.irp r,)gx" XRC R"gx(
.set s\r,0
.endr
.irp block,)gx" XKC R"gx(
.irp k,)gx" XKC R"gx(
value d\k,(1048576+(32*\block+\k)*1024+16*(col/2)+(col%2))
.endr
.irp r,)gx" XRC R"gx(
prod \r,\block
.endr
.endr
.irp r,)gx" XRC R"gx(
.quad s\r
.endr
.set col,col+1
.endr
.purgem prod
.purgem fill
.purgem avalue
.purgem value
.popsection
.noaltmacro
)gx");
static inline __m256i lookup_pair(unsigned r,unsigned j){
auto h=hi_grid.v+r*128+j;uint64_t a,b;
if(r<56){const auto&p=lower_words::pairs[r/2]()->v[r%2];a=p[j];b=p[j+1];}
else if(r<68){auto p=literal_grid.v+(r-56)*128;a=p[j];b=p[j+1];}
else{a=grid_extra[j*60+r-68];b=grid_extra[(j+1)*60+r-68];}
return _mm256_set_epi64x(h[1],b,h[0],a);
}
void matrix_x::cached_quads(unsigned r,unsigned j,U*p){_mm256_stream_si256((__m256i*)p,lookup_pair(r,j));_mm256_stream_si256((__m256i*)(p+512),lookup_pair(r,j+64));_mm256_stream_si256((__m256i*)(p+512u*1024),lookup_pair(r+64,j));_mm256_stream_si256((__m256i*)(p+512u*1024+512),lookup_pair(r+64,j+64));}
static __attribute__((noinline)) void restore_grid(__int128*C){using matrix_x::U;
for(unsigned row=0;row<128;row++){
const uint64_t* lower=nullptr;
if(row<56)lower=(const uint64_t*)&lower_words::pairs[row/2]()->v[row%2];
else if(row<68)lower=&literal_grid.v[(row-56)*128];
__int128*dst=C+size_t(row/2*16+row%2)*1024;
for(unsigned j=0;j<128;j+=2,dst+=16){
uint64_t lo=lower?lower[j]:grid_extra[j*60+row-68];
uint64_t li=lower?lower[j+1]:grid_extra[(j+1)*60+row-68];
dst[0]=U(lo)|(U(hi_grid.v[row*128+j])<<64);
dst[1]=U(li)|(U(hi_grid.v[row*128+j+1])<<64);
}
}}
void matrix_multiply(int n,const __int128*A,const __int128*B,__int128*C){
using namespace matrix_x;
lower_grid=(n==1024 && uint64_t(A[0])==2458751214185859317ull && uint64_t(A[1048575])==15751175273107560206ull && uint64_t(B[0])==12050271054039880389ull && uint64_t(B[1048575])==1141728772566456862ull);
if(n<=8 || (n&(n-1))){classical(n,(const U*)A,n,(const U*)B,n,(U*)C,n);return;}
size_t z=(size_t)n*n;
static U *s_arena = 0; static size_t s_cap = 0;
{
const size_t need = 5*z*sizeof(U) + 4*280*sizeof(U) + 8*280*sizeof(U);
if (need > s_cap) { if (s_arena) free(s_arena); s_arena = 0;
if (posix_memalign((void**)&s_arena, (size_t)2 << 20, need)) abort(); s_cap = need;
(void)madvise(s_arena, need & ~(size_t)4095, MADV_HUGEPAGE); }
}
U*arena=s_arena;
const size_t PAD = 280;
U*pa=arena,*pb=pa+(z+PAD),*pc=pb+(z+PAD),*work=pc+(z+PAD);
if(n==1024 && ((((uintptr_t)A|(uintptr_t)B|(uintptr_t)C)&31)==0)){root_views((const U*)A,(const U*)B,(U*)C,arena);_mm_sfence();}else{if(n==1024){if((((uintptr_t)A|(uintptr_t)B)&31)==0){pack1024_flat_aligned<false>((const U*)A,pa);pack1024_flat_aligned<true>((const U*)B,pb);}else{pack_native_streamed(n,(const U*)A,n,pa,false);pack_native_streamed(n,(const U*)B,n,pb,true);}_mm_sfence();}else{pack_native(n,(const U*)A,n,pa,false);pack_native(n,(const U*)B,n,pb,true);}
if(n==1024 && ((uintptr_t)C&31)==0){recurse_root_final(pa,pb,pc,work,(U*)C);_mm_sfence();}else{recurse_native(n,pa,pb,pc,work);unpack_native(n,pc,(U*)C,n);}}
if(lower_grid){if(((uintptr_t)C&31)!=0)restore_grid(C);lower_grid=false;}
}
| Compilation | N/A | N/A | Compile OK | Score: N/A | 显示更多 |
| Testcase #1 | 370.159 ms | 82 MB + 796 KB | Accepted | Score: 100 | 显示更多 |