Sophilex's Blog
  • Home
  • Archive
  • Category
  • Tags
  • About me
  • Friends
LANGUAGE MODEL COMPRESSION WITH WEIGHTED  LOW-RANK FACTORIZATION

LANGUAGE MODEL COMPRESSION WITH WEIGHTED LOW-RANK FACTORIZATION

引入Fisher information,在SVD分解时对参数进行加权,以及讨论LLM Compression的常见路线
2025-07-07
学习笔记
#LLM #Matrix_Decomposition
ASVD: ACTIVATION-AWARE SINGULAR VALUE DECOMPOSITION FOR COMPRESSING LARGE LANGUAGE  MODELS

ASVD: ACTIVATION-AWARE SINGULAR VALUE DECOMPOSITION FOR COMPRESSING LARGE LANGUAGE MODELS

引入关于input的信息辅助SVD分解
2025-07-07
学习笔记
#LLM #Matrix_Decomposition
Dual-Space Knowledge Distillation for Large Language Models

Dual-Space Knowledge Distillation for Large Language Models

尝试解决当前llm白盒蒸馏框架下只能对同词汇表模型之间进行蒸馏的局限
2025-06-23
学习笔记
#LLM #KD
Why Exposure Bias Matters: An Imitation Learning Perspective of Error Accumulation in Language Generation

Why Exposure Bias Matters: An Imitation Learning Perspective of Error Accumulation in Language Generation

提出了两个指标来直观观察lm的Error Accumulation现象
2025-06-23
学习笔记
#LLM #KD
NOT ALL LLM-GENERATED DATA ARE EQUAL: RETHINKING DATA WEIGHTING IN TEXT CLASSIFICATION

NOT ALL LLM-GENERATED DATA ARE EQUAL: RETHINKING DATA WEIGHTING IN TEXT CLASSIFICATION

尝试通过引入sample-wise loss weight来缓解train-inference mismatch问题
2025-06-23
学习笔记
#LLM #KD
hexo+reveal指南

hexo+reveal指南

搭建markdown+js写slides的框架,并依托hexo进行展示
2025-06-12
配置笔记
#配置
Different Designs For LLM KD Loss

Different Designs For LLM KD Loss

关于LLM KD中loss的一些讨论
2025-06-10
slides
#LLM #KD
服务器转发流量至本地

服务器转发流量至本地

懒人福音
2025-05-22
配置笔记
#配置 #ssh
练琴有感

练琴有感

我要练琴!我要当大佬!
2025-04-27
杂想
#杂想
关于浮点数存储精度

关于浮点数存储精度

存储精度的误差是相对原数而言的,有点神奇
2025-04-09
学习笔记
#学习笔记 #计算机系统
123456

搜索

Hexo Fluid
京ICP证123456号 | police-icon 京公网安备12345678号
载入天数... 载入时分秒...