Stable Diffusion工作原理（How does Stable Diffusion work-中文版）

缈昏瘧鑷狧ow does Stable Diffusion work

1. SD鑳藉仛浠€涔?/h2>

鏈€鍩烘湰鐨勫姛鑳芥槸锛氭枃鐢熷浘锛坱ext-to-image锛?/p>

SD浠ユ枃瀛楁彁绀轰负杈撳叆锛岃緭鍑轰笌鎻愮ず鐩搁厤鐨勫浘鍍?/div>

2. 鎵╂暎妯″瀷

SD灞炰簬鎵╂暎妯″瀷銆傛墿鏁ｆā鍨嬫槸涓€绫荤敓鎴愬紡妯″瀷锛屽畠浠璁捐鐢ㄤ簬鐢熸垚涓庤缁冩暟鎹被浼肩殑鏂版暟鎹€?/p>

鎵╂暎妯″瀷涓轰粈涔堝彨鈥滄墿鏁ｂ€濇ā鍨嬪憿锛熷洜涓烘ā鍨嬬敓鎴愭暟鎹殑杩囩▼绫讳技鐗╃悊涓婄殑鎵╂暎銆備笅闈互璁粌鐢熸垚馃惐馃悤鍥剧墖鐨勬墿鏁ｆā鍨嬩负渚?/strong>锛屼粙缁嶈缁冭繃绋嬨€?/p>

2.1 姝ｅ悜鎵╂暎

姝ｅ悜鎵╂暎锛團orward Diffusion锛?/strong>鏄煇?馃悤鍥剧墖杞负鏃犵壒鐐圭殑鍣０鍥剧殑杩囩▼锛?strong>浠庡櫔澹板浘鏄湅涓嶅嚭鏈€鍒濇槸馃惐杩樻槸馃悤鐨?/strong>銆?br> 鍍忎竴婊村ⅷ鎺夎繘涓€鏉按锛屼笉涓€浼氬効灏变細鎱㈡參鎵╂暎锛屽皢鑷繁闅忔満鍒嗗竷鍒版按涓紝灏遍毦浠ヨ鲸璁ゅ嚭鏈€鍒濆ⅷ婊村湪杈圭紭銆佷腑蹇冭繕鏄埆鐨勫摢閲屼簡銆?br>

涓€寮狆煇卞浘缁忚繃姝ｅ悜鎵╂暎锛屽彉鎴愰殢鏈哄櫔澹?/div>

2.2 閫嗗悜鎵╂暎

鑰?strong>閫嗗悜鎵╂暎锛圧everse Diffusion锛?/strong>鍒欎粠鏃犳剰涔夌殑鍣０鍥句腑澶嶅師鍑吼煇辨垨馃悤锛屾槸姝ｅ悜鎵╂暎鐨勨€滃€掓斁鈥濄€?br> 鎶€鏈笂锛屾瘡娆℃墿鏁ｅ寘鎷袱姝ワ細婕傜Щ锛坉rift锛?/strong>鍜?strong>闅忔満杩愬姩锛坮andom motion锛?/strong>銆傞€嗗悜鎵╂暎鐨勬紓绉昏涔堟湞鐫€澶嶅師鍑吼煇辩殑鏂瑰悜瑕佷箞鏈濈潃澶嶅師鍑吼煇曠殑鏂瑰悜锛岃€屼笉浼氭槸涓€涓腑闂村€笺€?/p>

3. 閫嗗悜鎵╂暎鎬庝箞瀹炵幇

閫嗗悜鎵╂暎杩欎釜idea鏃犵枒鏄仾鏄庝紭闆呯殑銆備絾浠峰€煎崈閲戠殑闅鹃鏄細瀹冩€庝箞瀹炵幇锛?/p>

3.1 璁粌noise predictor

涓轰簡灏嗘墿鏁ｈ繃绋嬮€嗚繃鏉ワ紝鎴戜滑闇€瑕佺煡閬?strong>涓€寮犲浘琚坊鍔犱簡澶氬皯鍣０銆傝繖鍙互閫氳繃璁粌涓€涓ā鍨嬮娴嬪浘涓婄殑鍣０绋嬪害鏉ュ疄鐜帮紝noise predictor鍦⊿D涓槸鐢变竴涓猆-Net妯″瀷锛堜竴绉嶅叏鍗风Н绁炵粡缃戠粶锛夊疄鐜扮殑銆傝缁冭繃绋嬪垎4姝ワ細

1. 鍙栦竴寮犺缁冨浘锛屾瘮濡備竴寮狆煇卞浘
2. 鐢熸垚涓€寮犲櫔澹板浘
3. 灏嗗櫔澹板浘浠ヤ笉鍚屽己搴﹀彔鍔犲埌璁粌鍥句笂鏉ョ牬鍧忚缁冨浘鍍忋€?/li>
4. 鏁欏櫔澹伴娴嬪櫒鍛婅瘔鎴戜滑娣诲姞浜嗗灏戝櫔澹?br> 杩欐槸閫氳繃璋冩暣寮哄害骞舵樉绀烘纭暟鍊兼潵瀹炵幇鐨勩€?/li>

姣忎竴姝ヤ腑鍙犲姞鏇村鍣０锛岃noise predictor浼拌姣忎竴姝ュ浘鍍忎腑鐨勫櫔澹?/div>

缁忚繃璁粌锛屾垜浠緱鍒颁簡鑳介娴嬪浘涓坊鍔犲櫔澹伴噺澶氬皯鐨刵oise predictor銆?/p>

3.2 閫嗗悜鎵╂暎姝ラ

鏈変簡noise predictor锛屾垜浠敤瀹冨疄鐜伴€嗗悜鎵╂暎銆?/p>

1. 鐢熸垚涓€涓畬鍏ㄩ殢鏈虹殑鍥惧儚锛屽苟瑕佹眰noise predictor鍛婅瘔鎴戜滑鍣０
2. 浠庡師濮嬪浘鍍忎腑鍑忓幓棰勬祴鐨勫櫔澹?/li>

閲嶅1.鍜?.鍑犳锛屽氨鑾峰緱馃惐鎴栶煇曠殑鍥惧儚銆?br> 鎴戜滑鏆傛椂杩樹笉鑳芥帶鍒跺鍘熷嚭鐨勫浘鐗囨槸馃惐杩樻槸馃悤锛屾帶鍒惰緭鍑哄彲浠ラ€氳繃鍔犳潯浠舵潵瀹炵幇銆?/p>

4. Stable Diffusion

鐜板湪鏈変竴涓潖娑堟伅銆傚垰鍒氭墍璇寸殑涓嶆槸SD鐨勫伐浣滄柟寮忋€傛墿鏁ｈ繃绋嬫槸鍦?strong>鍥惧儚绌洪棿杩涜鐨勩€傝繖鍦ㄨ绠椾笂鎱㈠埌闅句互鎺ュ彈锛岃€屼笖鍦ㄤ换浣曞崟涓殑GPU涓婇兘璺戜笉浜嗐€?12*512鐨凴GB鍥惧湪786,432缁寸殑绌洪棿銆傛垜浠渶瑕佷负涓€寮犲浘鎸囧畾鐨勫€煎お澶お澶氥€?br> Google鐨処magen鍜孫pen AI鐨凞ALL-E鏄?strong>鍍忕礌绾?/strong>鐨勬墿鏁ｆā鍨嬶紝瀹冧滑鏈変竴浜涘姞閫熺殑鎶€宸т絾杩樹笉澶熴€?br> 鑰孲table Diffusion姝ｆ槸涓鸿В鍐冲浘鍍忔墿鏁ｆā鍨嬬殑閫熷害闅鹃鑰岃璁＄殑銆?/p>

4.1 琛ㄥ緛锛圠atent锛夋墿鏁ｆā鍨?/h2>

Stable Diffusion鏄竴涓〃寰佹墿鏁ｆā鍨嬨€傚畠棣栧厛鎶婂浘鍍忓帇缂╁埌琛ㄥ緛绌洪棿锛屼互閬垮厤鍦ㄩ珮缁寸殑鍥惧儚绌洪棿杩涜鎿嶄綔銆傝繖灏卞揩澶氫簡銆?/p>

4.2 鍥惧儚琛ㄥ緛鍜屽鍘?/h2>

鍥惧儚鍒拌〃寰?/em>鍜?em>琛ㄥ緛鍒板浘鍍?/em>鐨勮浆鎹㈡槸閫氳繃VAE锛圴ariational Autoencoder锛夋潵瀹炵幇鐨勩€?br> VAE鍖呮嫭encoder鍜宒ecoder涓ら儴鍒嗐€?br> encoder灏嗗浘鐗囧帇缂╀负杈冧綆缁村害鐨勮〃寰侊紝decoder浠庤〃寰佷腑澶嶅師鍥剧墖銆?br>

VAE瀹炵幇鍥惧儚涓庤〃寰佺殑鐩镐簰杞崲

• 涓轰粈涔堜娇鐢ㄨ〃寰佹槸鍙鐨?br> VAE涓轰綍鍙互灏嗗浘鍍忓帇缂╀负琛ㄥ緛鑰屼笉浼氫涪澶变俊鎭紵杩欐槸鍥犱负鏄嚜鐒跺浘鍍忎笉鏄殢鏈虹殑銆備粬浠殑瑙勫緥鎬у緢楂橈細姣斿鑴搁伒寰溂鐫涳紝榧诲瓙锛岃劯棰婂拰鍢村反涔嬮棿鐨勭壒瀹氱┖闂村叧绯汇€傜嫍鏈夊洓鏉¤吙锛屾湁鐗规畩鐨勫舰鎬併€?br> 鎹㈠彞璇濊锛屽浘鍍忕殑楂樼淮搴︽槸浜轰负鐨勶紝鏈夊緢澶氬啑浣欎俊鎭€傝嚜鐒跺浘鍍忓彲浠ヨ交鏉惧湴鍘嬬缉鍒拌緝灏忕殑娼滃湪绌洪棿涓紝鑰屼笉浼氫涪澶变换浣曚俊鎭€傝繖绉颁负鏈哄櫒瀛︿範涓殑manifold hypothesis銆?/li>

4.3 SD鐨勮缁?/h2>

Stable Diffusion涓512*512鐨凴GB鍥惧儚鐨勮〃寰佹槸4*64*64鐨勩€傚緱鍒拌〃寰佸悗锛屼箣鍓嶈鐨勬墿鏁ｈ繃绋嬮兘鍦ㄨ〃寰佺┖闂磋繘琛屻€傛墍浠ヨ缁冩椂锛屾垜浠敓鎴愮殑鍣０涓嶆槸鍣０鍥捐€屾槸涓€涓〃寰佺┖闂寸殑闅忔満tensor锛岃繖涓猼ersor鍐嶅拰鍥惧儚鐨勮〃寰佽繘琛屼笉鍚岀▼搴︾殑鍙犲姞锛屼互璁粌noise predictor骞惰繘琛岄€嗗悜鎵╂暎锛堥€嗗悜鎵╂暎sampling鍜宻amplers: 璇﹁link锛夈€傞€嗗悜鎵╂暎姝ラ濡備笅锛?/p>

1. 鐢熸垚涓€涓殢鏈虹煩闃典綔涓哄浘鍍忚〃寰併€?/li>
2. noise predictor浼拌杩欎釜鐭╅樀鐨勫櫔澹帮紝浠庡浘鍍忚〃寰佷腑鍑忓幓浼拌鐨勫櫔澹般€?br> 锛堥噸澶?.鐩村埌鐗瑰畾鐨勯噰鏍锋楠わ級
3. VAE鐨勮В鐮佸櫒灏嗗浘鍍忚〃寰佽浆鎹负鏈€缁堝浘鍍?/li>

4.4 鍥惧儚鍒嗚鲸鐜?/h2>

鍥惧儚鍒嗚鲸鐜囧湪鍥惧儚琛ㄥ緛鐨剆hape涓婃湁浣撶幇锛岃繖涔熸槸鐢熸垚澶у浘浼氳€楄垂鏇村鏄惧瓨鍜屾椂闂寸殑鍘熷洜銆傝〃寰佺殑澶у皬鏄浘鐗囧ぇ灏忕殑1/48锛?68*512鐨凴GB鍥惧儚鐨勮〃寰佹槸4*96*64鐨勶級銆?br> StableDiffusion v1鏄湪512*512鐨勫浘鍍忎笂寰皟鐨勶紝鍥犳鐢熸垚姣?12*512鐨勬洿澶х殑鍥句細鍑虹幇閲嶅鐨勭墿浣擄紙姣斿锛屼袱涓剳琚嬶紙with solution锛?锛夈€?/p>

• 鐢熸垚澶у浘
  濡傛灉涓€瀹氳鐢熸垚澶у浘锛岄渶瑕佽嚦灏戜繚璇佸/楂樹腑鐨勪竴涓槸512锛屽啀鐢ˋI upscaler鎻愰珮鍒嗚鲸鐜囥€?/li>

4.5 VAE file

VAE file鐢ㄤ簬Stable Diffusion v1涓紝浠ユ敼鍠勭溂鐫涘拰闈㈤儴銆傚畠浠槸缁忚繃杩涗竴姝ュ井璋冪殑VAE decoder锛岃妯″瀷鍙互缁樺埗鏇寸簿缁嗙殑缁嗚妭銆?br> 锛堜箣鍓嶆彁鍒扮殑鍋囪骞堕潪瀹屽叏姝ｇ‘銆傚皢鍥惧儚鍘嬪叆娼滃湪绌洪棿纭疄浼氫涪澶变俊鎭紝鍥犱负鍘熷VAE娌℃湁鎭㈠缁嗚妭銆傚彇鑰屼唬涔嬬殑鏄紝VAE file涓殑decoder璐熻矗缁樺埗绮剧編鐨勭粏鑺傘€傦級

5. 鏉′欢鎺у埗锛圕onditioning锛?/h2>

涓婅堪瀵筍table Diffusion鐨勭悊瑙ｈ繕涓嶅畬鏁淬€傛€庝箞閫氳繃鏂囨湰鎻愮ず鏉ユ帶鍒跺嚭鍥惧憿锛?br> 杩欏氨闇€瑕丆onditioning锛屽叾鐩殑鏄?strong>寮曞noise predictor锛屼互渚块娴嬬殑鍣０浠庡浘鍍忎腑鍑忔帀鍚庤兘寰楀嚭鎴戜滑鎯宠鐨勪笢瑗裤€?/p>

5.1 鏂囨湰鏉′欢锛坱ext-to-image锛?/h2>

涓嬪浘鏄枃鏈彁绀猴紙text prompt锛夎澶勭悊骞跺杺缁檔oise predictor鐨勮繃绋嬨€?strong>Tokenizer鍏堟妸鎻愮ず涓殑姣忎釜璇嶅垏鍒嗗嚭鏉ヤ綔涓簍oken銆傛瘡涓猼oken浼氳杞寲涓轰竴涓?68缁寸殑embedding銆傞殢鍚巈mbedding琚€佸叆

灏嗘枃鏈彁绀虹壒寰佸寲锛屽苟閫佸叆noise predictor浠ユ帶鍒舵枃鏈敓鎴?/div>