Achieve the llama3 inference step-by-step, grasp the core concepts, master the process derivation, implement the code.

使用Decoder-only的Transformer进行时序预测，包含SwiGLU和RoPE(Rotary Positional Embedding)，Time series prediction using Decoder-only Transformer, Including SwiGLU and RoPE(Rotary Positional Embedding)

Fused SwiGLU Triton kernels

Simple and easy to understand PyTorch implementation of Large Language Model (LLM) GPT and LLAMA from scratch with detailed steps. Implemented: Byte-Pair Tokenizer, Rotational Positional Embedding (RoPe), SwishGLU, RMSNorm, Mixture of Experts (MOE). Tested on Taylor Swift song lyrics dataset.

Hackable PyTorch template for decoder-only transformer architecture experiments. Llama baseline with RoPE, SwiGLU, RMSNorm. Swap components, train, compare

World Structured SwiGLU FFN

Transformer Models for Humorous Text Generation. Fine-tuned on Russian jokes dataset with ALiBi, RoPE, GQA, and SwiGLU.Plus a custom Byte-level BPE tokenizer.

A 36M-parameter goldfish language model with a 10-second memory + pixel-art PWA desk pet. Runs in your browser, fully offline. Adopt it at den-sec.github.io/glublm/desk-pet/

169M parameter Mixture-of-Experts language model built from scratch in PyTorch. RoPE, GQA, SwiGLU, RMSNorm, MoE routing with load balancing. No pre-built architecture libraries.

I built a tiny LLM from scratch to understand how GPT-4 and LLaMA actually work. 10M params, trained on Shakespeare, modernized with RMSNorm + SwiGLU + RoPE + KV cache. Every mistake documented.

Build an LLM in PyTorch: BPE tokenizer, GPT-1/2 + LLaMA, end-to-end train/infer

my llama3 implementation

A from-scratch PyTorch LLM implementing Sparse Mixture-of-Experts (MoE) with Top-2 gating. Integrates modern Llama-3 components (RMSNorm, SwiGLU, RoPE, GQA) and a custom-coded Byte-Level BPE tokenizer. Pre-trained on a curated corpus of existential & dark philosophical literature.